A study of immersive technology for product usability improvement design based on comprehensive value evaluation

This experiment was a controlled trial consisting of a control group and an experimental group, with the control group going directly to the interview session to ask usability questions and design suggestions, while the experimental group first used the virtual reality tool we constructed and then conducted the same interviews as the control group. The evaluation of the usability questions and improved design solutions in the above experimental method can reflect the level of design benefit of this virtual reality design method, and then combined with the evaluation of the designer’s willingness to use it, determine whether the overall value of the method is higher than that of the interview method.

A total of 32 expert users of an industrial crane remote control were recruited offline as subjects, including 28 men and 4 women. They all had at least 1 year of work experience. They were divided equally into two groups of 16 each, based on their length of service, gender and the company they worked for. None of them had any experience of working in product design and had never received any training related to product design.

Head-mounted virtual reality devices, such as the HTC vive, deliver virtual reality video content with excellent viewing range, clarity, frame rate and 3D depth of field. It is cost-effective and has a wide range of software and hardware to make creating virtual reality content easy and efficient.

The virtual reality-based design tool consists of an HTC vive display headset set and a virtual scene built using Unity3D software, which consists of a virtual environment and a virtual model set of this remote control, consisting of a full 1:1 model and a set of component models scaled up 5 times and split into buttons and bases. These are simultaneously presented in the virtual environment and in which the participant are. They are presented simultaneously in a virtual environment where the subject can move around on foot and observe the models freely.

On top of this basic scenario, the design tool also features freehand interaction [15, 16] and 3D sketching [17–19].

The leapmotion controller, installed additionally in front of the HTC vive’s display headset, captures a real-time image of the subject’s hands, and converts it into a digital model for presentation in the virtual environment. Gesture recognition technology can help computers to obtain real-time movement data of the user’s hands in a non-contact form, allowing the user to interact with virtual objects and correctly understand the occlusion relationship between them. Researchers have listed and compared several representative gesture recognition devices, of which leapmotion is inexpensive, sensitive and versatile [21, 22]. The leapmotion SDK enables interaction between the two hand models and the virtual model of the remote control. When the hand model collides with the product model, the product model is grasped and followed by a clenched fist, and when the palm is opened, the product model is released and hovered in its current position. The purpose of the enlarged model is to make it easier for the subject to interact with the various parts of the model with their bare hands.

3D sketching is achieved using the tilt brush plug-in, which essentially controls the line render component through the controller, adding points in the positions with corresponding coordinates when the trigger button of the right-hand controller is pressed to give a line effect, while the Touchpad is touched with the thumb to change the thickness of the line. Simple Colour Picker is a PC-based colour picker that has been adapted to work in a virtual environment to change the colour of the lines. A Simple Colour Picker based colour picker panel was added to the virtual scene and a ray was sent to collide with this panel by pressing the Trigger button on the left-hand controller to change the material colour of the lines. The system design is shown in Figure 1.

Fig. 1

The design tool is aimed at users with little experience of using virtual reality equipment and participating in product design, so it should be easy to learn and use. In many studies, 3D sketching has been identified as an accessible and flexible method of creation, suitable for the early stages of product design where efficiency and creativity are required [23]. The gesture recognition technology provides intuitive interaction, while the real-time image of the hands also gives the user an important reference when sketching and analysing design problems. In terms of interaction with the design tool, there is no reliance on multi-level menus and all models and panels are placed directly in the virtual space, which minimises the number of steps the user has to take. The availability of scaled-up models makes it easier to show product details and reduces the user’s difficulty in handling them.

Prior to the start of the trial, each subject was informed about the trial, completed the necessary training and signed an information sheet. Once the trial had started each subject in the group was given free access to the virtual reality tool to reflect on usability issues with the selected existing industrial crane remote control product. This process was open-ended and allowed for breaks, but the researcher did not communicate any design-related issues with the subjects and only provided the necessary assistance in using the tool. A semi-structured interview was then conducted with each subject in the test and control groups to record the usability issues collected.

The interview data was collated by eight additional online invited industrial control equipment experts who had extensive experience in using a wide range of industrial control equipment including the remote control selected for this study and had >6 years of experience in industrial production related work. Each expert collated the data individually to form a preliminary coding scheme, and after repeated discussions in the online group meetings all experts reached a consensus and completed the coding of all the data.

Four postgraduate product design students were recruited to form a design team, and after understanding all of the collated usability issues, the two groups were guided through the design of two improved solutions, with a 2-week interval between designs to minimise disruption to each other. The renderings were required to be of similar modelling and rendering quality and the design specification should be in uniform format.

After collating the interview data, a total of 28 usability problems were identified by subjects in the control group, 32 usability problems were identified by subjects in the test group and 22 usability problems were identified jointly by subjects in both groups. The independent sample t-test showed that each subject in the test group identified significantly more usability problems than the control group (p < 0.05).

Eight industrial control equipment experts scored the importance of each usability issue on a 5-point Likert scale (1 unimportant, 2 relatively unimportant, 3 generally important, 4 relatively important, 5 important) and took the mean value. An independent samples t-test showed that the importance of the usability issues identified by all subjects in the test group was not significantly different from that of the control group (p > 0.05). Further calculations of the mean importance of all usability issues identified by each participant and t-tests showed that there was no significant difference in the mean importance of usability issues identified by each participant in the two groups (p > 0.05).

From the evaluation of the potential value of usability problems, it is clear that the subjects in the test group were slightly more able to identify usability problems than those in the control group, the main difference being that the subjects in the test group were able to identify a greater number of usability problems.

The design team designed improved solutions based on two sets of usability issues, guiding solution 1 for the usability issues identified in the pilot group and solution 2 for the usability issues identified in the control group. In all, 40 product design practitioners with at least 1 year’s experience and varying degrees of involvement in the design of industrial equipment were recruited via the Internet. In this study, they conducted a remote solution evaluation using solution renderings and design specifications.

3.2.1

Level and hierarchical analysis

For equipment products, a comprehensive evaluation method based on evaluation characteristics is usually used to reflect on the elements of product design quality in the process of evaluation to build a judgement tree, which constructs the product design elements into three levels: the target level, the criterion level and the indicator level. The criterion level is the prerequisite principle for generalising the target level, and the indicator level is several relevant refinement elements of the criterion. Referring to the standard DB621/T148-1993 ‘Evaluation method for modelling of mechanical and electrical products’, the product is evaluated in five aspects: form, colour, material, human-machine, and overall relationship, shown in Figure 2.

Fig. 2

Set the target level as A. Within the guideline level, morphological properties B1 include structural stability C1, proportional relationship C2, confirmation style C3 and morphological normality C4 of the product; colour properties B2 include colour coordination C5, colour comfort C6, colour matching C7 and colour stability C8 of the whole machine; material properties B3 include material matching C9, process complexity C10, material quality C11, material cost C12; human-machine properties B4 include assembly rationality C13, operational convenience C14, operational standardisation C15, human-machine interaction C16; overall relationship properties B5 include functional coordination C17 volume coordination C18, overall sense of assembly C19 and overall style C20.

Definition of scales: In the hierarchical analysis method, the numbers 1–9 and their reciprocals are cited as the judgement matrix scales, as shown in Table 1.

Table 1

Judgement matrix templates

Scale	Definition
1	Compare the two indicators, equal importance
3	Compare the two indicators, the former is slightly more important than the latter
5	Compare the two indicators, the former is more important than the latter
7	Compare the two indicators, the former is more strongly important than the latter
9	Compare the two indicators, the former is extremely more important than the latter
2, 4, 6, 8	Intermediate value of the above adjacent judgement
The reciprocal of the above values	If the ratio of the importance of indicators i and j is a, then the ratio of the importance of indicator j to indicator i is 1/a

Construct the judgement matrix: According to the above scale, eight industrial control equipment experts scored the elements of each level after discussion and compared them two by two to form the judgement matrix, and the results of the judgement matrix of the criterion level indicators relative to the target level are as follows (Table 2):

Table 2

Judgement matrix of criteria level indicators relative to target level

A	B1	B2	B3	B4	B5
B1	1	3	1	1/5	1/2
B2	1/3	1	1/3	1/7	1/4
B3	1	3	1	1/5	1/2
B4	5	7	5	1	3
B5	2	4	2	1/3	1

Calculating the weight of each indicator in the criterion level and consistency test: Based on the judgement matrix constructed, the weight of each indicator in the criterion level is derived in three steps, as follows.

Step 1: The judgement matrix is normalised by column. b¯ij=aij∑k=1nakj \begin{align*} \overline{b}_{ij}=\frac{a_{ij}}{\sum_{k=1}^na_{kj}} \end{align*}

Step 2: Sum by rows. W¯i=∑j=1nb¯ij \begin{align*} \overline{W}_{i}=\sum_{j=1}^{n}\overline{b}_{ij} \end{align*}

Step 3: The sum is normalised to give the weights. Wi=W¯i∑i=1mW¯i \begin{align*} W_{i}=\frac{\overline{W}_{i}}{\sum_{i=1}^{m}\overline{W}_i} \end{align*}

After the three steps above, the results are listed below (Table 3).

Table 3

Level hierarchical weights and consistency tests

Guideline level	B1	B2	B3	B4	B5
Weight	0.13	0.0471	0.13	0.4797	0.2132

After calculating the five guideline level index weights of morphological properties B1, colour properties B2, material properties B3, human-machine properties B4 and overall relationship properties B5, a consistency test is required to determine whether the weighting results are reasonable. The consistency test is judged by the value of the consistency ratio CR. If the CR value is <0.1, the consistency test is passed and the resulting weights are reasonable; if the CR value is >0.1, the consistency test is not passed, and the judgement matrix needs to be reconstructed for calculation. CR is calculated as follows.

CR=CIRI \begin{align*} CR=\frac{CI}{RI} \end{align*}

CI is calculated as follows.

CI=λmax−nn−1 \begin{align*} CI=\frac{{\lambda{}}_{max}-n}{n-1} \end{align*}

In this equation, n represents the number of indicators, while max denotes the maximum eigenvalue of this judgement matrix (Table 4).

Table 4

RI criteria values for judgement matrices of different orders

Steps	1	2	3	4	5	6	7	8	9
RI	0	0	0.52	0.89	1.12	1.26	1.36	1.41	1.46

The judgement matrix CI = 0.0187, RI = 1.12, CR = 0.0167, which is <0.1, indicating that it passes the consistency test, that is, the weighting results are reasonable.

Calculate the relative weights of indicator-level indicators and consistency test: Repeat the above process, calculate the weights of indicator-level indicators relative to each criterion-level indicators, and conduct consistency test, and obtain the weights of each level indicators as shown below (Table 5).

Table 5

Weighting of indicators at each level

Guideline level	Weight	Indicator level	Relative weights
Morphological properties B1	0.13	Structural stability C1	0.1819
		Proportional relationship C2	0.3636
		Confirmation style C3	0.0909
		Morphological normality C4	0.3636
		Colour coordination C5	0.3844
		Colour comfort C6	0.3844
Colour properties B2	0.0471	Colour matching C7	0.1432
		Colour stability C8	0.088
		Material matching C9	0.096
		Process complexity C10	0.2772
Material properties B3	0.13	Material quality C11	0.1609
		Material cost C12	0.4659
		Assembly rationality C13	0.2857
		Operational convenience C14	0.2857
Human-machine properties B4	0.4797	Operational standardisation C15	0.1429
		Human-machine interaction C16	0.2857
		Functional coordination C17	0.522
		Volume coordination C18	0.2073
Overall relationship properties B5	0.2132	Overall sense of assembly C19	0.2073
		Overall style C20	0.0634

The 40 product design practitioners who completed the styling evaluation of the modified design solution continued to participate in this study. Each person was divided into five groups according to their company, with three groups of eight people each, and two groups of 6 and10 people, respectively. The reason for grouping by company is that there is a high degree of variability in business focus, working methods, staff composition and financial status between companies and it is uncertain in the initial validation to what extent this will affect the judgement of cost of use, therefore the feedback from each designer will be grouped and discussed individually.

The cost of using our virtual reality design system was clearly higher than the user interview method. The researcher conducted online focus group sessions with each group of designers to discuss the details of how to use the system, the caveats, and the costs of using the design system in practice over time and how to categories them. There was general agreement on the types of usage costs included in the design, namely: portability of the device, complexity of operation of the device, difficulty of using the software, time consumption, price of use and overall usage costs. Each designer’s willingness to accept each of these costs was evaluated individually using a 5-point Likert scale (1 not at all, 2 rather unacceptable, 3 average, 4 rather acceptable, 5 completely acceptable) and the average level of acceptance of each of the five groups is shown in Figure 3.

Fig. 3