A Fabric-Based Integrated Sensor Glove System Recognizing Hand Gesture

It is known that lengthwise loops of weft plain knitted fabrics are called wales and crosswise loops are called courses. For fabric-based sensors knitted by conductive yarns, stretching along the lengthwise and crosswise direction generates change in effective contact area between adjacent loops, and as a result, equivalent resistance of the sensor varies. Generally, the extensibility of knitted fabric along the lengthwise direction is larger than that of crosswise direction. For knitted sensors located on chest band or T-shirt, their electrical-mechanical performance is studied based on the crosswise direction, along which fabrics deform. In this study, considering that finger bending leads to skin stretch along the lengthwise direction, properties of knitted sensor was tested all along the fabric lengthwise direction.

In the previous study [19], conductive weft plain-knitted fabrics made of silver-plated nylon yarn and elastic nylon/spandex yarns were stretched along the lengthwise direction within certain strain range and found that the sensor resistance increased with good linearity and sensitivity. Hence, this structure is adopted to design new sensors of the glove system here. Figure 1a shows the construction of knitted sensor, where conductive area (i.e., sensor area) was knitted by nylon/spandex yarns and silver-plated nylon yarn, other area was made of nylon/spandex yarns and elastic polyester yarns, which is non-conductive. Owing to addition of spandex yarns, both the two areas have elasticity.

Figure 1

By designing knitting procedure on computer, an integrated glove with five effective sensors can be obtained from a SHIMA Seiki SWG 061N-15G computer flat knitting machine directly. For thumb, the embedded sensor cover one interphalangeal joint (IP), but for other four fingers, each sensor covers the proximal interphalangeal (PIP), and the distal interphalangeal joint (DIP), as Figure 1b–c shows. Due to glove knitting sequence and design limitation, intarsia technology cannot be used to knit sensors covering all IP, PIP, and MCP joint consequently. Therefore, IP/DIP and PIP are selected and studied here. Being off from knitted machine, two ends of silver yarns of every sensor are connected with read-out electronics so as to obtain resistance value. In addition, knitted elastic glove is washed and ironed to insure stabilization before wearing and testing, washing and ironing parameters are controlled equally for different glove samples as described later. When experimenter bends fingers, elastic sensors clinging to skin tightly will be stretched and exhibit resistance variation.

In the investigation [19], the strain range where resistance exhibits linear response of elastic knitted sensor is limited (0~10%), however, stretch of finger skin in lengthwise direction can be more than 35% when people do some simple gestures in daily life. As is reported, elastic deformation process of knitted sensors has effect on their electrical-mechanical performance [20]. As the elastic deformation process depends on fabric elasticity and for the sake of obtaining applicable embedded sensor with targeted linear working range in this study, three nylon/spandex yarns of various content of spandex are used to fabricate flexible sensors with different elasticity. Resistance response was recorded by the VICTORY 4105A ground resistance tester (GRT) and YG (B) 026ET fabric mechanical analysis machine (FMA, supplied by Wenzhou Darong) was used to stretch knitted sensor.

As the glove and sensors needs integration, sizes have become important parameters during integration, which includes size of glove (Gs) and size of embedded sensors (ESs). Glove sensor system being too loose or snug for a person may affect sensing process. Here, three sizes of glove are knitted and performance of one single sensor on index finger is compared to determine appropriate Gs. It's worth mentioning that size of finger being monitored and fabric shrinking percentage of post processing should be considered to obtain appropriate Gs. Secondly, ESs also play an important role in affecting sensor performance. In this part, electrical-mechanical performances of different sizes of glove and sensors were recorded by GRT and FMA.

Preparative gloves are worn by experimenter and connected with a self-made electronic read-out system where five resistance channels are set to obtain resistance signals of one hand, as shown in Figure 1c. Each channel records resistance information of one embedded sensor. In the read-out software used, five resistance response curves were displayed simultaneously.

Nylon/spandex yarns that include 20/75 (nylon 8.3 tex, spandex 2.2 tex), 40/75 (nylon 8.3 tex, spandex 4.4 tex), and 70/30 (nylon 3.3 tex, spandex 7.8 tex) are chosen as plating yarn in whole fabric and in addition, silver-plated nylon yarn (44 dtex/12 fx8, resistivity of each filament is 15 Ω/cm) and elastic polyester yarn (33.3 tex) are used as ground yarn of sensor areas and other areas, respectively. Three kinds of knitted sensors are fabricated and elasticity is measured, and Table 1 shows those values. Fixed elongation test method is selected to obtain sensor elasticity, in which the fixed elongation is set as 20%, clamping distance is 100 mm, stretching speed is 100 mm/min, the pre-tensioning force is 1N, the stagnant time is 60 s, the recovery time is 180 s, and number of stretch cycle is 5 times. Three sensors are stretched along the lengthwise direction in one cycle and repeated cycles, and their resistance-strain values are recorded, and Figure 2 depicts these values.

Figure 2

It can be seen from Table 1 and Figure 2 that elasticity of knitted sensor is affected by content of spandex in used yarns, and elasticity obviously influences the electrical-mechanical performance of knitted sensors. Experiments results show that all sensors exhibit increased resistance with good linearity and sensitivity during the first elongation phase, but the corresponding strain range also called as effective working range will be larger when the sensor has better elasticity. Beyond that, it is known that the sensor with better elasticity shows better repeatability during repeated stretch process. In brief, effective working range of sensor S3 can meet the requirement of finger dorsum skin strain (up to 35%), and it has the best repeatability against fabric plastic deformation. Hence, sensor S3 is selected to be used in this glove system.

To produce an integrated glove with five effective sensors, size parameter of every sensor needs to be considered. Size parameters of embedded sensor include number of rows and wales. In the previous investigation [19], experiment results show that knitted sensor with less wales exhibits better sensitivity but have only smaller effective sensing range (where resistance exhibits growth linearly). On the contrary, knitted sensors with more wales show larger sensing range but lower sensitivity. Here, four sensors containing 5 wales are made but under four different rows 10, 20, 30, and 40, respectively and the yarn material is same as used in sensor S3. Photos and longitudinal electrical-mechanical curves of these sensors are shown in Figure 3.

Figure 3

It can be observed from Figure 3 obviously that all four sensors show good linearity over their effective sensing range. Nevertheless, knitted sensor containing 40 rows exhibits better sensitivity, while the sensor having 5 wales and 10 rows exhibits lower sensitivity. But the effective sensing range of four sensors are similar, which is about 0~50%.

Except for pursuing better linearity and sensitivity, ESs also need to meet the requirements of covering DIP and PIP joints and to save the used materials as much as possible. It is concluded that knitted sensor containing less wales but more rows should be selected based on the fact that it will be able to cover targeted finger joints. Considering length of each finger is different, various sizes of sensor are determined for 5 wales but with 45, 60, 60, 45, 35 rows, respectively, from thumb to little finger.

In addition, three sizes of expected glove are fabricated and the index finger sensors’ performances are investigated simply. First of all, default Gs in design system is chosen and a glove with random size is obtained. After washing and ironing, a shaped glove is generated and its Sp of post processing can be figured up. Hand sizes (Hs) of appointed experimenter are also measured, and expected size of sensor gloves (EGs) can be calculated by the formula (1) and (2): (1)K=(EGs−Hs)/HsK = \left( {EGs - Hs} \right)/Hs(2)Sp=(DGs−EGs)/DG2Sp = \left( {DGs - EGs} \right)/DG{\it 2} where K is the value reflecting glove's size (larger value demonstrates looser glove), DGs is designed size of sensor gloves, including length and circumference of fingers and palms. For a glove to be elastic and to cling to finger tightly, K must be a negative number. Here, K is selected to be 0, −0.1, −0.2, and three corresponding sensor gloves are obtained and named K₁, K₂ and K₃ respectively, which is shown by Figure 4a. Embedded sensors’ specification of three gloves is completely identical.

Figure 4

For the appointed experimenter, it's obvious the K₁ is loosest and K₃ is tightest among those three sensor gloves. Sensor embedded on the snug gloves gets broadened after being worn, as Figure 4b shows. After wearing the three gloves, the experimenter followed finger motion trace, as Figure 4b shows, to observe and compare detected resistance signal of the index finger sensor. It can been seen from Figure 4d that K₁ exhibits linear increase and decrease when the finger bends and returns once in a sequence, and so the resistance signal only generates one summit in one motion cycle. However, both of K₂ and K₃ exhibit two summits in one cycle, while K₃ is more remarkable. Two summits appearing in one cycle means resistance response of two phases (bending and recovering) is non-linear, which may be due to the small size of these gloves. After wearing tight gloves, knitted sensor has been pre-stretched before monitoring fingers motion, which leads to nonlinear resistance (changing) phenomenon later. In a word, K₁ is selected as a size reference for fabricating following glove system.

Herein, combining chosen elastic materials, size parameters of embedded sensors and glove, a whole fabricating process is determined and an integrated sensing glove is obtained, as shown in Figure 5. Five pairs of electric wires drawn from embedded sensors are connected to the central processing unit part (CPU); in turn CPU gets and sends resistance signals to related computer software. Experimenter performs numbered gestures from one to nine, each gesture is done for about 7 s, and resistance responses are exhibited as shown in Figure 6. It can be seen that each embedded sensor reflects corresponding finger motion successfully. Due to difference of joint angle in various gestures, resistance signal presents diverse characteristics. For instance, all five sensors showed higher resistance value when the hand performed the number 9 than number 5. In summary, the sensor glove with five effective embedded sensors can monitor and recognize simple gestures well now.

Figure 5

Figure 6

However, just obtaining five channels resistance signal is not enough to recognize hand gestures further. To distinguish various number gestures, a mass of raw digits of each gesture are recorded to extract feature data.

In this part, number gesture ‘5’ is selected as a reference position, where five fingers are in “straighten” position. Experimenter changes hand from gesture ‘5’ to other number gestures and repeats for 150 times for respective gestures. For example, Figure 7 demonstrates four cycles of five-channels resistance changing from gesture ‘5’ to gesture ‘6’, where the arrow points to resistance of sensor on ring finger. When the experimenter performs gesture ‘5’, original resistance (OR) is obtained. When the experimenter performs gesture ‘6’, ambition resistance (AR) is got. Here, the ratio of OR to RR is chosen as a key characteristic value to distinguish different gesture, which is called feature C later. Figure 8 depicts a boxplot of ratios, that is, the value C extracted from 150 cycles for each gesture, where T, I, M, R, L represents the sensors on thumb, index finger, middle finger, ring finger and little finger, respectively. The red short line in the middle of the boxplot is the median of the data, which demonstrates average level; the upper and lower line reflects quartiles of the data, and hence, the box contains 50% of the data. The width of the box reflect degree of data fluctuations; the upper and lower short line corresponds to the maximum and minimum value of the data, while red dots represent the abnormal value.

Figure 7

Figure 8

According to Figure 8, the feature C of some gestures shows larger box width, which means there is larger data fluctuation. For example, C of the sensor on ring finger of gesture ‘6’ varies from 0.3 to 1, which is also affected by difficulty in repeating accurate and identical gestures completely. In general, from Figure 8, contrast between feature data of different numerical gestures is obvious from Figure 8. So, we can know that the manufactured sensor gloves do have the potential to distinguish various hand gestures with advantages of portability and flexibility. Further, hand gesture recognition will be investigated using machine learning in future studies and accurate recognition rate will be obtained.