Deep Learning for Sign Language Recognition: A Comparative Review

The communication gap that exists between normal and deaf individuals is the most important motivation in designing and building an interpreter to facilitate communication between them. When embarking on the design of such a translator, a comprehensive set of objectives must be taken into account. These include ensuring accuracy, speed, efficiency, scalability, and other factors that contribute to delivering a satisfactory translation outcome for both parties involved. However, numerous challenges have been identified in the realm of sign language recognition, necessitating the development of an efficient and robust system to address various issues related to environmental conditions, movement speed, occlusions, and adherence to linguistic rules. Deep-learning-based sign language recognition models have gained significant interest in the last few years due to the quality of the recognition and translation that they provide and their ability in dealing with the various sign language recognition challenges

The main contributions of this work are:

Provide a description of important concepts related to sign language including acquiring methods, types of sign language, and a description of many public datasets in different languages around the world.

This paper is organized into eight main sections as described in Fig1. To facilitate a better and smoother reading for this review, a detailed description of each section is presented below:

Introduction: Provides a brief introduction about sign language, deep learning, describes the motivation behind this review, introduces the main contributions, and illustrates the main layout of this work.

Figure 1:

Paper Organization

The signs of sign language must be captured and attained to provide input for the recognition system and there are various acquisition techniques that provide several types of input such as image, video, and signals. Basically, the main acquisition methods for any sign language recognition system depend on one of these acquisition techniques.

1- Vision-Based: In this type of system signs are captured using single or multiple images capturing devices in the form of single images or video stream and in some cases uses an active and invasive device, to collect the depth information that represent an accurate information associated to the distance between the image plane and the relevant object in the intended image [14]. This category is easy and presents a low computational. There are many imaging devices for signs capturing images in the form of RGB and depth data including [15]:

Single camera: Refers to a filming technique or production method that involves using only one camera such as Webcam, digital cam, video cam, and smartphone cam.

Generally, the major advantages of vision methods are that it is not costly, convenient, and non-intrusive. The user simply needs to communicate using sign language naturally in front of an image capturing device. This makes it suitable for real-time applications [16]. However, the use of vision-based input suffers from a set of problems including [17]: -

Too much redundant information causes low recognition efficiency.

2- Hardware-Based: This type mainly depends on the use of some types of hardware devices that can capture or sense the signs performed by the user when attached to his/her arm, hand, or fingers, and convert these signs into signals, or images, or in some cases video. Motion sensors are the most widely utilized devices that can track the movements, position, shapes, and velocity of fingers and hands [18]. Electronic gloves serve as the predominant sensor technology employed for capturing hand pose and associated motion. They are affixed to both hands to acquire precise data on hand movements and gestures. The hand’s position, orientation, and location are calculated precisely due to the hundreds of sensors supplemented in the gloves. The most significant advantage of this method is its fast reaction [19], so it is highly accurate. However, since it depends on costly sensors it cannot be considered an affordable method for the common deaf people. Moreover, it suffers from relatively low accuracy or complicated structures, and the insufficient amount of information provided by the wearable sensors often affects the overall performance of this system. Some popular examples of sensors are described below [20]: -

Inertial Measurement Unit (IMU): It is an electronic device employed to measure and report an object's specific force, position, angular rate, and sometimes orientation with respect to an inertial reference frame, and acceleration. It typically consists of a combination of accelerometers, gyroscopes, and sometimes magnetometers.

In general, although these methods exhibit higher speed and accuracy, the necessity for individuals to wear sensors remains impractical for the following reasons [21]: (1)

It may cause a burden on the users because they must take electronic devices with them when moving.

3- Hybrid-based: In this type, the vision-based cameras together with other types of sensors, such as infrared depth sensors, are combined to acquire multi-mode information regarding the shapes of the hands [22]. This approach requires calibration between the hardware and vision-based modalities, which can be particularly challenging. The purpose of a hybrid system was to enhance data acquisition, and accuracy, and attempt to reduce the challenges and problems of both visions and hardware-based approaches [23].

Feature extraction is the process used to select and /or combine variables into features, and effectively reduce the amount of data that must be processed, while still accurately and completely describing the original data. It has a role in addressing the problem of finding the most compact and informative set of features, to enhance the efficiency of data storage and processing. Defining feature vectors remains the most common and convenient means of data representation for classification and regression [34]. In the context of sign language recognition, the process of extracting pertinent features plays a vital, and decisive role. Irrelevant features, on the other hand, can result in misclassification and erroneous recognition [35]. Within the realm of DL techniques for data classification, automatic feature extraction holds paramount importance. Integrating various features extracted from both training and testing images without any data loss is a crucial step that greatly impacts the recognition accuracy of sign language. In general, two types of features are considered in sign language including manual, and non-manual features. Manual features encompass the movements of hands, fingers, and arms. On the other hand, nonmanual features represent the fundamental component in sign language [36] like facial expressions, eye gaze, head movements, upper body motion, and positioning. The combination of manual and non-manual features offers a comprehensive representation of sign language. In the domain of DL, features can be classified into two categories: spatial and temporal. Spatial features pertain to the geometric representation of shapes within a specific coordinate space, while temporal features account for time-related aspects during movement especially when dealing with a sequence of images as input. By employing feature fusion and combining multiple types of features in the process of sign language recognition using DL, one can achieve the desired outcomes effectively [37].

The variability in the environment during sign capture poses a significant technical challenge with a notable impact on sign language recognition. When capturing an image, numerous factors, such as lighting conditions (spectra, source distribution, and intensity) and camera characteristics (sensor response and lenses), exert their influence on the appearance of the captured sign. Additionally, skin reflectance properties and internal camera controls [38] further contribute to these effects. Moreover, noise originating from other elements present in the background and landmarks can also influence sign recognition outcomes.

The movements in sign language are dynamic acts, exhibiting trajectories with distinct beginnings and ends. The representation of dynamic sign language involves both isolated and continuous signing, wherein signs are performed consecutively without pauses. This introduces challenges related to similarity and occlusion, arising from variations in hand movements and orientations, involving one or both hands in different angles and directions [39]. The determination of each sign's precise beginning and end presents a significant hurdle, resulting in what is termed Movement Epenthesis (ME) or transition segments. These ME segments act as connectors between sequential signs when transitioning from the final position of one sign to the initial position of the next. However, ME segments do not convey any specific sign information; instead, they contribute to the complexity of recognizing continuous sign sequences. The lack of well-defined rules for making such transitions poses a significant challenge [40], demanding careful attention and a demonstrable approach to address effectively.

The segmentation process stands out as one of the most formidable challenges in computer vision, especially in the context of sign language recognition, where the extraction of hands from video frames or images holds particular significance due to their critical role in the recognition process. To address this, image segmentation is employed to isolate relevant hand data while eliminating undesired elements, such as background and other objects in the input, which might conflict with the classifier operations [41]. Image segmentation restricts the data region, enabling the classifier to focus solely on the Region of Interest (ROI). Segmentation methods can be categorized as contextual and non-contextual. Contextual segmentation takes spatial relationships between features into account, often using edge detection techniques. Conversely, non-contextual segmentation does not consider spatial relationships; rather, it gathers pixels based on global attributes. Hand tracking can be viewed as a subfield of segmentation, and it typically poses challenges, particularly when the hand moves swiftly, leading to significant appearance changes within a few frames [42].

In the realm of sign language recognition, the classifier's selection and design require meticulous attention. It is essential to carefully determine the architecture of the classifier, encompassing its layers and parameters, in order to steer clear of potential problems like overfitting or underfitting. The primary objective is to achieve optimal performance in classifying sign language. Furthermore, the classifier's ability to generalize effectively across diverse data types, rather than being confined to specific subsets, is of paramount importance [43].

Real-time recognition of sign language is considered to be an important concern and one of the main problems that needs a real solution in order to give an efficient interpreter to fill up the communication gap between the deaf and public communities. The time problem arises from the need to process video data in real-time or with minimal delay. Computational complexity, both in hardware and software, can be quite demanding and may present challenges for the deaf community to effectively deal with [44].

The acquired signs may exhibit issues such as low quality, noise, varying degrees of orientation, or enormous size. Therefore, the preprocessing step becomes indispensable to rectify these issues in sign images and videos, effectively eliminating any environmental influences that might have affected them, such as variations in illumination and color. This phase involves the application of filters and other techniques to adjust the size, orientation, and color, ensuring improved data quality for subsequent analysis and recognition. The primary advantage of preprocessing is enhancing the image quality, which enables efficient hand segmentation from the scene for effective feature extraction. In the case of video streams, preprocessing serves to eliminate redundant and similar frames from the input video, thereby increasing the processing speed of the neural network without sacrificing essential information. Many sign language recognition using DL overcome the environmental condition problem using a variety of techniques. Table 3 illustrates the most important related work, the environmental condition being addressed, and the proposed technique. Fig5 shows a sample of the NUSII dataset to show the environmental condition problem.

Figure 5:

Sample images (class 9) from NUS hand posture dataset-II (data subset A), showing the variations in hand posture sizes and appearances.

Another challenge arises when attempting to recognize signs, particularly in the dynamic type, where movement is considered one of the key phonological parameters in sign phonology. This pertains to the variations in hand location, speed, orientation, and angles during the signing process [104]. A consensus on how to characterize and organize movement types and their associated features in a phonological representation has been lacking. Due to divergent approaches and perspectives, there remains uncertainty about the most suitable and standardized way to define and categorize movements in sign language. In general, there are three main types of movements in sign language [105,106]:

Movement of the hands and arms: include waving, pointing, or tracing shapes in the air.

The movement involved in demonstrating sign language also involves a significant challenge, which includes dealing with similar paths of movement (Trajectory), and Occlusion. The arm trajectory formation refers to the principles and laws that invariantly govern the selection, planning, and generation processes of multi-joint movements, as well as to the factors that dictate their kinematics, namely geometrical and temporal features [107]. The sign language movement trajectory swerves to some extent, due to the action speed, and arm length of the user; even for the same user, the psychological changes resulted in inconsistent implementation speed of sign language movement [108]. Movement trajectory recognition is the key link of sign language translation research, which influence directly on the accuracy of sign language translation, in which the same signs match with variant movement trajectory predominantly refer to two variant meanings, that is, illustrating different sign language [109]. On the other hand, occlusion means that some fingers or parts of the hand would be covered (not in view of the camera) or hidden by other parts of the scene, so the sign cannot be detected accurately [110]. The occlusion may appear in various parts including hand/hand, and hand/face depending on the movement and the captured scene. The occlusion has a great effect on the segmentation procedure especially skin segmentation techniques [111]. Table 4 summarizes the most important related DL works that handle these types of problems in sign language recognition.

Detecting the signer hand in a still image or tracking it in a video stream is challenging and affected by many factors discussed earlier in the preprocessing phase such as environment, movement, hand shape, and occlusion. Hence, the careful choice of an appropriate segmentation technique is of utmost importance, as it profoundly influences the recognition of sign language and the work of the subsequent phases (feature extraction and classification). The hand segmentation identifies the beginning and end of each sign. This is necessary for accurate recognition and understanding of the signer's message. Through the process of segmenting the sign language input, the recognition system can concentrate on discerning individual signs and their respective meanings, thereby avoiding the interpretation of the entire continuous signing stream as a single sign. In addition to enhancing recognition accuracy, segmentation contributes to system efficiency and speed. By dividing the input into distinct signs, the system can process each sign independently, reducing computational complexity and improving response time. Furthermore, segmentation facilitates advancements in sign language recognition technology by enabling the creation of sign language corpora annotated with information about individual signs. Such resources are valuable for training and evaluating sign language recognition systems and conducting linguistic research on sign language structure and syntax. Various segmentation techniques are employed, including Background subtraction [112], Skin color detection [113], Template matching [114], Optical flow [115], and Machine learning [116]. Table 5 presents DL for sign language recognition-related works that focus on addressing the segmentation and tracking challenges to achieve optimal system performance.

The feature extraction goal is to capture the most essential information about the sign language gestures while removing any redundant or irrelevant information that may be present in the input data. The process of feature extraction offers numerous advantages in sign language recognition. It enhances accuracy by effectively representing the distinctive characteristics of each sign and gesture, thereby facilitating the system's ability to differentiate between them. Moreover, feature extraction reduces both processing time and computational complexity, as the extracted features are typically represented in a more compact and informative manner compared to raw input data. Additionally, feature extraction confers robustness against noise and variability, as features can be designed to be invariant to specific types of variations, such as changes in lighting conditions or background clutter [117,118]. This enables the recognition system to maintain its performance even in challenging and diverse environments. Table 6 shows related DL works for sign language recognition that focus on solving the problem of features extraction.

Classification is the final phase of any sign language recognition system and used before transferring the sign language into another form of data whether text or sound. In general, a particular sign is recognized by comparing it with the trained dataset, in which it categorizes the data into respective classes, depending on the feature vector obtained. Moreover, the system can calculate the probability associated with each class, allowing the data to be categorized under the respective class based on probability values. Overall, the classification conditions for sign language using DL involve selecting appropriate data representation, feature extraction techniques, classification algorithms, evaluation metrics, and ensuring sufficient and diverse training data. These factors collectively contribute to the accuracy and effectiveness of the sign language classification system. However, it may have some kinds of problems such as overfitting. In the realm of DL, overfitting occurs when a neural network model becomes too specialized in learning from the training data to the extent that it fails to generalize effectively to new, and unseen data. In other words, the model "memorizes" the training examples instead of learning the underlying patterns or relationships. When a DL model overfits, it performs very well on the training data but struggles to accurately predict or classify new instances that it has not encountered during training [119]. Various causes and indicators of overfitting exist, including a high model complexity with numerous parameters, insufficient training data, lack of regularization, excessive training epochs, and reliance on training data for evaluation [120]. To mitigate overfitting in deep models, several effective techniques can be employed. These include regularization methods [121], the incorporation of dropout layers [122], early stopping criteria [123], data augmentation strategies [124], and increasing training data [125]. These techniques can help to enhance model generalization and prevent the adverse effects of overfitting. Table 7 summarizes some related work of sign language recognition systems using DL that focuses on solving the problem of overfitting.

Another critical issue that must be considered when designing a deep model for sign language recognition is the generalization, which refers to the capability of a model to operate accurately on unseen data that is distinct from the training one. The model demonstrates a high degree of generalization ability by consistently achieving impressive performance across a wide range of diverse and distinct datasets [126]. Having consistent results across different datasets is an important characteristic for a model to be considered robust and reliable, which demonstrates that it can be applied effectively to various real-world scenarios. The datasets can have different characteristics, biases, or noise levels. Therefore, it is crucial to carefully evaluate and validate the model's performance on each specific dataset to ensure its reliability and generalization ability [127]. Table 8, presents relevant works in sign language recognition using DL, focusing on the model's generalization ability by evaluating its performance on diverse datasets.

The choice of DL layers significantly influences the classification model's performance, as it determines the model's architecture and its ability to learn and represent intricate patterns in the input data. Selecting the right layers involves a comprehensive understanding of the data's characteristics, problem complexity, and available resources for training and inference. Often, it necessitates experimentation, tuning, and domain expertise to discover the optimal combination of layers that maximizes classification performance for a particular task [128]. In sign language recognition. numerous authors have designed and utilized deep models to achieve desired performance levels, as depicted in Table 9.

In real-world classification scenarios using DL, time and delay is a principal factor to consider. It is important to strike a balance between achieving accurate classification results and minimizing the time required. The specific requirements and constraints of the application, such as the desired response time or the available computational resources, should be considered when designing and deploying DL models. As a result, one of the major requirements that make the recognition system of sign language efficient is the recognition time. Table 10 illustrates the related DL works for sign language recognition that focus on improving the recognition time.

Table 10: Related works on SLR using DL that aim to minimize the required time.