Deep Learning with 3D ResNets for Comprehensive Dual-Lane Speed Climbing Video Analysis

High-definition stationary cameras were used for recording, with a resolution of 1920×1080 pixels and a frame rate of 30 frames per second. This ensured sufficient video clarity to support motion analysis and climbing outcome classification.

Data was collected between March 4, 2024, and May 13, 2024, covering the spring training season.

All videos were recorded on speed climbing walls that met the standards set by the International Federation of Sport Climbing (IFSC). The dual-lane design adhered to uniform width, slope, and standardized climbing wall layouts (see Figure 2).

Figure 2.

Diagram of the official speed climb wall including numbered hand holds. Adapted from (Walltopia 2020). *Final button or ‘hold 21’ (Lau, 2021)

The dataset includes the following three dual-lane scenarios (see Figure 3):

Dual-lane, dual-climber scenario: Two athletes climb simultaneously, one on the left lane and the other on the right lane.

Figure 3.

Images of the three training situations for speed climbing, (1)Dual-Lane double climbing, both lanes are occupied, (2)Dual-Lane single climbing on the left lane, the left lane is occupied and the right lane is empty, (3)Dual-Lane single climbing on the right lane, the right lane is occupied and the left lane is empty.

As shown in Figure 4, the climbing state for each lane is categorized into the following three types:

Flash: The athlete successfully taps the top timing panel to complete the climb. The timer stops, and the top light turns green.

Figure 4.

Images of the Three States in Speed Climbing: Flash, Slip, and Fall, (1)Flash, the climber has tapped the timer to stop and the display light is green, (2)Slip, the climber’s left foot steps out of the air and slips (3)Fall, he climber’s feet are dangling in the air, falling downward slowly, and the timer continues to keep time and the display light is red.

Two professional coaches annotated the dataset, labeling the climbing results of both lanes for each video to create independent classification data. The label categories are as follows:

1: Flash

Excluding invalid scenarios where both lanes are empty, the dual-lane state combinations result in 15 unique labels (see Table 1). The labels are encoded as integers (0–14) to align with the model training requirements. Multiple rounds of review were conducted to ensure annotation accuracy and consistency.

The average video duration is 11.38 seconds, capturing the full sequence of the athletes’ climbing process.

The average file size is 12,163,891.33 bytes (approximately 11.6 MB), ensuring sufficient resolution for feature extraction.

The dataset includes 15 dual-lane state combinations, with detailed distributions shown in Table 1.

Frames were extracted at 30 frames per second, using multi-threaded extraction via OpenCV and Python’s ThreadPoolExecutor (Joshi, Escriva, & Godoy, 2016). This process ensured the capture of dynamic climbing details, enabling the model to learn temporal features effectively.

Frames were cropped from 1920 to 1600 pixels in height using OpenCV to focus on athletes' movements and the climbing wall, standardizing the format, reducing irrelevant background, lowering computational load, and improving training efficiency.

Frames were resized to 112×112 pixels and organized into temporal sequences with a shape of (30, 112, 112, 3), representing 30 time steps, spatial resolution, and RGB channels. These sequences were paired with labels and stored as NumPy arrays for efficient loading.

After preprocessing, the dataset included the processed results of 872 videos, totaling 10,937 frame sequences. The labels covered 15 states with codes ranging from 0 to 14, fully aligned with the model’s classification requirements. The total dataset size was approximately 11.4 GB.

The input to the model is temporal sequence data with the shape (30,112,112,3), where each sequence consists of 30 frames, each being a 112×112 pixel RGB image. The input layer is designed to capture dynamic features of the athlete’s climbing process, with a temporal resolution of 30 frames per second. This input dimension balances spatiotemporal resolution with computational resource requirements, optimized for the efficient handling of short temporal sequences by the 3D ResNet model.

The initial convolutional layer employs a 3D convolutional kernel of size 7×7×7 with a stride of (2,2,2), containing 64 filters. The 3D convolution simultaneously processes spatial and temporal dimensions, extracting low-level spatiotemporal features from the input video. The convolutional layer is followed by batch normalization (Batch Normalization) and a ReLU activation function to stabilize the training process and introduce nonlinearity, enhancing the model’s expressiveness. A MaxPooling3D layer is then added with a pooling kernel size of 3×3×3 and a stride of (2,2,2), which reduces dimensionality to decrease computational complexity while retaining critical features.

The core of the model consists of four groups of residual modules, each containing two to three residual units. The structure of each unit includes the following components:

Convolutional Layers: Each unit uses a 3×3×3 convolutional kernel. The first group of modules contains 64 filters, with subsequent groups increasing the filter count to 128, 256, and 512, respectively. The stride increases progressively from (1,1,1) to (2,2,2).

After passing through all residual modules, the feature maps are flattened into a one-dimensional vector and fed into fully connected layers designed as follows:

Global Feature Extraction: A Dense layer with 512 neurons and ReLU activation is used to extract global features.

The model has a total of 37,425,231 parameters, with 37,415,503 trainable parameters and 9,728 non-trainable parameters. As illustrated in Figure 5, the model leverages multi-level feature extraction and classification processes, efficiently capturing the spatiotemporal features of dual-lane speed climbing videos to classify complex actions accurately.

Figure 5.

Simplified diagram of the 3D ResNet model architecture designed for this study.

The initial learning rate was set to 0.0001. A ReduceLROnPlateau callback was applied to dynamically reduce the learning rate by a factor of 0.2 when the validation loss showed no significant improvement for two consecutive epochs. The minimum learning rate was set to 0.00001.

Adam incorporates a momentum mechanism for gradient descent, using default values of β₁ = 0.9 and β₂ = 0.999. These parameters smooth gradient updates and reduce the impact of local extrema on weight updates.

Compared to traditional optimization algorithms such as stochastic gradient descent (SGD), Adam demonstrates superior robustness in handling high-dimensional data and sparse gradients. This enables the model to converge more quickly to a global optimum.

The dataset was split into 80% training set and 20% testing set randomly.

A random seed of 42 was used to ensure reproducibility of the split.

Given that the dataset comprises a total of 10,937 samples, its size was sufficient to directly divide it into training and testing sets for model evaluation.

The validation set was not initially included for the following reasons:

Due to the long training cycles of the model, preliminary experiments focused on assessing generalization performance using the test set.

The 80% training and 20% testing split ensured diversity within both sets, and the size of the test set was sufficient for evaluating the model’s performance.

Due to the large scale of the dataset, the training process required significant computational resources. All training tasks were conducted on a high-performance computer with the following specifications:

Processor (CPU): 20-core Intel processor

To achieve efficient training and optimal classification performance, the following hyperparameters were tailored to the 3D ResNet architecture and dataset size:

Batch Size: Set to 8. A smaller batch size alleviates GPU memory pressure and helps capture more sample details in smaller datasets.

These configurations and mechanisms ensured efficient training, improved model performance, and minimized the risk of overfitting, laying a solid foundation for further experiments.

The model achieved outstanding F1 scores close to 0.99 for specific classes such as Class 12 (“L-Empty; R-Flash”), Class 13 (“L-Empty; R-Slip”), and Class 14 (“L-Empty; R-Fall”). These results highlight the model’s robustness in recognizing climbing states involving empty lanes. The distinctiveness of these categories, with one lane being empty and less dynamic, likely contributed to the model’s high accuracy.

Overall, the model maintained precision, recall, and F1 scores above 0.88 for most classes. This consistent performance reflects the model’s strong adaptability to diverse climbing scenarios, making it well-suited for real-world applications in speed climbing analysis.

The optimized architecture, with lightweight residual modules and reconfigured kernels, enabled efficient processing. With a sample processing time of approximately 100 milliseconds on an NVIDIA RTX 3090 GPU, the model shows potential for near real-time video analysis, which is critical for both training feedback and competition evaluation.

Class 0 (“L-Flash; R-Flash”) represents the largest category in the dataset, with 594 samples, making it a dominant class in the feature space. While the model performed well in recognizing Class 0, this dominance introduced misclassification challenges. Specifically, Class 1 (“L-Flash; R-Slip”), Class 2 (“L-Flash; R-Fall”), and Class 4 (“L-Slip; R-Flash”) had 8, 15, and 28 samples, respectively, misclassified as Class 0. These errors suggest overlapping features in shared “Flash” states and the model’s tendency to prioritize these dominant signals. Furthermore, reverse misclassifications occurred where 9 samples from Class 0 were classified as Class 1, 6 as Class 4, and 6 as Class 10 (“L-Fall; R-Fall”). This bidirectional misclassification highlights the difficulty in distinguishing subtle feature variations between “Flash” and other dynamic states like “Slip” and “Fall,” especially in categories with less distinctive features or lower sample representation.

The model struggled with Class 11, achieving a recall of only 0.79. This indicates difficulty in distinguishing this category from others, particularly Class 3 (“L-Flash; R-Empty”), with 4 misclassified samples. The shared “R-Empty” state likely caused feature overlap, complicating the classifier’s decision-making process. Additionally, the limited representation of Class 11 in the dataset (24 samples) constrained the model’s ability to generalize effectively.

The model’s parameters were reduced to 37 million, significantly lower than the original 3D ResNet (63 million) (Tran et al., 2015) and C3D (79 million) models (Hara et al., 2018). Simplified residual modules, reconfigured kernels, and constrained fully connected layers collectively enhanced computational efficiency. This lightweight architecture enables deployment on a range of hardware, including high-performance laptops and mobile GPUs (Foo, Gong, Fan, & Liu, 2023).

Hierarchical kernels (7×7×7 in initial layers, 3×3×3 in residual modules) were employed to effectively capture both global and local climbing features, addressing the complexities of speed climbing video analysis. This multi-scale feature extraction, supported by prior studies (Hara et al., 2018; Kong, Satarboroujeni, & Fu, 2016; Tran et al., 2015), improves the model’s ability to distinguish nuanced action categories and temporal dependencies.

With Table 3, the optimized 3D ResNet achieved a test accuracy of 92.78%, significantly outperforming 2D CNN (25.62%) and C3D (27.15%). Its loss convergence (0.57) further demonstrated stability and efficiency during training. The model processes each sample in approximately 100 milliseconds on an NVIDIA RTX 3090 GPU, highlighting its potential for near real-time applications in both training and competition settings.

The absence of a validation set limited intermediate evaluations during training. Future work should incorporate one to refine hyperparameter tuning (Ghosh et al., 2024).

The dataset, collected from a single venue and athlete group, limited adaptability. Broader data sources would enhance generalization to diverse environments (Krawczyk, 2016).