A Baseline for Violence Behavior Detection in Complex Surveillance Scenarios

Behavior detection datasets typically contain data collected from sources such as videos, sensors, etc. And are used to train and test algorithms for recognizing and analyzing human behavior. The UCF101 [1] datasets is among the biggest datasets of human behavior that are currently accessible, containing 101 action categories, almost 13,000 video snippets, totaling 27 hours of footage. Real user-uploaded films with crowded backdrops and camera motions make up the database. HMDB with Joint Annotation (JHMDB) [2] datasets A subset of the Human Metabolome Database (HMDB) [3] datasets contains 21 action categories, each involving the movement of a single character. The dataset was annotated with 2D joint model, providing information on the character's pose, optical flow, and segmentation for analyzing action recognition algorithms. The Kinetics [4] datasets is a human action video datasets introduced by DeepMind that contains 400 human action categories, each with 400 video snippets, each lasting roughly 10 seconds, from different YouTube videos. The dataset covers a wide range of action categories, including human-object interaction and human-human interaction.

Before the popularization of deep learning techniques, researchers used traditional features to process image information. The technique mostly included manually removing characteristics from video frames, which were then fed into support vector machines and decision trees for further behavioral analysis and identification. Xu [5] et al. suggested a technique for detecting violent videos that uses sparse coding and MoSIFT characteristics. Initially, the low-level description of the video is extracted using the MoSIFT algorithm, then feature selection is performed by Kernel Density Estimation (KDE) to eliminate noise, and finally the selected MoSIFT features are further processed using a sparse coding scheme to obtain highly discriminative video features. Febin [6] proposed a new descriptor Motion Boundary SIFT (MoBSIFT) to more effectively identify the characteristics of violent actions in the video. This module is able to filter out the random motions in the nonviolent behaviors, and represent and classify the violent videos by sparse coding technique, which has high accuracy and robustness in detecting violent behaviors.

By receiving the hidden state of the preceding moment, a recurrent neural network (RNN) may model the frames in a movie as an ordered sequence, which affects the state of the next moment, and the extracted temporal features are able to express human behavior. With networks like Long Short-Term Memory (LSTM), this behavior detection technique first extracts spatial data from the ordered sequence of frames, and then it goes on to extract temporal features from the video. Sudhakaran [7] proposed ConvLSTM, which aggregates frame-level violent behavioral features in the video by capturing the spatiotemporal features and captures the differences between consecutive frames by computing the motion changes, which reduces the amount of data to be processed. Liang [8] et al. used GhostNet and ConvLSTM to construct a long-term recurrent convolutional network and introduced a multiple attention mechanism in the video preprocessing stage to enhance the attention to the key information in the video, which improves the ability of detecting violent behavior in the video.

Multi-stream neural networks usually have many branches, before employing a classifier to identify behaviors, each branch independently extracts many feature streams from a large number of samples and aggregates the extracted features. Feichtenhofer [9] et al. designed a SlowFast network based on frame rate speed. The network contains two paths, Slow path and Fast path, to extract spatial semantic information and motion information at lower and higher frame rates, respectively, to enhance behavior detection. Next, Okan [10] proposed a multi-modal parallel module You Only Watch Once (YOWO) based on a dual channel structure. The network has two branches: one uses 2D-CNN to extract the spatial properties of key frames, while the other uses 3D-CNN to extract the spatio-temporal features of the video segment made up of earlier frames, and finally, fuses the features using channel fusion and the attention mechanism to perform frame-level detection for behavioral Localization of actions. Li [11] et al. suggested a novel technique for detecting violence based on a multi-stream detection model, which combines three distinct streams—a temporal stream, a local spatial stream, and an attention-based spatial RGB stream—to improve the performance of violent behavior recognition in videos. Islam [12] et al. suggested an effective dual-stream deep learning architecture using pre-trained MobileNet and LSTM (SepConvLSTM), in which one stream manages frame background suppression and the other handles frame differences between neighbors. In order to provide discriminative features that aid in differentiating between violent and nonviolent activities, a straightforward input preprocessing technique highlights moving objects in the frames while suppressing the nonmoving background and recording the inter-frame actions.

Conventional 2D convolutional neural networks that have been trained on single-frame images are unable to reflect the correlation between consecutive frames, while 3D convolutional networks are able to directly extract frames from the video, and then fed into 3D CNNs to extract the spatio-temporal features in the frame sequences, the network learns the characterization of the behaviors in the video after multilayered convolutional and pooling operations, and accurately detects the behaviors in the video, and it is currently an important research direction. Carreira [13] based on the Inception network and extended it from 2D to 3D, proposed the network Inflated 3D ConvNet (I3D) which is able to process video data for behavioral detection. Direkoglu [14] computes optical flow vectors for every frame to produce a motion quantum image (MII), It is then used to train a Convolutional Neural Network (CNN) to identify abnormal behavioral events in a crowd. The proposed MII is mainly based on the optical flow magnitude and angular difference calculated from the optical flow vectors in consecutive frames, which helps to distinguish between normal and abnormal behavior. Dong [15] et al. suggested the attentional residual 3D network (AR3D) and the residual 3D network (R3D), which were model ed by upgrading the current 3D CNNs by adding the residual structure and attention mechanism, The behavior detection performance of the model has been improved in different degrees. Li [16] et al. establish a 3D-DenseNet dense connectivity model, extract spatio-temporal features using 3D-DenseNet algorithm, redistribute the weights of each feature using the Squeeze-and-Excitation Networks (SENet) channel attention model, and then use the transition layer sampling, and then pass the outcomes to the fully connected layer using the global average pooling technique to finish the violence detection task. XU [17] et al. proposed the SR3D algorithm, which adds a BN layer before the 3D convolutional operation and presents the ReLu activation mechanism to enhance the network's learning capabilities while, extends the SE attention mechanism to 3D by introducing it into the 3D convolutional model and boosts the weights of the important channels, which improves the ability to detect the human behaviors in the video in the network.

The temporal fusion attention model(TFAM) [19] is an attention mechanism module for improved video object detection, which improves object representation by combining multi-frame and single-frame attention modules and dualframe fusion modules, but it has too much computation and weak generalization ability, which is not conducive to the application of violent behavior detection. Therefore, Channel Fusion and Attention Mechanism (CFAM) is introduced in this paper to effectively integrate the temporal features obtained by I3D network with the spatial features obtained by CSPDarkNet-Tiny network to record the inter-channel dependencies. Fig.2(e) displays the CFAM model's structure diagram.

The following is how feature fusion specifically works: firstly, the feature maps obtained from the first two networks are spliced to obtain the feature map A ∈R^{(C1+C2)×H×W}, then the correlation between the feature maps is captured using the local receptive fields of the convolutional layers, and the correlation feature map B ∈R^c×H×W is produced by passing the feature map A through two convolutional layers. Since direct correlation calculation would make the computation complicated, a reshaping operation is performed on B to obtain a reshaped feature map F. The elements of each channel in the feature map are converted into one-dimensional vectors to simplify the computation. The expression is as follows: 3B∈RC×H×W→ vectorization F∈RC×H×WB \in {R^{C \times H \times W}}\buildrel {{\rm{ vectorization }}} \over \longrightarrow F \in {R^{C \times H \times W}}

First, the resulting feature map F is dot-producted with its transposed feature map F^T to obtain a covariance matrix G ∈R^C×N, where N=H×W. This matrix reveals the correlation between different features. Its expression is as follows: 4G=F×FTG = F \times {F^T} 5Gi,j=∑k=1NFik×Fjk{G_{i,j}} = \sum\limits_{k = 1}^N {{F_{ik}}} \times {F_{jk}}

Where G_i,j represents the inner product between the feature map F and F^T. After that, the resulting matrix G is subjected to softmax operation to generate the channel attention feature map M ∈R^C×C. The softmax function is able to transform the values between the range of 0-1, which represents the attention weight of each position. the expression of M feature map is as follows: 6Mi,j=eGij∑j=1CeGij{M_{i,j}} = {{{e^{{G_{ij}}}}} \over {\sum\limits_{j = 1}^C {{e^{{G_{ij}}}}} }}

In order for the attention map M to have an effect on the original feature map, the matrix F′ is obtained by dot-product multiplication of M with the reshaping matrix F, which makes the features of the parts with high weights more prominent. Then F′ is reshaped to F″∈R^C×H×W× of the same size as B. 7F′=M×F{F^\prime } = M \times F

To alleviate the gradient vanishing problem and accelerate the model convergence, F″ is multiplied with the hyperparameter α and superimposed with the feature map B using the expression in (8) to get the feature map C ∈R^C×H×W, the final spatiotemporal feature map D ∈R^C×H×W with theattention weights is obtained by consecutively applying two convolutions to the resultant feature map C. 8C=α×F′′+BC = \alpha \times {F^{\prime \prime }} + B

The loss function proposed in this paper contains three components: classification

prediction loss L_cls, localization loss L_rect, and confidence loss L_obj.

The classification prediction loss formula is as follows: 9yi=sigmoid(xi)=exi∑n=1Nexn{y_i} = {\mathop{\rm sigmoid}\nolimits} \left( {{x_i}} \right) = {{{e^{{x_i}}}} \over {\sum\limits_{n = 1}^N {{e^{{x_n}}}} }} 10Lcls(y,yi)=−1N∑n=1NLBCEcls{L_{cls}}\left( {y,{y_i}} \right) = - {1 \over N}\sum\limits_{n = 1}^N {{L_{BC{E_{cls}}}}} 11LBCEcls=y×log(yi)+(1−y)×log(1−yi){L_{BC{E_{cls}}}} = y \times \log \left( {{y_i}} \right) + (1 - y) \times \log \left( {1 - {y_i}} \right) where x_i is the category's projected value and N is the total number of categories in the datasets, y_i is the current category probability, and y is the true value of the current category.

The localization loss formula is as follows: 12v=4π(arctanwgthgt−arctanwh)2v = {4 \over \pi }{\left( {\arctan {{{w^{gt}}} \over {{h^{gt}}}} - \arctan {w \over h}} \right)^2} 13α=v1−IoU(B,Bgt)+v\alpha = {v \over {1 - {\mathop{\rm IoU}\nolimits} \left( {B,{B_{gt}}} \right) + v}} 14Lrect(B,Bgt)=CIoU(B,Bgt)−p2(B,Bgt)c2−α{L_{rect}}\left( {B,{B_{gt}}} \right) = {\mathop{\rm CIoU}\nolimits} \left( {B,{B_{gt}}} \right) - {{{p^2}(B,Bgt)} \over {{c^2}}} - \alpha

Where w^gt as well as h^gt are the target real frame's width and height, the target predicted frame's width and height, v, which is the value of the projected frame normalized by extrapolating the width-to-height ratio of the predicted frame to the actual frame, and p², which is the distance between the predicted frame's centroid and the real frame's centroid, where α represents the balance between the loss resulting from the measurement of aspect ratio and the loss due to IoU. The confidence loss is publicized as follows: 15Lobj(C,Ci)=−1N∑n=1NLBCEobj{L_{obj}}\left( {C,{C_i}} \right) = - {1 \over N}\sum\limits_{n = 1}^N {{L_{BC{E_{obj}}}}} 16LBCEobj=C×log(Ci)+(1−C)×log(1−Ci){L_{BC{E_{obj}}}} = C \times \log \left( {{C_i}} \right) + (1 - C) \times \log \left( {1 - {C_i}} \right) where C is the current grid region's confidence, C_i is the expected value of confidence, and N is the number of feature points.

The three loss functions above are integrated to get the total loss function with the following formula: 17L=α1×Lcls+α2×Lrect +α3×LobjL = {\alpha _1} \times {L_{cls}} + {\alpha _2} \times {L_{{\rm{rect }}}} + {\alpha _3} \times {L_{obj}}

To ensure that the weights of the various loss terms are balanced, the hyperparameters α₁, α₂ and α₃ are set. where α₁ has a value of 0.4, α₂ has a value of 0.3 and α₃ has a value of 0.3.

The Kinectics datasets are used to train the model suggested in this research, and the custom datasets VioData are used to refine it.

In order to be able to enrich the training set and make the model better acquire the effective features in the video frames, three data enhancement operations are adopted in this paper, including horizontal flipping, random scaling, and color enhancement. The data enhancement operations expand the datasets, reduce overfitting, enhance the generalization ability of the model, and improve the robustness of the model.

The training settings are displayed in Table I below

Configure the model parameters to be saved once per ten iterations while the model is being trained, and save the output of the training loss and validation loss once at the completion of each epoch. When the loss iteration reaches 180 rounds, the training loss is still decreasing, but the loss of the validation set starts to rise, indicating that the model has been overfitted, so the model parameters at the end of the 180th epoch are saved as the optimal parameters.

The effect of the network model for violence detection on the VioData datasets is visualized as shown in Fig.3, where a video of a violent act is subjected to model inference to obtain the location of the violent act and its category, proving the effectiveness of the VioData datasets.

To compare with other violence detection techniques and show the efficacy of the suggested enhanced modules in the suggested violence detection model, we conducted numerous tests in this work. The experiments are conducted on three datasets (UCF101-24, JHMDB, and VioData).