Long-term Target Tracking Based on Template Updating and Redetection

In the state judgment module, the geometric features, appearance features and discrimination features are integrated according to the time sequence information, and the sequence matrix is input into the three-level cascade gated cycle unit. Ultimately, the two fully connected layers are employed to evaluate the reliability of the current tracking state, specifically determining whether the template should be updated in the present frame. The state judgment module mainly consists of two parts: information extraction and state awareness network.

a)

Information extraction

In the basic local tracker part, the geometric features, appearance features and discrimination features of the local tracker in the current frame are mined, and then the sequence matrix is formed by combining the timing information in the previous frame within a given period of time, which is used as the input information of the state-aware network.

Geometric features that describe the location and size of a target. The target tracking algorithm SiamRPN will output a four-dimensional vector every frame, which can be used to calculate the position information of the boundary box. In the t frame, the bounding box ^{b_t = [x_t,y_t, w_t, h_t]} obtained from the tracker is used as the tracking result, where ^(x_t,y_t) represents the top-left corner coordinates of the target and ^(w_t,h_t) represent the target's width and height, respectively.

As can be seen from the coordinate information of the bounding box, it can only provide the geometric position data of the target being tracked in the frame at this moment. Nevertheless, in the target tracking task, it is usually necessary to model the motion state of the target. Since the position, shape and size of the target fluctuate between successive frames, the motion state of the target can be estimated by comparing the boundary frame information between successive frames, and then the speed and acceleration of the target can be obtained. It is easier to capture the motion mode of the target and improv e the robustness and accuracy of the tracker by describing ^b_t in the upper left corner and upper right corner (xt1,yt1,xt2,yt2)\left( {x_t^1,y_t^1,x_t^2,y_t^2} \right)

Discriminant features, used to differentiate the target from surrounding background information. The SiamRPN algorithm finally outputs a feature response graph ^R_t whose maximum response value can be used to represent the confidence score of the bounding box ^b_t as shown in formula (1).1Ft=max(Rt){F_t} = \max \left( {{R_t}} \right)

Figure3. shows the confidence scores in the tracking process. The results show that the confidence scores of frames 89 and 261 are unstable, so the quality assessment value is used for auxiliary discrimination and the discrimination information in the response graph is thoroughly mined. The calculation formula is shown in Equation (2) 2A=μ1| Fmax−mean(Fmax) |mean(Fmax)+μ2|apce−mean(apce)|mean(apce)A = {\mu _1}{{\left| {{F_{\max }} - mean\left( {{F_{\max }}} \right)} \right|} \over {mean\left( {{F_{\max }}} \right)}} + {\mu _2}{{|apce - mean(apce)|} \over {mean(apce)}}

Figure 3.

Confidence score chart

Among them, A represents the quality evaluation value, ^F_max represents the highest response data on the response map, ^apce represents the average peak correlation quantity, The formula for its calculation is provided in the following equation (2.3) 3apce =| Fmax−Fmin |2mean(∑x,y(Fx,y−Fmin)2){\rm{apce }} = {{{{\left| {{F_{\max }} - {F_{\min }}} \right|}^2}} \over {mean\left( {\sum\nolimits_{x,y} {{{\left( {{F_{x,y}} - {F_{\min }}} \right)}^2}} } \right)}}

Among them, F_min represents the minimum value of the response graph, F(x,y) represents the response value associated with the coordinates (x,y).

Appearance feature, which is utilized to indicate the similarity between the appearance of the target template and the current frame target. Using noise samples for template updating usually makes the response graph insensitive to appearance changes, so the method of template matching can be used as an important supplement and similarity score can be defined, as shown in formula (4).4St=cos(It,I0){S_t} = \cos \left( {{I_t},{I_0}} \right) Where ^I₀ represents the initial template feature and ^I_t represents the tracking result of the current frame. Timing information: geometric features, discriminant features and appearance features are combined into column vectors ^X_t, as shown in formula (5) 5Xt=[ xt−ts+1,…,xt−1,xt ]{X_t} = \left[ {{x_{t - {t_s} + 1}}, \ldots,{x_{t - 1}},{x_t}} \right] Where ^t_s is the time step utilized to balance historical and current information, so that the temporal information X_t includes both the motion and appearance changes of the target. The temporal information is then fed into the state-aware network to judge the target state and decide whether to update the target template information.

Most trackers use linear interpolation or a straightforward average weighting strategy, as illustrated in formula (10), to update the template.10T˜i=(1−∂)Ti−1+∂Ti{{\tilde T}_i} = (1 - \partial ){T_{i - 1}} + \partial {T_i} Where ⁱ denotes the number of frames of the video sequence, ^T_i represents the new template derived from the frame at this moment, ^T̃_i signifies the cumulative template, ^η is the update rate, set to a fixed value of 0.01.

However, there are two problems in using the simple weighted average strategy: (1) The update rate is a constant value, leading to a somewhat simplistic update mechanism; (2) No initial template frame information is used, which easily leads to tracking drift. Based on this, this excerpt uses a generic function ^φ derived from adaptive update template features, where the function ^φ is implemented using the UpdateNet network model, which is capable of learning from extensive datasets. The new template information is derived by integrating the initial template frame ^T₀, the previously accumulated template frame ^T̃_i–1, and the template of the target position estimated by the frame at this moment ^T_i,as shown in equation (11).11T˜i=φ(T0,T˜i−1,Ti)+T0{{\tilde T}_i} = \varphi \left( {{T_0},{{\tilde T}_{i - 1}},{T_i}} \right) + {T_0}

Figure2. Gray dashed line box describes the specific structure and overall framework of the template update module, using the feature extraction network proposed in Chapter 3 to extract the feature information of the target from the image. During the course of template update, the information of the first frame is real and reliable, so the template features ^T₀ can be extracted at the target boundary box position given in the initial frame. To get ^T_i, we first need ^T̃_i–1 to ascertain determine the position of the target of the frame at this moment, and then use the feature extractor to extract the feature information ^T_i of the current frame within this region. The input to UpdateNet is a triplet of ^T₀ (leftmostfeature map, initial template feature), ^T̃_i–1 (dashed line connection, previous frame accumul ated template feature) and ^T_i (rightmost feature map, current frame template feature). In the first frame, since there are no previous frames, so ^T̃_i–1, ^T_i and ^T₀ is initialized. Of the three inputs to UpdateNet, Only the information from the initial frame is true and reliable, while the information in subsequent frames is predicted by the tracking algorithm, the rest is predicted by the tracking algorithm, so ^T₀ can be used as a reliable signal to guide the model update. Based on this, skip connections are used to combine the initial template feature f0 with the output of UpdateNet, to achieve the most accurate template features.

Figure 4. shows the Jogging effect of SiamRPN algorithm in OTB2015 dataset. The top graph represents the confidence score you get when you track a video sequence with SiamRPN. The following picture illustrates the tracking results of SiamRPN algorithm across various frames. The red bounding box indicates the tracking output of SiamRPN algorithm, while the black bounding box indicates the actual location of the target. From the figure, it can be seen that among the initial frames of the sequence, two girls are Jogging into the field of vision, and the girl wearing black pants is the target being tracked. The target object is always moving from the initial frame to the 39th frame, and SiamRPN algorithm keeps tracking it accurately. However, between the 39th frame and the 73th frame, a telegraph pole appears, and the target object is completely covered and disappears into the field of view during this period. simultaneously, the confidence score decreases sharply. When the target is lost because of occlusion and other factors, the confidence score will also be reduced, so the confidence score can be used to judge the disappearance of the target. However, due to the integration of the adaptive template update mechanism in the tracker, the confidence score does not decrease significantly. Therefore, this section uses the initial template features obtained in the first frame to make further judgment on the basis of the confidence score obtained by the algorit.

Figure 4.

SiamRPN tracking results

Firstly, the Euclidean distance between the target initial template z and the tracking result x predicted by the algorithm is calculated as the similarity, and the formula is shown in equation (12) 12D=∥z−x∥2D = \parallel {\rm{z}} - {\rm{x}}{\parallel _2}

Then the similarity score D is combined with the confidence score^s to judge the disappearance of the target, the formula is presented as follows.13r=mean(D+s)r = mean(D + s)

Defined r as evaluation score, the evaluation score of consecutive video frames is utilized to assess the disappearance of the target.

During the tracking process, the target may be lost only a few frames or the algorithm itself has calculation errors, so the delay judgment is also needed to ensure the compatibility of the algorithm and the stability of the tracking. The detailed flowchart is presented in Figure 5. In general, when the evaluation score ^r falls below the specified threshold ^t, the cumulative number of failures is set to 0 when the evaluation score is greater than the given threshold. When the evaluation score ^t is less than and the number of failures ^f or better than the loss threshold ^c, the target is considered lost and the number ^f of failures is set to 0.

Figure 5.

Flow chart of target loss judgment mechanism

When the local tracker identifies that the target has been lost through the target loss judgment mechanism, it must initiate a global search to redetect the target within the subsequent frame's image area and identify the most likely location of the tracked target. Consequently, the redetection algorithm must swiftly scan the entire image and accurately pinpoint the target's location without relying on historical frame information. Based on this, a redetection algorithm based on template matching is proposed.

As shown in Figure 6, the redetection algorithm based on template matching mainly is primarily composed of three components, namely, feature extraction module, candidate frame extraction module and precise positioning module. To enhance the redetection algorithm's ability to differentiate between the background and the target amidst similar interference, a cross-query loss function is employed to optimize the algorithm.

Figure 6.

Template-based re-detection algorithm

The template update module is trained using a three-stage training method. First, the training of ~ the first stage obtained he cumulative template ^T̃_i by a simple average weighting strategy. The calculation formula is shown in equation (18).18T˜i=(1−η)·T˜i−1+ηTi{{\tilde T}_i} = (1 - \eta )\cdot{{\tilde T}_{i - 1}} + \eta {T_i}

Among them, ^T_i denotes a new template calculated in the first stage of training using the current frame, parameter^η=0.01. Secondly, during the second and third stages of training, the cumulative template is derived by updating the module with the adaptive template proposed in this chapter, and the weight data is obtained by using the parameters trained in the previous stage. The LaSOT dataset is a seminal resource in the realm of long-term target tracking, characterized by its complex and varied video sequences. However, the template update module only contains two layers of convolutional neural networks, so the update module with 20 video sequence training templates is sufficient to meet the requirements.

In the first stage of training, set the starting learning rate as ¹⁰⁻⁶, and with the weights initialized randomly. After each epoch is trained, the learning rate will be logarithmically decayed; In the second stage of training, the parameters of the optimal model obtained from the first stage are utilized to initialize the weights, and the learning rate is attenuated from ¹⁰⁻⁷,¹⁰⁻⁸,¹⁰⁻⁹ to ¹⁰⁻⁹,¹⁰⁻¹⁰,¹⁰⁻¹¹; The third stage imports the optimal model from the second stage, and the learning rate is attenuated from ¹⁰⁻⁸,¹⁰⁻⁹,¹⁰⁻¹⁰to¹⁰⁻⁹,¹⁰⁻¹⁰,¹⁰⁻¹¹. The stochastic gradient descent algorithm is selected to train the template update module, in which the weight attenuation and momentum are set to 0.0005 and 0.9 respectively.

The COCO data set is used to train the redetection module, and data enhancement techniques are used to generate more image samples [12] in the pre-processing stage. The model was trained 50 times in total. The average loss of candidate region extraction module and precision positioning module was used as the total loss function, and the SGD method with momentum of 0.9 was used to optimize [13] the network model.

The size of the time step ^t_s of the status judgment module is crucial to the tracking results, ^t_s including the information of the present and the historical frame, its value determines the richness of the obtained timing information. The success rate and precision of the short-term local tracker based on the state judgment module were calculated on the OTB2015 dataset for different values of, and the experimental results are shown in Figure7. Among them, the horizontal coordinate ^t_s represents the size, the left and right vertical coordinates represent the success rate and accuracy rate, Z are represented by red and green curves respectively. As illustrated in the figure, when ^{t_s = 25} both the success rate and accuracy rate of the local tracker on the OTB2015 dataset have achieved the maximum value, and the tracking performance is effective at this time.

Figure 7.

Different t_s corresponding success rates and accuracy

In the experimental verification of the target loss judgment module, use ^μ as the evaluation score threshold to judge whether to switch from local tracking to global tracking, use ^δ as the number of lost, and then test it on the OTB2015 dataset, use the sum of accuracy and success rate as the judgment standard, and select the appropriate threshold. Figure 8 is the performance score chart under different values ^δ and values ^μ, where Y axis is the evaluation score threshold, Z axis is the sum of the benchmark success rate and accuracy rate, and X axis represents the number of lost times. In the experiment, the value ^μ is 1 to 12, ^μ from 0.08 to 0.2, with the constant change of ^{μ δ}, it can be seen from the figure that when ^{δ = 10}, ^{μ = 0.13}, the tracking performance is the best.

Figure 8.

Results of parameter optimization

Figure 9.

Diagram of LaSOT experimental results