Robust single target tracking using determinantal point process observations

Visual tracking is a challenging computer vision task with applications in human-computer interaction, video surveillance and crowd monitoring among others. Modern visual tracking systems may use complex object detection schemes for estimating the current state of a target in any particular video frame. However, this approach does not fully exploit the temporal structure of the estimation problem. Visual tracking can be also thought of as a dynamic model with observed features and latent states representing the position/velocity of an object (Maggio and Cavallaro, 2011). In this context, the generative model for visual tracking requires not only the correct specification of the model and its parameters but also the ability to capture the variations of the system (Wang et al., 2015).

The Bernoulli filter is a powerful algorithm that allows objects to appear and disappear, using extracted features from the image as observations (Vo et al., 2010). Similar approaches for visual tracking have been proposed (also known in the literature as Track-Before-Detect). Nevertheless, these methods rely on unreliable background subtraction operations or the likelihood function being in a separable form (Hoseinnezhad et al., 2012, 2013).

Current state-of-the-art trackers are based on either correlation filters (Bolme et al., 2010), deformable parts models (Hare et al., 2016) or convolutional neural networks (Li et al., 2018). These trackers learn a discriminative model from a single frame and then update the model using new frames. Furthermore, tracking performance can be increased when using more discriminative features such as HOG (Henriques et al., 2015; Solis Montero et al., 2015; Xu et al., 2019). On the other hand, even when Bernoulli filters have demonstrated being useful models for tracking in complex scenarios, it is still hard to rely on such features for increasing their performance.

A point process is a random pattern of points in a possibly multi-dimensional space (Kingman, 1993). A simple point process can be defined in one dimension, which is usually times and can be used to describe the random times where the events can occur with no coincident points.

Let X_k = {x} be a Bernoulli point process and Z_k = {z₁, z₂,…, z_m} a DPP observed at frame K. The result from superposition, translation and thinning transformations is also a Bernoulli point process X_k ∼ p(X_k|X_k−1) (Ristic et al., 2013). The predicted point process can be written as the linear superposition of a π_s thinned point process with Markov translation f(x|x′) and a π_b Bernoulli birth process. In order to measure the quality of the observations, we must introduce a random variable L such that p(L|Z)  ∝  det(L(Z)), where L(Z) is a positive definite kernel matrix that depends on the observed features Z. The L(Z) kernel can be written as a Gram matrix: (15) L i j ( Z ) = g x ( z i ) ϕ ( z i ) T ϕ ( z j ) g x ( z j ) , (15) (16) = g x ( z i ) S i j ( Z ) g x ( z j ) . (16)

The function g_x(z_i) = ∑_c p(z_i|c)p(c|x) is used to model the quality of the item z_i and S(Z) the diversity of the set Z. If we let W be a subset of detections arising from the target (Reuter et al., 2013): (17) η ( W | { x } ) ≈ { ( 1 − π d ) if W = ∅ π d g x ( w 1 ) det ( S w 1 ) if | W | =1 | W | ! π d m ∏ i g x 2 ( w i ) det ( S W ) if | W | = m . (17)

The DPP Z can be treated as the union of two independent sets Z = C ∪ W, where C = {c₁, …, c_m} represents clutter. The clutter density becomes: (18) κ d ( C ) = | C | ! ∏ i f c 2 ( c i ) det ( S C ) . (18)

The likelihood function for the standard measurement model using determinantal observations becomes: (19) P ( Z | { x } ) = ∑ W ⊆ Z η ( W | { x } ) κ d ( Z \ W ) , (19) (20) p ( Z k | { x } ) = κ d ( Z k ) [ (1 − π d ) M k + ∑ Z ∈ Z k | Z | ! ( M k − | Z | ) ! M k ! π d | Z | ( 1 − π d ) M k − | Z | ∏ i [ g x ( z i ) f c ( z i ) ] 2 det ( S Z ) det ( S Z k \ Z ) det ( S Z k ) ] . (20)

Now, we want to derive the posterior distribution for Bernoulli point process given DPP observations: (21) p ( Z k ) = κ d ( Z k ) [ ( 1 − q k | k − 1 ) + q k | k − 1 ( 1 − π d ) M k + ∑ Z ∈ Z k Ξ k ∫ ∏ i g x 2 ( z i ) p k | k − 1 ( x ) d x ∏ j f c 2 ( z j ) ] , (21)where:Ξ k = | Z | ! ( M k − | Z | ) ! M k ! π d | Z | ( 1 − π d ) | Z | − M k det ( S Z ) det ( S Z k \ Z ) det ( S Z k ) .

The updated binomial point process can now be derived as follows: (22) q k | k = 1 − Δ ˘ k 1 − q k | k − 1 Δ ˘ k q k | k − 1 , (22)where: (23) Δ ˘ k = 1 − ( 1 − π d ) M k − ∑ z ∈ Z k Ξ k ∫ ∏ i p ( z i | x ) p k | k − 1 ( x ) d x ∏ j f c ( z j ) , (23)and: (24) p k | k ( x ) = ( 1 − π d ) M k + ∑ z ∈ Z k Ξ k ∫ ∏ i g x ( z i ) p k | k − 1 ( x ) d x ∏ j f c ( z j ) 1 − Δ ˘ k . (24)

In practice, it is difficult to store and compute the power set with all possible configurations of Z_k in the likelihood term (see Equation (20)). An approximation can be constructed by truncating the likelihood and focusing only on the more likely elements. Let Z_k^* = arg max_Z⊂ Zk η(Z|{x}) be a subset of Z_k whose elements are detections arising from the target. The likelihood becomes: (25) p ( Z k * | { x } ) = | Z k * | ! ∏ i g x 2 ( z i ) det ( Z k * ) . (25)

DPPs have been proposed in the literature as an alternative to other object refinement techniques such as non-maximum suppression (Lee et al., 2016). These methods operate over object proposals and eliminate redundant detections. For DPPs, mode finding can be tackled using the following greedy algorithm (Kulesza and Taskar, 2011):

Conversely, by using the truncated likelihood from Equation (25), the Sequential Monte Carlo algorithm for the Bernoulli filter can be used to estimate the single-target posterior (Ristic, 2013).

In order to demonstrate the advantages of the proposed model updating approach over other discriminative approaches, we evaluate the tracking results on six challenging video sequences from the Visual Object Challenge 2014 (VOT) data set

http://www.votchallenge.net/vot2014/

The parameter setting the Greedy Mode Finding algorithm is described in Table 2.

The sequence jogging is a challenging example containing full occlusions, rotations and background clutter. Figure 1 shows one frame of the sequence and the estimates using the proposed approach and other state-of-the-art methods.

Figure 1:

The Bernoulli DPP filter maintains a balance between the observed features and the quality of the observations (see Figure 1). The observation model uses a simple histogram comparison and no template update is performed, so the model is not robust to object deformation or rotation. Even that, as seen in Figure 1 the Bernoulli-DPP tracker achieves good performance in cases such as full occlusion where the other discriminative tracking methods fail. Performance is measured using widely used precision and success metrics

http://cvlab.hanyang.ac.kr/tracker_benchmark/benchmark_v10.html

The precision metric describes the percentage of frames whose center location error is below a given threshold. Table 3 shows the overall precision metric averaged over all sequences on five different runs for each one of the algorithms.

The success measure accounts for bounding box overlap. Table 4 shows the number of success frames whose overlap is above some threshold, averaged over the sequences on five different runs. Quantitative analysis shows improved performance for the proposed approach when compared to the discriminative trackers in six different video sequences.

Figure 2 shows the precision metric against the location error threshold for all of the six tested sequences. The red line indicates the best performing method among the four different algorithms. Since the bolt and jogging sequences have background clutters (the background near the target has similar appearance as the target), the proposed Bernoulli DPP tracker reduces redundant observations and improves precision.

Figure 2:

Figure 3 shows the ratio of the frames whose tracked box has more overlap with the ground-truth box than a threshold. The success metric can be associated with the tracker algorithm ability to maintain long-term tracks. Since the Bernoulli DPP filter accounts for missed detections, the proposed approach improves the area under the curve of the success metric in 67% of the tested sequences.

Figure 3:

In this paper, a novel algorithm for joint detection and tracking a single object in video has been presented. The proposed approach takes into account the detection score and the similarity of the observed features. Then, a Bayesian filter using a Bernoulli point process estimates the state of the target from a diverse subset of object proposals. Experimental evaluations show that the results are comparable to other state-of-the-art techniques for visual tracking in only 6 of the 25 sequences of the data set. In this paper, we only considered a simple observation model (distance to a reference LBP histogram), which might hinder the performance of this approach in the overall data set. This observation model is not robust to scale and rotation changes and no model updating strategies are considered in this paper. Nevertheless, our model is expected to increase its performance when using a more complex observation model (such as deep learning features), model updating and ensemble post-processing techniques for combining the output from different tracking schemes.