Object Localization Algorithm Based on Meta-Reinforcement Learning

As shown in Figure 2, this paper uses the reinforcement learning target localization model with joint action network and regression network as the task training model of the model. It mainly consists of three parts: feature extraction network, action network and regression network. The feature extraction network is the improved GAP-VGG16 network. The model matches the task to the feature space corresponding to the meta-parameter pool according to the feature values extracted by the feature extraction network, and stores the updated parameters after completing a batch of training. At the same time, the feature vector extracted by the feature is fused with the memory vector and sent to the action network. The action network is responsible for taking adjustment actions according to the current environment state, until the stop action is generated, the regression network is used for regression operation, and the final positioning result is output.

Figure 2.

For the few-shot object localization task, the model extracts a few samples from the task for feature extraction, and matches the meta parameters with the largest similarity from the meta parameter pool according to the feature value type extracted by the task as the initial parameters of the few-shot object localization task for retraining. The parameters of the retrained model are updated in the corresponding meta-parameter pool to retain the newly learned task memory. Through the method of this paper, the model has a certain ability to learn the task at the beginning, avoiding the agent to explore in a completely unfamiliar environment. The model regards each image input as a reinforcement learning environment, and selects the exploration action according to the fusion state of the input and historical exploration. The model gives feedback according to the pre-designed reward function to judge the quality of the model action selection, and updates the network parameters of the model through the process of circulation to improve the accuracy of target localization. The detailed design of the model states, actions, and rewards is as follows.

The process of human searching for the target is not only related to the current visual field, but also involves the memory of the past historical exploration in the brain. Human beings realize the accurate recognition of the target by combining the brain memory with the current visual field. The state S of this paper is the procedural simulation of this process, which is represented by a tuple s_t = (o_t, h_t) related to time t, which represents the fusion information of o_t and h_t, and the agent makes the next action selection according to this fusion state s_t.

The action taken by the Agent acts on the adjustment of the candidate box, which is divided into horizontal movement (left and right), vertical movement (up and down), scale transformation (horizontal expansion, horizontal reduction, vertical expansion, vertical reduction), and stop action). Each action is adjusted discretized according to the multiple of. Among them, the output termination action indicates that the target is in the field of view of the agent. The specific classification of actions is shown in Figure 3.

Figure 3.

The model selects the actions of the agent through the DQN network, and uses ε – greedy [15] strategy to make the agent explore new actions [16], so as to ensure that the agent takes the optimal action under long-term exploration. ε – greedy strategy is shown in (1). 1 π(a|s)←{ 1-ε+ε| A(s) |,ifargmaxaQ(s,a)ε| A(s) |,ifargmaxaQ(s,a)

Where s is the current state the Agent is in, and a is the action taken by the Agent based on the current state. A(s) is the set of actions that the Agent can choose at states, and |A(s)| denotes the number of actions that can be chosen. ε ∈ [0,1] Is the exploration factor, and π(a | s) is a policy, which represents the probability distribution of possible actions taken by the agent at a given state s. For the state at a given time in the policy, the agent selects the action corresponding to the output with the maximum probability to adjust the attention field.

The good or bad of the action taken by the agent can be intuitively seen through the reward function, and in this paper, the reward function is set by the IOU change of the attention field after the state change s after the agent takes the action. As shown in (2), where b is denoted as the visible area of the Agent and g is the real labeled area of the target object. 2 IOU(b,g)=area(b∩g)area(b∪g)

At each time step, after taking an action, the agent will obtain a new visual area. By calculating IOU between this area and the real area, the reward value of the agent’s state change after taking an action can be obtained, which is defined by the reward function shown in (3). 3 Ra(a,s→s′)=sign(IOU(b′,g)−IOU(b,g))

This function indicates that after the agent’s state has changed. If the value of IOU increases upward, it means that the agent has obtained positive feedback, and the model will store this state and action tuple (s, a, r, s′, b, g) as experience in the experience pool, which is used as a reference for the agent to explore the target position. On the contrary, if the IOU decreases after the state change, it indicates that the action is poor and negative feedback is obtained. For the determination of stop action, when the IOU value rises above 0.6 after the agent makes an action, it is determined that the target is in the field of view of the agent, and the stop action is taken, and the regression network is selected to take a smaller step to fine-tune the field of view frame. The reward function for the stop action is given in (4). 4 Rt(s→s′)={ +ηifIOU(b,g)≥τ−ηotherwise

In order to ensure the training efficiency, when the IOU of agent reaches 40 steps, it is determined that the exploration fails, and the regression network is not used for fine-tuning. (7) is used to update the parameters of the two networks.

The action network consists of two parallel fully connected networks with the same structure and different parameters. One produces the “predicted value” and the other produces the “target value”. In the training phase, the “target value” is calculated to assist the learning of network parameters. In the testing phase, the “target value” is not calculated, but when the fusion information ht is received, a random action is selected with the help of the ε-greedy strategy with probability ε , as shown in Figure 4:

Figure 4.

The regression network is a fully connected network fc(128*128*4). When the termination action is generated during the learning of an epoch, and the IoU between the visible area and the real marked area is greater than 0.6, the network will fine-tune the coordinates of the current visible area to obtain the offset that needs to be adjusted in the corresponding direction of the bounding box (∇X, ∇Y, ∇W, ∇H), as shown in Figure 5:

Figure 5.

The tasks of the training set are mapped to different positions in the feature space by means of feature mapping for the generation of meta-parameters. Due to the problem of parameter redundancy and computational complexity in the original VGG-16 network, in this paper, the original fully connected layer is replaced by the global average pooling layer, and GAP-VGG16 is constructed as the feature extraction network, as shown in Figure 6. The feature types of each task can be obtained through feature extraction, and the number of feature types is controlled by specifying the range of mapping (set to 10 in this paper), and each feature range corresponds to the storage space in a meta-parameter pool.

Figure 6.

The feature mapping result of few-shot task is vector form a_i. For each meta-parameter Meta_θ obtained by training, there is a feature vector b_i corresponding to it, and the corresponding meta-parameter Meta_θ is matched by measuring the difference d(a_i, b_i) between vectors a_i and b_i. In this paper, the Euclidean distance between two vectors is used to judge the mapping space region, as well as the degree of similarity between tasks. The calculation formula is given in (5). 5 d(ai,bi)=∑i=1n(ai−bi)

In this paper, the feature vector corresponding to the task of the first meta-parameter of model training is used as the benchmark vector, as the label of 1 meta-parameter stored in the meta-parameter pool position, the difference value κ is set, and the multiple of the difference value κ is set to select the meta-parameters in the model meta-parameter pool, that is, the feature extraction of a few-sample training task is carried out. The similarity between the extracted feature vector and the feature vector label corresponding to the first position of the meta-parameter pool is calculated, and the matched meta-parameter position is obtained by dividing it with κ and adding 1. As shown in (6). 6 metaθi=d(ai,b1)κ+1

Here, Meta_θi represents the initial meta-parameters matched during retraining of the i^th few-shot dataset, and b₁ represents the feature vector corresponding to the meta-parameter at the first position of the meta-parameter pool.

In order to avoid forgetting the historical tasks of the meta-parameters, this paper uses (7) to update the meta-parameters, and retains the previous memory at each update of the meta-parameters. As shown in (7).

The meta-parameter pool O stores the meta-parameters of N kinds of tasks and updates them. For the new training task, the region Oi in the meta-parameter pool is matched according to the way of feature mapping, and the corresponding meta-parameters Meta_θ are selected as the initial parameters for retraining. The retrained meta-parameters allow the retention of the previous memory, and the trained meta-parameter pool maintains the optimal loss value for tasks of the same task type. The update process of the meta-parameter pool is shown in Figure 7.

Figure 7.

The meta-parameters in the meta-parameter pool are updated by continuous learning, and (7) is used to preserve the historical memory when the agent is updated. Where θ_i represents the meta-parameter after the i^th update and represents the learning coefficient, which is used to prevent the model distortion caused by too large parameter changes.

The parameters of the target localization model include the parameters of the action network and the regression network, namely θ = (θ_a, θ_g). The historical experience of the action network is represented by the tuple (s, a, r, s′, b, g), as shown in Figure 3. There are multiple exploration tasks in the same environment, and each exploration task will generate an MDP sequence. Expressed as the strategy of the agent, the loss function of each task Ti is shown in (8). 8 θi*=θ−α∇LTi(fθ)

The comprehensive loss of the target localization model includes the loss of the action network and the loss of the regression network and (7) is the loss function. The weighted sum of the losses of the action network and the regression network is used as the comprehensive loss of the target localization model. The mean square error loss function is used for the action network and the smoothL₁ loss function is used for the regression network. The overall loss function is defined as in (9) - (11). 9 L(s,a,t)=Laction+λLreg

Among them, 10 Laction=1Naction∑ [ (yi−Q(s,a;θi))2 ] 11 Lreg=1NregS(ti−ti*)

Where N_action and N_reg are the number of execution steps of the action network and the number of execution steps of the regression network, and their losses are averaged as the loss of the exploration action and the regression action. Q(s, a; θ_i) is the predicted value derived from the “prediction branch” of the action network, and y_i is the target value derived from the “target branch” of the action network. In the regression network loss L_reg, t_j = {t_x, t_y, t_h, t_w} denotes a vector of dimension size 4. Here, ∑L_Ti(f_Φ) and t_y are the center coordinates obtained by the regression network, respectively, and t_h, t_w are their corresponding heights and widths. tx∗ and ty∗ are the center coordinates of the true labeled regions of the regression network, respectively, and th∗, tw∗ are the corresponding heights and widths. Where S is the smoothL₁ function, see (12). 12 soomthL1(x)={ 0.5x2,if| x |<1x|−0.5,oterwise a is the loss balance coefficient, which is used to balance the loss of the action network and the regression network, and s stands for the regression loss.

Each RL goal localization task T_i contains an initial state distribution LTi and transition distribution q_i(x_t+1 | x_t, a_t), and the loss LTi corresponds to the negative reward function R. In this paper, we treat each object localization task as a Markov Decision Process (MDP) with an initial state S0, which allows the agent to retrace historical exploration trajectories and perform learning across tasks T_i the model is defined with respect to task T_i and loss L_Ti(f_φ) as shown in (13). 13 LTi(fφ)=−Ext,at~fφ,qTi[ ∑t=1HRi(xi,at) ]

For each of the k tasks Task1-k in the meta-parameter pool O_i, the model generates H exploration actions (x1,a1,… xH,aH) and the corresponding loss L_Ti(f), K accumulated losses ∑L_Ti(f_Φ) generated for the model are used for the meta-parameter Meat_θ (T_1-k) corresponding to the task type in the meta-parameter pool O_i region, see (14). 14 Metaθ=θ-β∇θ∑T1−k~p(T)LTi(fθi′)

Using meta-parameters when training on a new task leads to convergence in fewer exploration steps. In this paper, meta-reinforcement learning includes two parts InnerLoop and outerLoop , and the agent uses network random parameters as initialization parameters in each task pair. In InnerLoop, the agent continuously explores the target to generate an MDP-sequence, which converges to θi* through multiple training gradient updates. The outer loop of s is classified according to the feature types of the training tasks, and the corresponding meta-gradients of s are cumulatively updated to obtain the meta-parameters of multiple task types, that is, the meta-parameter Metaθi maximizes the sum reward of multiple task rewards. The trained meta-parameter Metaθi is stored in the meta-parameter pool module, waiting for matching and updating during retraining. Figure 8 shows the parameter training process of meta-reinforcement learning.

Figure 8.

In this paper, Torch deep learning platform is used to train model parameters with each kind of reinforcement learning object recognition task in the joint data set of VOC2007 +VOC 2012 as a task type, and the training of the object localization model is tested with the Test data set of VOC2007. In order to test the generalization performance and convergence efficiency of the proposed model for new tasks, six samples of different task types in VOC 2007+VOC 2012 dataset are selected for retraining. For the other 14 kinds of task datasets used for training update model meta-parameter pool. The experiment measured the convergence speed of the target localization model with different initial parameters, the localization precision (ap), the return rate (recall) of the trained model, and the number of required positioning steps.

In this section, the random initial parameter θ_random, the MAML-RL based meta-parameter θ_maml and the adaptive meta-parameter θ_meta of this paper are compared in the convergence speed of retraining with few samples on VOC 2007+VOC 2012 datasets, respectively. In this paper, only one type of target is located in the target location part. As shown in Figure 9, the Loss value of the model with random parameter θ_random is 2.8583 in the initial training, and converges to 0.2 after 2000 training times. Using the meta-parameter θ_maml based on MAML-RL, the Loss value of the initial training of the model is 2.0525, indicating that the model has a certain learning ability, and converges to around 0.2 after 1000 training times. For the method in this paper, the model uses adaptive meta-parameters to have a low Loss value (1.3347) for few sample data in the initial case, indicating that the model automatically matches meta-parameters adapted to the new task as initial parameters, and the model reaches convergence in a few steps (500). In addition, with the same number of training steps, the few-shot task overall fits more than r and m through retraining. It shows that the proposed method has good generalization and learning ability for few-shot data training.

Figure 9.

In this paper, six categories cat, bicycle, aeroplane, cow, tvmonitor and DOGin VOC 2007+VOC 2012 dataset are selected for testing with the meta-parameters based on the proposed method as initial parameters. By testing the test samples after the same training batch, it is found that the proposed method can make the target localization model have better recognition accuracy for the target in a few steps, and part of the samples can capture the target position in one action step. Figure 10 shows the test results of the test sample, where Iteration represents the number of steps explored by the model, and the blue wireframe represents the prediction of the model on the target field of view.

Figure 10.

Figure 11 shows the precision rate (Ap) and Recall rate (Recall) of the model under the test data set and the proposed method under the VOC 2007 test set for the new tasks bird, motorbike, diningtable and train categories. The initial discount rate of the model is set to 0.5 and decreases to 0.1 in steps of 0.1, which shows the probability that the agent chooses the optimal action according to the model. It can be seen from Figure a and Figure b that the accuracy and recall of the model decrease when the learning rate increases from 0.1 to 0.5, and the accuracy and recall of the model for different tasks reach the highest when the learning rate is 0.1.

Figure 11.