A study on the visual effect and user response of infomercials based on neural network analysis

The algorithmic theory of generalized regression neural networks is based on nonlinear regression analysis, if the joint probability density of x, y two random variables is assumed to be f(x, y), and the observations of x are known to be x₀, then the regression of y with respect to the random variable x can be expressed as:

E(y|x0)=(x0)=∫−∞0yf(x0,y)dy∫−∞0f(x0,y)dyE(y|{x_0}) = ({x_0}) = {{\int_{ - \infty }^0 y f({x_0},y)dy} \over {\int_{ - \infty }^0 f ({x_0},y)dy}} where y(x₀) is the predicted value y if the input is x₀. If the Parzen nonparametric estimation method is applied, it is possible to estimate the density function f(x₀, y) using the sample data set { xi,yi }i=1n\{ {x_i},{y_i}\} _{i = 1}^n in the form of (1): 1f(x0,y)=1n(2π)p+12σp+1∑i=1ne−d(x0,xi)e−d(x0,xi)f({x_0},y) = {1 \over {n{{(2\pi )}^{{{p + 1} \over 2}{\sigma ^{p + 1}}}}}}\sum\limits_{i = 1}^n {{e^{ - d({x_0},{x_i})}}} {e^{ - d({x_0},{x_i})}}

Where: d(x0,xi)=∑p[ (x0j−xij)/σ ]2,d(y,yi)=[ y−yi ]2d({x_0},{x_i}) = \mathop \sum \limits^p {[({x_{0j}} - {x_{ij}})/\sigma ]^2},d(y,{y_i}) = {[y - {y_i}]^2}. where n is the sample size, p is the dimension of the random variable x, and σ is the smoothing factor, i.e., the standard deviation of the Gaussian function, which is obtained by substitution: 2y(x0)=∑i=1n(e−d(x0,xi)∫−∞+∞ye−d(y0,yi)dy)∑i=1n(e−d(x0,xi)∫−∞+∞e−d(y0,yi)dy)y({x_0}) = {{\sum\limits_{i = 1}^n {({e^{ - d({x_0},{x_i})}}\int_{ - \infty }^{ + \infty } y {e^{ - d({y_0},{y_i})}}dy)} } \over {\sum\limits_{i = 1}^n {({e^{ - d({x_0},{x_i})}}\int_{ - \infty }^{ + \infty } {{e^{ - d({y_0},{y_i})}}} dy)} }}

It is known that ∫−∞+∞xe−x2dx=0\int_{ - \infty }^{ + \infty } x {e^{ - {x^2}}}dx = 0, from which the simplification follows: 3y(x0)=∑i=1nye−d(y0,yi)∑i=1ne−d(x0,xi)y({x_0}) = {{\sum\limits_{i = 1}^n y {e^{ - d({y_0},{y_i})}}} \over {\sum\limits_{i = 1}^n {{e^{ - d({x_0},{x_i})}}} }}

This is the predicted value of y if the input is x₀.

From the predicted value formula, it can be seen that the numerator is the weighted sum of y_i obtained from all samples, where the weight value is e^{–d(x₀,x_t)}. Another point of interest is the value of the smoothing factor σ, which has a great impact on the performance of the network itself because generalized regression neural networks do not need to be trained, and in practice there is no need to change the value of the connection weights of the neurons in the network, and it is only necessary to search for the optimal value of the smoothing factor in order to find the optimal model. In general, if the value of σ is very large, then d(x₀, x_i) tends to 0, and the predicted value y(x₀) is approximately equal to the mean of all sample explanatory variables. If the value of σ tends to 0, then the predicted value y(x₀) will be very close to the sample value, but if non-sample values are entered, the predicted value y(x₀) will be much different, which is known as the overlearning phenomenon.

From the above, it can be seen that the generalized regression neural network does not need to be trained, but only needs to be adjusted by adjusting the value of the smooth factor σ to achieve changes in the connection weights between neurons in the network and thus optimize the network. Smooth factor increment △σ in the range of (σ_min, σ_max) sequential incremental changes in the learning samples to remove a few groups of samples, to be determined after the smooth factor to use the removal of these groups of samples for prediction, and then derive the predicted value of the group of samples and the sequence of error between the samples and the computed error columns mean square deviation: 4E=1n∑i=1n[y^i(xi)−yi]2E = {1 \over n}\mathop \sum \limits_{i = 1}^n {[{{\hat y}_i}({x_i}) - {y_i}]^2}

When generalized regression neural network performance evaluation metrics, the smaller the value of the mean square error E, the better the performance of the network. In practical applications, the root mean square error (RMSE) is also often used as an evaluation metric for the network, i.e: 5RMSE=1n∑i=1n[ y^i(xi)−yi ]2RMSE = \sqrt {{1 \over n}\sum\limits_{i = 1}^n {{{\left[ {{{\hat y}_i}({x_i}) - {y_i}} \right]}^2}} }

Generalized regression neural networks are used as a special form of radial basis neural networks and therefore have some similarities with them in terms of neurons. The input vector of the network is [x_n.1, x_n.2, …, x_n.M], the center vector is [c_j.1, c_j.2, …, c_j.M], ||dist|| denotes the Euclidean distance between the input vector and the center vector, b is the threshold, and f is its transfer function, which is usually a Gaussian function, i.e., a normal distribution function.

The network consists of four layers, namely, input layer, pattern layer, summation layer and output layer, where the input layer is X = [x₁, x₂, …, x_n]^T and the corresponding output layer is Y = [y₁, y₂, …, y_k]^T. The number of neurons in the input layer and the number of neurons in the pattern layer are equal to the input sample vectors.

The input layer is the sample input layer, the dimension of the input vector in the learning sample is the number of neurons in the network, and each neuron is a simple distribution unit, and the transfer function is a simple linear function, which transfers the input variables to the second layer of the network, i.e., the pattern layer.

The neurons in the pattern layer are radial basis neurons, the number of which is equal to the learning samples, and each radial basis neuron corresponds to a different sample. The transfer function of the radial basis neuron is: 6pi=exp[ −(x−xi)T(x−xi)2σ2 ]i=1,2,…,n{p_i} = \exp \left[ { - {{{{(x - {x_i})}^T}(x - {x_i})} \over {2{\sigma ^2}}}} \right]i = 1,2, \ldots ,n where x is the input variable of the generalized regression neural network and x_i is the sample corresponding to the ird neuron. Usually, the output of a neuron is equal to the exponent of the square of the Euclidean distance between the input variable and the corresponding sample.

In the third summation layer of the generalized regression neural network, there are two types of neurons to be summed.

The formula for the first type is: 7∑i=1nexp[ −(X−Xi)T(X−Xi)2σ2 ]\sum\limits_{i = 1}^n {\exp } \left[ { - {{{{(X - {X_i})}^T}(X - {X_i})} \over {2{\sigma ^2}}}} \right]

This type allows arithmetic summation of neuron outputs from all pattern layers. In this type, the connection weight between the pattern layer and the individual neurons is 1 and the transfer function is: 8SD=∑i=1nPi{S_D} = \sum\limits_{i = 1}^n {{P_i}}

The formula for the second type is: 9∑i=1nYiexp[ −(X−Xi)T(X−Xi)2σ2 ]\sum\limits_{i = 1}^n {{Y_i}} \exp \left[ { - {{{{(X - {X_i})}^T}(X - {X_i})} \over {2{\sigma ^2}}}} \right]

This type allows for a weighted summation of the neuron outputs from all pattern layers. In this type, the value of the connection weight between the i st neuron in the pattern layer and the j nd molecular summation neuron in the summation layer is the jth element in the ird output sample Y_i with the transfer function: 10SNj=∑i=1nyijPij=1,2,...,k{S_{Nj}} = \sum\limits_{i = 1}^n {{y_{ij}}} {P_i}\quad j = 1,2,...,k

The number of neurons in the output layer is equal to the dimension k of the output vector in the learning sample, and each neuron divides the output of the summation layer, and the output of neuron j sums up to the jth element in the corresponding estimation result Ŷ = (X) with the following formula: 11yj=SNjSDj=1,2,...,k{y_j} = {{{S_{Nj}}} \over {{S_D}}}\quad j = 1,2,...,k

Visual saliency prediction, as a key image analysis technique in the field of computer vision, can mimic the human eye attention mechanism to quickly capture the focused and intriguing regions in an image. In this paper, we propose a collaborative attention network of text features and visual features to solve the problem of visual saliency prediction for infomercials.

This subsection outlines how to combine the visual and textual features of advertisement images and weight them through the collaborative attention mechanism. This paper demonstrates that the construction process of the advertisement visual saliency prediction method is shown in Fig. 1.

Figure 1.

Construction process

The CAM mainly contains five modules: input module, visual feature learning module, text feature learning module, collaborative attention module, and click rate prediction module. The first is the input module, which is responsible for inputting the visual and textual information of the target advertisement image. Next, the second module of CAM is the visual feature learning module, whose purpose is to learn to acquire the visual features of the image, which include color, hue, and so on. The third module is the textual feature learning module, in which the textual information is segmented to generate a sequence of phrases, and a low-dimensional word vector representation is generated for each phrase by the pre-trained word embedding model word2vec. Then, the word vectors learned by the pre-trained word embedding model are learned by Bi-LSTM to generate the corresponding text features. Next is the co-attention module, which is mainly responsible for learning the attention weights of text features and visual features in pairs. The module is able to process both textual and visual features, generate corresponding attention weights for each feature, and then combine these weights to determine the effect of each feature on the output. The collaborative attention module enables the model to pay more attention to key visual and textual information, thus improving the performance of the model. Finally, based on the textual and visual features weighted by the collaborative attention mechanism, the prediction module obtains the final prediction results through a multilayer perceptual machine. Through the synergistic work of these modules, the CAM is able to obtain information about the advertisement image from multiple aspects, thus improving the accuracy of the prediction.

The visual features G ∈ R^d × N_g and textual features V ∈ R^d × N_v (i.e., spliced vectors of all phrases) of the advertisement image xi extracted through the two modules are used as inputs to the synergetic attention layer, and then the similarity matrix C ∈ R^{N_d × N_v} is computed to obtain the similarity matrix by the following formula: 12C=tanh(GTWbV)C = \tanh ({G^T}{W_b}V) where W_b ∈ R^{d × d} is the corresponding weight matrix.

By obtaining the row and column maxima of the similarity matrix and weighting the original textual features V and visual features G with them, a standard collaborative attention vector can be derived. However, in this paper, we tend to consider the similarity matrix as a feature and compute the textual feature and visual feature attention matrices by the following formula: 13Hv=tanh(WvV+(WgG)C){H^v} = \tanh ({W_v}V + ({W_g}G)C) 14av=softmax(whvTHv){a^v} = soft\max (w_{hv}^T{H^v}) 15Hg=tanh(WgG+(WvV)CT){H^g} = \tanh ({W_g}G + ({W_v}V){C^T}) 16ag=softmax(whgTHg){a^g} = soft\max (w_{hg}^T{H^g})

Where W_v, W_g ∈ R^k×d and w_hv, w_hg ∈ R^k are the corresponding weights, a^v ∈ R^N and ag∈RgN{a^g} \in R_g^N are the attentional weights for each textual feature v_i and each image region visual feature g_i, respectively. The previously mentioned similarity matrix C can convert the text feature space to visual feature space, and vice versa, the similarity matrix C^T can convert the visual feature space to text feature space. Based on the above attention weights, the final textual feature and final visual feature attention vectors.

Are obtained by calculating the weighted sum of each textual feature and each visual feature, i.e: 17v^=∑i=1Nvaivvi,g=∑j=1Ngajggj\widehat v = \sum\limits_{i = 1}^{{N_v}} {a_i^v} {v_i},g = \sum\limits_{j = 1}^{{N_g}} {a_j^g} {g_j}