A mathematical model of PCNN for image fusion with non-sampled contourlet transform

NSCT is mainly composed of two parts: non-down sampling tower decomposition (NSP) and non-down sampling directional-filter-bank (NSDFB). The structure diagrams of both are expressed in Figure 1(a). Among them, NSP is used to complete the scale decomposition of the image and divide the image into multiple frequency levels. The number of scale decomposition is usually set between 2 and 4. This decomposition has no downsampling operation. Therefore, the size of the high-frequency component and low-frequency component obtained after decomposition is the same as that of the source image. Due to the lack of directional description of the high-frequency subband information, NSDSB is used in high-frequency components to further divide each high-frequency component into the information of each direction component, and the number of directional subbands is usually an integer power of 2 (Figure 1b).

Fig. 1

The structure of NSP consists of two filter banks with non-sampling characteristics. There is no sampling process, so it is translation invariant. After NSP decomposes the image into various scales, NSDFB can divide an image into arbitrary power directions of 2 on each scale, and meet the translation invariance. NSCT is redundant. The redundancy of NSCT is W=∑i=0I−12di W = \sum\nolimits_{i = 0}^{I - 1} {2^{{d_i}}} , where 2^d_i represents the number of directions of decomposition corresponding to the high-frequency scale components 2⁻ⁱ.

PCNN breaks the limitation of the existing artificial neural network, which only uses the limited attributes of biological neurons for model construction. It is a network with a feedback structure formed by the interconnection of multiple neurons based on eckhorn, which imitates the neuronal activities of the biological visual cortex. Structurally, each neuron can be thought of as having three parts: receiving part, modulation part and pulse generation part (as shown in Figure 2).

Fig. 2

When processing the image of M×N, the idea of PCNN is to operate pixel values in the spatial domain and take them as the input of neurons. Therefore, the size of the neural network composed of PCNN neurons is consistent with the image size. Among them, the activity of single-neuron N_ij can be expressed by the following formula: (1) Fij[n]=e−αFFij[n−1]+Sij+VF∑klMijklYkl[n−1] {F_{ij}}[n] = {e^{ - {\alpha _F}}}{F_{ij}}[n - 1] + {S_{ij}} + {V_F}\sum\nolimits_{kl} {M_{ijkl}}{Y_{kl}}[n - 1] (2) Lij[n]=e−αLLij[n−1]+VL∑klMijklYkl[n−1] {L_{ij}}[n] = {e^{ - {\alpha _L}}}{L_{ij}}[n - 1] + {V_L}\sum\nolimits_{kl} {M_{ijkl}}{Y_{kl}}[n - 1] (3) Lij[n]=e−αLLij[n−1]+VL∑klMijklYkl1[n−1]+VN∑klMijklYkl2[n−1] {L_{ij}}[n] = {e^{ - {\alpha _L}}}{L_{ij}}[n - 1] + {V_L}\sum\nolimits_{kl} {M_{ijkl}}Y_{kl}^1[n - 1] + {V_N}\sum\nolimits_{kl} {M_{ijkl}}Y_{kl}^2[n - 1] (4) Uij[n]=Fij[n](1+βLij[n]) {U_{ij}}[n] = {F_{ij}}[n]\left( {1 + \beta {L_{ij}}[n]} \right) (5) Tij[n]=e−αTTij[n−1]+VTYij[n] {T_{ij}}[n] = {e^{ - {\alpha _T}}}{T_{ij}}[n - 1] + {V_T}{Y_{ij}}[n] (6) Yij[n]={1Uij[n]≥Tij[n]0otherwise {Y_{ij}}[n] = \left\{ {\matrix{ 1 \hfill & {{U_{ij}}[n] \ge {T_{ij}}[n]} \hfill\cr 0 \hfill & {{\rm{otherwise}}}\hfill \cr } } \right. Where: S_ij, U_ij and Y_ij respectively represent the external input of neurons, also known as external stimulation, internal behaviour parameters and external output parameters; L_ij and F_ij respectively represent the link domain input channel of neurons and the feedback domain input channel of neurons. The connection weight coefficient between neurons is described by a matrix and represented by M and W, respectively. V_F and V_L (V_N) correspond to the connection domain amplification coefficient and feedback domain amplification coefficient, respectively; T_ij is the output of the threshold function, and V_T is the threshold amplification coefficient; α_L, α_F and α_T are time constants, α_L corresponding to link domain, α_F corresponding to feedback domain and α_T corresponding to variable threshold function.

The unique neuron capture characteristic of the PCNN model – the ignition of a neuron will cause the capture and ignition of adjacent neurons with similar brightness to the neuron, which can automatically realise information transmission and information coupling. Therefore, this unique characteristic of PCNN lays a foundation for the application of PCNN in image fusion. This paper adopts the bilateral PCNN model with parallel structure, which is characterised by selecting any one of the two images to be fused as the master neuron of PCNN, and the remaining one is the slave PCNN neuron. The activity equation of the master neuron is calculated by equations (1), (3) ∼ (6), and the activity equation of the slave neuron is calculated by equations (1), (2) and (4) ∼ (6). With the help of the capture characteristics of PCNN, the double-layer parallel PCNN network can input the ignition information of the slave PCNN neuron into the link domain of the corresponding neuron of the main PCNN, to fuse the neuron information in the slave PCNN with the corresponding neuron information of the main PCNN. That is, after setting the master and slave neurons to ignite a neuron in the slave PCNN, due to the setting of master and slave neurons, when a neuron in PCNN ignites, the ignition information of the neuron will be transmitted to the connected neurons and the corresponding neurons of the master PCNN at the same time. Although this method increases certain system complexity, it can make the image fusion effect more ideal.

Choosing reasonable fusion rules can enrich the information update of the fused image. The high and low frequency region components of the image correspond to different image features. Therefore, in practice, selecting different fusion rules for operation is conducive to the overall image fusion. Since NSCT has a large low-frequency coefficient and is the area where most information about the image is stored, equation (7) is used to calculate the corresponding fusion result, and the fusion weight is expressed in ω, as follows: (7) SLF(n,m)=ω*SLA(n,m)+(1−ω)*SLB(n,m) S_L^F(n,m) = \omega *S_L^A(n,m) + (1 - \omega )*S_L^B(n,m)

In the low-frequency region, selecting proper fusion weights can increase the effect of consolidation. The basic idea of the ‘golden section method’ is to estimate the position of the optimal weight by using the optimal search method. The specific process is to eliminate the ‘bad points’, retain the ‘good points’ interval, and constantly narrow the search interval. After repeatedly comparing the function value of the trial position, the position of the optimal weight can be obtained. In the process of optimisation, the objective function selects the edge fusion quality index Q_E(ω) [19], which can well reflect the retention of detailed information such as the contour of the image.

After image registration, image fusion processing can be carried out. Taking two image fusion as an example, the fusion algorithm constructed in this paper is based on the exhaustive method, constantly searching for the best coefficient value of low-frequency region components based on the NSCT domain and the processing of high-frequency components by parallel PCNN. The specific operation process is shown in Figure 3.

Fig. 3

Firstly, the original images to be merged with A and B are decomposed by layer NSCT, respectively. NSP is applied to decompose the image to achieve low component on various scales, that is, the low component is {SlA(n,m),0≤l≤L−1} \{ S_l^A(n,m),0 \le l \le L - 1\} , {SlB(n,m),0≤l≤L−1} \{ S_l^B(n,m),0 \le l \le L - 1\} , and the passband is {Dl,iA(n,m),0≤l≤L−1,1≤i≤kl} \{ D_{l,i}^A(n,m),0 \le l \le L - 1,1 \le i \le {k_l}\} , {Dl,iB(n,m),0≤l≤L−1,1≤i≤kl} \{ D_{l,i}^B(n,m),0 \le l \le L - 1,1 \le i \le {k_l}\} . Then, The high-frequency directional subbands are obtained by NSDFB decomposition, which are, {Dl,iA(n,m),0≤l≤L−1,1≤i≤kl} \{ D_{l,i}^A(n,m),0 \le l \le L - 1,1 \le i \le {k_l}\} , {Dl,iB(n,m),0≤l≤L−1,1≤i≤kl} \{ D_{l,i}^B(n,m),0 \le l \le L - 1,1 \le i \le {k_l}\} , k_l represents the number of subbands in the high component direction, l refers to the corresponding decomposition layer, and Dl,iA(n,m) D_{l,i}^A(n,m) is the ith directional subband of the image. The specific fusion steps are as follows:

Step 1: for SlA(n,m) S_l^A(n,m) , SlB(n,m) S_l^B(n,m) , the golden section method is used to realise the adaptive search of the optimal low-component fusion weight ω^*;

Here, H (x) represents the average amount of information of the fused image, also known as information entropy, and p(x) is the probability.

Objective evaluation index of image merging:

Mutual information (MI): MI is used to reflect the correlation degree of two data. In image merging technology, the correlation between the two merged images can be reflected by MI. The size of this data value directly reflects the amount of merged information. In practice, the higher the parameter value, the better the merging effect. (9) IFB=∑k=0L−1∑j=0L−1PFB(k,j)log2PFB(k,j)PF(k)PB(j) {I_{FB}} = \sum\limits_{k = 0}^{L - 1} \sum\limits_{j = 0}^{L - 1} {P_{FB}}(k,j)\log 2{{{P_{FB}}(k,j)} \over {{P_F}(k){P_B}(j)}} (10) IFB=∑k=0L−1∑j=0L−1PFB(k,j)log2PFB(k,j)PF(k)PB(j) {I_{FB}} = \sum\limits_{k = 0}^{L - 1} \sum\limits_{j = 0}^{L - 1} {P_{FB}}(k,j)\log 2{{{P_{FB}}(k,j)} \over {{P_F}(k){P_B}(j)}}

Where, the probability densities of images A, B and F can be represented by P_A, P_B and P_F respectively, and P_FA(k, i) and P_FB(k, i) are the joint probability density of the fusion results and the images to be fused, respectively. Equation (11) MI is defined as: the merged image contains the ratio of the total sum of the A and B information in the merged images to the sum of the information entropy of the original A and B images t and normalises the interval [0,1], that is: (11) MI=IFA+IFBHA+HB MI = {{{I_{FA}} + {I_{FB}}} \over {{H_A} + {H_B}}}

Edge-dependent fusion quality index (EFQI). As an advanced parameter index for objective evaluation to evaluate the image fusion results, EFQI can effectively reflect the edge preservation of the fusion results and whether there is a ringing effect in the details such as contour and edge, and can measure its size. (12) QEFAI(A,B,F)=Qω(A,B,F)1−ε×Qω(A1,B1,F1)ε {Q_{EFAI}}(A,B,F) = {Q_\omega }{(A,B,F)^{1 - \varepsilon }} \times {Q_\omega }{({A_1},{B_1},{F_1})^\varepsilon } here Q_EFAI(A,B,F) is described as EFQI and Q_ω(A,B,F) is the evaluation index of weighted fusion. ɛ ∈ [0, 1], the parameter describes the importance of contour details in this image. When it is close to 1, it indicates that the contour is more important.

The images in the experiment are decomposed into four layers. In the wavelet-based fusion strategy, when selecting the wavelet basis function, we should consider the problem that the fusion result may produce artificial effects on vision, especially ringing and jitter. This is related to the discrete characteristics of sampling when downsampling is applied. If a non-integer number of signals is shifted and there is a constant local area connected to the sharp edge, the ringing will be enhanced. After interpolation, translation and resampling, the new sampling cannot be expressed as a constant in the transform domain but tends to oscillate (Gibbs phenomenon). In the wavelet-based fusion strategy, short decomposition or reconstruction filters should be used to avoid ringing. However, a very short filter will make the frequency selectivity worse. Considering comprehensively, Daubechies filter with 8 or 10 coefficients can provide better execution results for multi-scale image fusion. The wavelet basis selected in WTF and SWTF is’ db8 ’. Both CT and NSCT adopt the classical ‘bior’ tower decomposition and ‘DFB’ directional banks. The decomposition number of subbands from fine-scale to coarse-scale is 16, 8, 4 and 4. The fusion rule used in the multi-scale and multi-direction image merging method as a comparison in the experiment is: the approximate approximation coefficient at the minimum scale takes the mean value, other decomposition coefficients are generally in complex form, so the mode, i.e. absolute value, is selected, then the size is compared, and the larger one is selected as the final fusion coefficient.

Table 1 lists the comparison of fusion objective indexes obtained by several fusion methods. Through the comparison of experimental data, the required information, which can be clearly reflected in the table, is that the average amount of information, edge fusion quality and cross-correlation information of various algorithms are compared. The algorithm in this paper has improved in these three indicators compared with other algorithms, indicating that the algorithm using dual fusion rules can retain more details in image merging.

Figures 4 and 5 show the fusion effect of the image merging method using wavelet transform and NSCT as scale decomposition. It is obvious from the image that the constructed merging technology can preserve most of the contour and other details of the image, such as contour, edge and texture. Because NSCT is a non-down sampling transform, the edge is smooth and there is no ‘artefact’ phenomenon.

Fig. 4

Fig. 5

From Figures 4 and 5, there is a ringing phenomenon in WTF, which is caused by the fact that WTF is not a non-sampling transformation. The fusion effect of SWTF and CTF is not ideal, because they do not fully consider the difference between coefficients.