Influence analysis of piano music immersion virtual reality cooperation based on mapping equation

In order to realize the color migration between two uncorrelated images, it is necessary to establish a color channel in lab space. Therefore, it is necessary to establish a relationship between the color features of color images and shape images. Then first extract the main color descriptor of the two images, set the main color of the color image as F_y, and its expression is: (1) Fy={pa,ca,va}, a=1,2,…,N {F_y} = \left\{{{p_a},{c_a},{v_a}} \right\},\,a = 1,2, \ldots,N

Where N represents the number of main color pixels in the color image, p_a is the percentage of the a-th main color in the image pixels, and its calculation method is to divide the number of main color pixels and the number of pixels in the image. In formula (1), c_a represents the vector of the a-th main color, and v_a represents the variable of the main color near the main color vector. Since the number of main color pixels extracted from the color image and the shape image is the same, which is convenient for mapping and matching, it is necessary to calculate the distance between the main colors of the two images. The distance calculation method is shown in equation (2): (2) d(cs,cr)=min∑a=1Ndc(cφ(a)s, car) d\left({{c^s},{c^r}} \right) = \min \sum\limits_{a = 1}^N {{d_c}\left({c_{\varphi \left(a \right)}^s,\,c_a^r} \right)}

Where c_s and c^r represent the main color vector of color image and shape image respectively, and φ(a) belongs to a group of pixel arrangement in n. Through the obtained main color distance results, the color image and shape image are segmented according to the main color region, and the features are extracted in the region. According to the calculation results of the main color distance, calculate the color histograms of the two images in RGB space, merge the parts belonging to the same main color, and divide the whole merging process into multiple steps [6]. Color feature extraction is carried out in each segmented region. All pixels in a segmented region are represented by set SR₁, which contains n pixels. Then the average brightness value in the segmented region can be calculated by equation (3) (3) f1(a)=1n(x,y)∈SRa∑I(x, y) {f_1}\left(a \right) = {1 \over {{n_{\left({x,y} \right) \in S{R_a}}}}}\sum {I\left({x,\,y} \right)}

In equation (3), A (x, y) represents the brightness of the pixel with the position of (x, y) on the image, so the feature results about brightness can be extracted. Similarly, the feature values about transparency, chromaticity and so on can be extracted from the image.

Perform boundary smoothing on the source image to solve the error caused by segmentation, and use bilateral filter to smooth color and spatial information at the same time: (4) D(A(x,y),A′(x′, y′))=b×e−((x−x′)2+(y−y′)2σ)+(1−b)×e−(|A(x, y)|−|A′(x′, y′)|σ) \matrix{{D\left({A\left({x,y} \right),{A^{'}}\left({{x^{'}},\,{y^{'}}} \right)} \right) = b \times {e^ -}\left({{{\sqrt {{{\left({x - {x^{'}}} \right)}^2} + {{\left({y - {y^{'}}} \right)}^2}}} \over \sigma}} \right)} \hfill \cr {+ \left({1 - b} \right) \times {e^ -}\left({{{\left| {A\left({x,\,y} \right)} \right| - \left| {{A^{'}}\left({{x^{'}},\,{y^{'}}} \right)} \right|} \over \sigma}} \right)} \hfill \cr}

In the formula, parameter a represents the balanced color coefficient of the image, and σ is the dynamic selection coefficient of spatial structure. Generally, a and σ take constants. The spatial smoothing process of the source image ensures the smooth transition of the region boundary color after color migration.

Generally, the format of the image is RGB format, which is the spatial standard of one color in the image, in which R represents red, G represents green and B represents blue. Other different colors can be obtained by changing these three colors and superimposing points. However, in the process of color migration, it is necessary to convert the format of target image and source image from RGB color space format to color space. Because the color transfer algorithm between images involves reverse mapping, it is easy to cross map in RGB color space. The correlation between the three channels of the image in lαβ space is small, so the problem of channel crossing will not occur when mapping in this space [7]. Since RGB cannot directly convert the image with lαβ space, to convert the image from RGB space to lαβ space, you first need to convert it to LMS color space through equation (5). (5) [LMS]=[0.14780.5470.02400.24980.36770.03650.06420.17450.9854]=[RGB] \left[ {\matrix{L \cr M \cr S \cr}} \right] = \left[ {\matrix{{0.1478} & {0.547} & {0.0240} \cr {0.2498} & {0.3677} & {0.0365} \cr {0.0642} & {0.1745} & {0.9854} \cr}} \right] = \left[ {\matrix{R \cr G \cr B \cr}} \right]

Then, the image under LMS can be converted into lαβ space through equation (6). (6) [laβ]=[1311116111132][00000−20−10]=[LMS] \left[ {\matrix{l \cr a \cr \beta \cr}} \right] = \left[ {\matrix{{{1 \over {\sqrt 3}}} & 1 & 1 \cr 1 & {{1 \over {\sqrt 6}}} & 1 \cr 1 & 1 & {{1 \over {\sqrt {32}}}} \cr}} \right]\left[ {\matrix{0 & 0 & 0 \cr 0 & 0 & {- 2} \cr 0 & {- 1} & 0 \cr}} \right] = \left[ {\matrix{L \cr M \cr S \cr}} \right]

In lαβ color space, l represents gray scale, and α and β are color information.

In order to ensure the accuracy of color migration, it is necessary to reverse map the tone pixels in the image, and its processing flow is shown in Figure 1.

Figure 1

It can be seen from the figure that the color needs to be edited and defined before reverse mapping. The initial color feature of pixel I in the defined color image is represented by ac, and its color feature is represented as ac′ a_c^{'} after color feature transformation. The editing function of ac′ a_c^{'} can be expressed by equation (7): (7) ac′=f(ac,ec) a_c^{'} = f\left({{a_c},{e_c}} \right)

Where e_c represents the color editing parameter of i pixel. The color editing intensity is represented by g_c and w_c respectively, and its value varies from 0 to 1. Then the frame color editing of color migration algorithm can be regarded as a mapping process [8].

Taking the color brightness between images as an example, the average value of logarithmic brightness is used as the scale factor of its characteristic value, and the input brightness value is linearly scaled, then the expression can be expressed as: (8) fc=αFavgF0 {f_c} = {\alpha \over {{F_{avg}}}}{F_0}

In equation (8), F₀ is the input brightness value. Then the inverse mapping formula of color migration about brightness between images can be expressed as: (9) Fout=(fc)1γ−Fmin1γFmax1γ−Fmin1γ {F_{out}} = {{{{\left({{f_c}} \right)}^{{1 \over \gamma}}} - F_{\min}^{{1 \over \gamma}}} \over {F_{\max}^{{1 \over \gamma}} - F_{\min}^{{1 \over \gamma}}}}

Where F_out is the result of reverse mapping and γ is the coefficient of reverse mapping, which is often taken as 2.2.

According to the result of tone reverse mapping, the K-means clustering method is used to cluster the pixels of the same type, and the pixels of the same area between the two images are matched as the color migration range. Firstly, K pixels are selected as the clustering center in the color image, then the distance between each pixel in the color image and the clustering center is calculated, and the calculation results are compared to assign it to the nearest clustering center until each pixel in the image is assigned. Then, the region is allocated according to the clustering results, the closest region is matched first according to the reverse mapping method, and then the region of the remaining shape image and the region of the color image are matched in turn, so as to achieve the best matching result.

This paper uses HTC Vive virtual reality suite as the hardware platform of virtual piano. A complete set of HTC Vive virtual reality equipment is composed of a helmet, two handles and two locators.

Vive uses the indoor positioning technology of lighting house. The two rows of LED lights in the locator send out scanning beams six times per second and scan the positioning space of 15×15 feet in turn in the horizontal and vertical directions. The HTC Vive helmet and handle have more than 70 photosensitive sensors. When the beam is scanned, the helmet starts counting. After the sensor receives the scanning beam, the accurate position relative to the locator is calculated by using the relationship between the sensor position and the receiving laser time. With the help of positioning technology, users can walk around the virtual piano within the detection range, and even play the piano on the other side of the keyboard to further increase the sense of three-dimensional and immersion.

Under normal conditions, HTC Vive needs to operate the menu through the handle. However, holding the handle is not convenient for playing the virtual piano. After wearing the helmet, you can't observe the surrounding environment and pick up the handle. Therefore, the virtual piano developed in this paper does not use the handle, but controls the virtual piano through gesture recognition, such as timbre setting and rhythm adjustment. The Leap Motion sensor is directly installed on the Vive helmet.

HTC Vive helmet and Leap Motion sensor are connected to the computer through HDMI and USB interfaces respectively. The configuration properties of the computer and built-in independent graphics card used in this paper are shown in Table 1.

Unity3D is a professional game engine developed by unity technologies, which allows game developers to easily complete various game creativity and 3D interactive development. Unity3D has rendering resources such as physical simulation, normal mapping, screen space environment masking and dynamic shadow. Compared with other game development tools, it has two main advantages: it provides extremely high visual workflow and multi-dimensional cross platform support. With the help of visual workflow, scene layout editing, resource binding and interactive object scripting language coding can be carried out conveniently. In terms of deployment targets, Unity3D spans multiple platforms and can run on 21 platforms, including current Windows, Mac, Wi i, iPhone, WebGL, Windows phone8 and Android. Unity3D is usually used for ordinary 3D game development. To develop virtual reality projects, you need to obtain the VR plug-in steamvt SDK. The steamvt SDK can be downloaded for free on the official website of Unity3D or HTC Vive [9–10].

From ordinary 3D game development to virtual game development, the key is to convert ordinary cameras to VR cameras. This can be achieved by setting up prefabricated parts. The preform is composed of parent and child objects. For example, the parent object [CameraRig] is a standard application combination of StreamVR, including VR camera, two handles and locator. When creating a scene, you need to drag the preform into the editor to change it into a VR scene. Sub objects include left hand, right hand and head display. There is a handle model under the left hand. In the actual project, the handle can be customized as the model 1 required by the user. Prefabricated parts play an important role in project development. Prefabricated parts are required for piano keyboard modeling and hand modeling in the next section.

An ordinary piano consists of 88 black-and-white keys separated by half tones. From left to right, there are bass area, midrange area and treble area respectively. For simplicity, the virtual piano developed in this paper sets 7 tones of do, re, mi, fa, suo, la and si in the bass area, midrange area and treble area respectively, with a total of 21 white keys and 15 black halftone keys. The piano interface in Unity3D uses canvas, which carries the area of all UI elements. This component is bound to the piano key object in the hierarchy panel. There are three canvas rendering modes: screen space overlay, world space and screenspace camera. Screen space - the canvas in the overlay mode will fill the entire screen space, and all UI elements under the canvas will be placed on the top layer of the screen, which will not be obscured by any object, and its contents can be seen without a camera; The world space mode is the world space mode. In this mode, the canvas will be located in the game scene like a 3D object, and its position and size can be set.

This paper adopts the screen space camera mode. The screen space camera mode is similar to the screen space overlay mode. The canvas also fills the whole screen space. If the screen size changes, the canvas will automatically change its size to match the screen. The difference is that in this mode, the canvas is placed in front of the camera. In this rendering mode, the canvas looks like it is drawn on a plane at a fixed distance from the camera. All UI elements are rendered by the camera. Canvas can also specify the camera for rendering, so the camera settings will affect the UI picture. Using this mode to display the piano model on the UI, the effect setting will be better than the other two.

Dynamically zoom according to the distance between the camera and the canvas and the size of the visual cone of the camera. When an object is closer to the camera than the canvas, the piano will be displayed in front of the canvas. Objects farther from the camera than the canvas will be blocked. Adjust the plane distance between the canvas and the camera to 100, set the reference resolution, select the scale with screen size / match width or height scaling method in UI scale mode to correspond to different size resolutions, and finally achieve the piano effect [11].

This section uses sliding window segmentation technology to extract a single gesture action. The data segmentation technology based on sliding window is to segment a fixed length window from a continuous motion signal. The window slides along the time axis and averages all the data in the window. Assuming that the average value represents the average speed of fingers in the window, if the average speed is very small and lower than a certain threshold, it is considered that there is no gesture movement in the time period of the window; If the average speed reaches the maximum, it indicates that there is gesture movement in the window time period, and the window time period just covers a single gesture movement. The instantaneous closing speed of the finger tip and the average closing speed in the whole window can be calculated by the following formula. (10) vi=(xt+1−xt)2+(yt+1−yt)2+(zt+1−zt)2Δt {v_i} = {{\sqrt {{{\left({{x_{t + 1}} - {x_t}} \right)}^2} + {{\left({{y_{t + 1}} - {y_t}} \right)}^2} + {{\left({{z_{t + 1}} - {z_t}} \right)}^2}}} \over {\Delta t}} (11) Vt=argmaxt[1k∑tt+k−1vt] {V_t} = \arg \mathop {\max}\limits_t \left[ {{1 \over k}\sum\limits_t^{t + k - 1} {{v_t}}} \right]

In formulas (10) and (11), v_i represents the three-dimensional instantaneous closing velocity of the i-th sample point. Since the time interval between two adjacent sampling points is fixed, Δt = 1 and k is the window width. Because the data sampling rate of Leap Motion is 100fps / s and the general gesture action is no more than 1s, k = 100 can be taken. Set the minimum and Lang value to 1. If the average closing speed in the window is lower than this threshold, it is considered that there is no gesture movement [12].

Figure 2 shows a schematic diagram of segmentation and extraction of finger click action. As can be seen from the figure, through the sliding window segmentation technology, the window can be accurately positioned at each click action time, so as to segment and extract a single click action.

Figure 2