Application research of bel canto performance based on artificial intelligence technology

Traditional AI thinks in the top–down manner; its essential characteristic is to simulate the human brain neuron. This method has two characteristics: first, calculation of the weight of the neighbouring neuron through the corresponding output function and its further processing; second, the weight value is used to determine the information transmission relationship between neurons, and the weight is constantly optimised and adjusted in this process [10]. At the same time, the NN also depends on a large amount of data during processing. Therefore, the NN shows non-linearity, distributed configuration, parallel structure, adaptability, self-organisation and other characteristics. Take people's music creation as an example: usually through the perception of music (appreciation), music imitation and writing, they finally achieve independent creation. The process of composition also includes the study of composition techniques, harmony theory and other aspects. Learners constantly improve their creative ideas in practice and under the guidance of teachers. This learning method can be simulated by using the structure of NN to a large extent, thus laying a foundation for the application of this technology [11].

The operation process of NN requires the architecture of input, output, weight and multi-level perception (Figure 1). NNs can be regarded as a ‘black box.’ As long as there are enough training sets, the desired value Y can be obtained after X is given at the input end [12]. Specifically, the workflow of an NN is to determine the inputs and outputs, then to find a set or sets of algorithms that can be input and output, and then to find a set of known solutions to train the model; and then, over and over again, repeat the process; thus, conclusions can be drawn and the process can be constantly modified.

Fig. 1

Music is an art of time, and much of its message is based on time. On this basis, a variety of methods can be adopted to complete the processing of the timeline information, namely recurrent NN (RNN) [13, 14], which is an NN that can learn on the basis of existing and generated data. RNN is a time-shifted NN, which measures depth based on time. Circular networks generally have the same input layer and output layer. This is because the circular network expects the next entry to be the input of the next step, so as to generate the order; so RNN is a good implementation method [15].

Long short-term memory (LSTM) is a special RNN structure that belongs to feedback NN. LSTM is an NN formed to solve the problem of gradient disappearance or burst in an RNN cyclic network. RNN technology can process time dimension information at the same time, but if the data is stored for too long, it becomes very difficult to store the data, which becomes a big problem. Therefore, LSTM is a special hidden unit whose natural property is long-time storage. The main change to LSTM is the addition of three gates, namely the input gate, the output gate and the forget gate. In practice, LSTM has been proved to be superior to RNN [16]. It has been widely used in machine translation, conversation generation and compilation. LSTM can describe the logical thinking and cognitive activities of more complex people, so it is the most significant research field at present.

The autoencoder (AE) is shown in Figure 2. It uses an NN to convert an image or sound into a set of numbers. The goal is to make the image and sound searchable and then use that number to reconstruct the image and sound.

Fig. 2

The VAE is an improvement of the AE. Its structure is similar to that of the AE, but it also includes an encoder and a decoder [17]. The hidden vector generated by the variational autocoding method must satisfy the standard normal distribution. The VAE mode was first used in The Google Music VAE mode, where the user can input two videos, which are then inserted by the model to produce a continuous transit.

Transformer is the first machine translation model introduced by Google Brain in 2017. The advent of this model has shaken the RNN in the field of deep learning. In many experiments, the transformer performed better than the RNN. Google also made a name for itself in 2018 by publishing an article on Music Transformer to solve the music generation problem [18]. The transformer model mechanism is shown in Figure 3.

Fig. 3

It encodes the sequence of phonemes, note durations and pitches into a sequence of dense spatial vectors. Musical score = musical score; Phoneme ID = Phoneme to ID – encoded by phoneme embedding; Note duration = duration – encoded by duration embedding; Note pitch = pitch – encoded by pitch embedding; These three input vectors are then ‘stacked’ and thrown to multi-layer transformer encoder layers. As one can see, the position encoding information is also incorporated, followed by each FFT block [19, 20, 21]. The structure of the score encoder is shown in Figure 5.

Fig. 5

The sheet music usually contains the lyrics, pitch and length of the note so that people can get hold of the sheet music and sing it. As shown in Figure 5, the first step is to convert the word form into phonetic phonemes (such as Chinese characters to pinyin), where each syllable (such as fang->f + ang, a vowel + consonant form, so that both f and ang exist as phonemes) is subdivided into several phonemes. Each pitch is converted to a pitch ID according to musical instrument digital interface (MIDI) standards [22, 23, 24, 25].

In addition, the length is converted to frame count (the number of phoneme frames) via music Tempo. Note pitch and Note duration are repeated to fit the length of each phoneme. Thus, it can be considered that the shape of the input score is (1) S∈RN×3 S \in {R^{N \times 3}}

Type: N is the number of phonemes.

For example: WO ai ni Zhong Guo - wo ai n I zhong guo.

After three embedding layers respectively, the corresponding vectors of phoneme, duration and pitch will be added [26].

An FFT block consists of a multi-head self-attention layer, a two-layer 1D convolution layer (similar to the feed-forward linear layer) and the rectified linear unit (ReLU) activation function [27, 28]. There is no layer norm or residual connection in the paper as shown in Figure 6:

Fig. 6

I - > Wu O O three phonemes; - converts to phoneme ID; And - > h e e; - > pitch, a quantified pitch number; You - > n I I three phonemes. - > beats, first converted to a specific time length (according to MIDI tempo) and then converted to 10 ms as a cut. For example: 1 s = 1000 ms - > 1000 ms/10 ms = 100.

According to the vector sequence obtained by the encoder, obtain phoneme duration, i.e. the duration of each phoneme (one phoneme, how long should be sung in the song) [29, 30]. What it yields is the duration of each phoneme (for example, a sequence of integers). This predictor includes several 1D convolutional networks, similar to those in FastSpeech. In addition to the duration of phonemes, the duration of syllables also plays an important role in learning the rhythmic patterns of ‘singing speech synthesis.’ So, in an attempt to learn better rhythm, we propose to add syllable-based level (pronunciation of individual characters: Wu o o = I, now it is one: wo, the control). A syllable can correspond to one or more notes. Therefore, the duration loss for a syllable (de-syllable level), loss of syllable duration (L-SD) is designed to reinforce the similarity between the ‘syllable duration for reference answer’ and ‘the sum of the predicted duration of all phonemes in that syllable as predicted by the model [31].’ These are as follows: (2) Ldur=wpd*Lpd+wsd*Lsd {L_{dur}} = {w_{pd}}*{L_{pd}} + {w_{sd}}*{L_{sd}}

This is the loss of duration prediction. L-pd refers to the loss of phoneme duration and L-sd refers to the loss of syllable duration. W-pd and W-sd are their respective losses.

Its main work is to ‘extend’ the length of the ‘vector sequence obtained by the encoder’ according to the predicted tone length [32]. For example, if a phoneme lasts for 5 s (‘Ah... Ahh’), then the regulator expands it several times according to how many time slices correspond to 5 s (say, 20 ms time slices) (here 5000/20=250 times). The results for the encoding vector sequence after length extension are as shown in Figure 7.

Fig. 7

The decoder, shown in Figure 8, is responsible for generating acoustic features from an expanded sequence of coding vectors [32, 33].

Fig. 8

In this paper, the World Vocoder is used, which requires the decoder to be able to predict mel-generalised coefficient (MGC) and band aperiodicity (BAP), not mel spectrum. The definition of loss here is as follows: (3) Lspec=wm*Lm+wb*Lb {L_{spec}} = {w_m}*{L_m} + {w_b}*{L_b}

The left side represents the loss in spectral parameters. On the right, L-M stands for loss of MGC; L-b indicates loss of BAP [34].

W-m and W-b are their respective weights. Compared with speech, singing has a more complex and sensitive F0 profile. For example, it has a wider range from 100 Hz to 3500 Hz. The dynamics of F0 movements, such as trills and overtones, help convey emotions more expressively [35].

Some prior research has shown that even a slight deviation from standard pitch can seriously damage the listening experience. On the other hand, it is difficult to cover all pitch ranges in sufficient cases using training data. This means that F0 prediction can be problematic if the pitch of the input note is not displayed or is rarely present in the training data. Data enhancement solves this problem by implementing pitch conversion on training data. But this is not economical and can lead to longer training sessions. Instead, we propose a residual connection between the input and output pitches (here, we use logF0, the logarithmic scale of F0) so that the decoder only needs to predict human bias based on the standard note pitch, which is more robust for rare or invisible data. Later experiments confirmed this [36, 37].

The prediction of F0, as usual, is accompanied by the voiced/unvoiced (V/UV) decision. Since the V/UV decision is binary, a logistic regression is used here. Thus, the loss function of the final decoder is as follows: (4) Ldec=Lspec+wf*Lf+wu*Lu {L_{dec}} = {L_{spec}} + {w_f}*{L_f} + {w_u}*{L_u}

Here, L-f stands for loss of logF0; L-u stands for loss of V/UV decision and W-f and W-u are their respective weights.

Fig. 9

The data included 1000 pieces of Chinese Mandarin bel canto music, with one girl singing, in a professional recording studio at 48 Hz and under 16-bit quantisation. Then, it underwent cuts for no more than 10 s. We ended up with 10,360 segments with 70 h of data. From this pool, 8756 fragments are used as training sets, and the remaining 1032 are used as validation sets and test sets. Acoustic features were extracted by World: 15 ms frame shift, 60-D MGC, 5-D BAP, 1-D logF0 and 1-D V/UV flag. The duration label (integer) of the phoneme is obtained by a hidden Markov model (HMM) forced alignment (probably something like HM-based speech synthesis system [H-triple-S or HTS]). In XiaoiceSing (Figure 1), six layers of FFT blocks are shared. The phoneme word list size is 72. Encoder output is a 384-D vector (one vector per phoneme); decoder output is 67-D acoustic signature. The loss of MGC, BAP and logF0 is calculated separately using L1 regularisation. The V/UV decision uses binary cross entropy loss, Adam optimiser NVIDIA P100 GPU, a batch size= 3230K iteration step convergence.

In this experiment, baseline and XiaoIceSing systems were compared to evaluate overall performance. We first conducted MOS test on the two systems. For each system, we prepared 30 audio samples, each completed in about 10 s. We asked 10 listeners to rate the MOS score of each sample based on articulation accuracy, tone quality and naturalness. The MOS score is the average of all samples to obtain the final score (see Table 1). The MOS score for pronunciation accuracy was 4.52, close to the recorded score of 4.89 and 1.28 higher than the baseline. The sound quality and naturalness also improved by 1.54 and 1.48, respectively, from the baseline. The standard deviation on the MOS was smaller than the baseline, indicating better performance stability across all test cases.

Some objective indicators were further calculated (Table 2). The quality of the synthesised song was measured by root mean square error of duration (Dur RMSE), duration correlation (Dur CORR), F0 RMSE, F0 CORR, mel cepstral distortion (MCD), band aperiodic distortion (BAPD) and voiced/unvoiced error rate (V/UV error). It shows that XiaoiceSing achieved lower RMSE and higher CORR in both duration and F0 than the baseline. Meanwhile, MCD, BAPD and V/UV errors were lower than baseline for XiaoiceSing. We can see that small ice stars have a greater ability to generate accurate phoneme duration, F0 and spectral features.

Experimental results show that compared with the baseline system, it has a great advantage in terms of sound quality, pronunciation accuracy and naturalness. In particular, F0 perception performs very well due to the residual linkage between note pitch and predicted F0, and the improvement in duration prediction is significant due to the addition of a syllable duration constraint between the expected syllable duration and the predicted phoneme duration.