<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.</p>
</abstract>

The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Journal of Artificial Intelligence and Soft Computing Research

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the...

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Published Online: Dec 25, 2023

Page range: 5 - 23

Received: Jun 14, 2023

Accepted: Nov 15, 2023

DOI: https://doi.org/10.2478/jaiscr-2024-0001

Keywords
convolutional neural networks, emotion recognition, audio features analysis, explainable machine learning

© 2024 Lukasz Smietanka et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Published Online: Dec 25, 2023

Page range: 5 - 23

Received: Jun 14, 2023

Accepted: Nov 15, 2023

DOI: https://doi.org/10.2478/jaiscr-2024-0001

Keywordsconvolutional neural networks, emotion recognition, audio features analysis, explainable machine learning

© 2024 Lukasz Smietanka et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
convolutional neural networks, emotion recognition, audio features analysis, explainable machine learning