<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec>
<h3>Purpose</h3>
<p>Nowadays, public opinions during public emergencies involve not only textual contents but also contain images. However, the existing works mainly focus on textual contents and they do not provide a satisfactory accuracy of sentiment analysis, lacking the combination of multimodal contents. In this paper, we propose to combine texts and images generated in the social media to perform sentiment analysis.</p>
</sec>
<sec>
<h3>Design/methodology/approach</h3>
<p>We propose a Deep Multimodal Fusion Model (DMFM), which combines textual and visual sentiment analysis. We first train word2vec model on a large-scale public emergency corpus to obtain semantic-rich word vectors as the input of textual sentiment analysis. BiLSTM is employed to generate encoded textual embeddings. To fully excavate visual information from images, a modified pretrained VGG16-based sentiment analysis network is used with the best-performed fine-tuning strategy. A multimodal fusion method is implemented to fuse textual and visual embeddings completely, producing predicted labels.</p>
</sec>
<sec>
<h3>Findings</h3>
<p>We performed extensive experiments on Weibo and Twitter public emergency datasets, to evaluate the performance of our proposed model. Experimental results demonstrate that the DMFM provides higher accuracy compared with baseline models. The introduction of images can boost the performance of sentiment analysis during public emergencies.</p>
</sec>
<sec>
<h3>Research limitations</h3>
<p>In the future, we will test our model in a wider dataset. We will also consider a better way to learn the multimodal fusion information.</p>
</sec>
<sec>
<h3>Practical implications</h3>
<p>We build an efficient multimodal sentiment analysis model for the social media contents during public emergencies.</p>
</sec>
<sec>
<h3>Originality/value</h3>
<p>We consider the images posted by online users during public emergencies on social platforms. The proposed method can present a novel scope for sentiment analysis during public emergencies and provide the decision support for the government when formulating policies in public emergencies.</p>
</sec>
</abstract>

Purpose
Nowadays, public opinions during public emergencies involve not only textual contents but also contain images. However, the existing works mainly focus on textual contents and they do not provide a satisfactory accuracy of sentiment analysis, lacking the combination of multimodal contents. In this paper, we propose to combine texts and images generated in the social media to perform sentiment analysis.


Design/methodology/approach
We propose a Deep Multimodal Fusion Model (DMFM), which combines textual and visual sentiment analysis. We first train word2vec model on a large-scale public emergency corpus to obtain semantic-rich word vectors as the input of textual sentiment analysis. BiLSTM is employed to generate encoded textual embeddings. To fully excavate visual information from images, a modified pretrained VGG16-based sentiment analysis network is used with the best-performed fine-tuning strategy. A multimodal fusion method is implemented to fuse textual and visual embeddings completely, producing predicted labels.


Findings
We performed extensive experiments on Weibo and Twitter public emergency datasets, to evaluate the performance of our proposed model. Experimental results demonstrate that the DMFM provides higher accuracy compared with baseline models. The introduction of images can boost the performance of sentiment analysis during public emergencies.


Research limitations
In the future, we will test our model in a wider dataset. We will also consider a better way to learn the multimodal fusion information.


Practical implications
We build an efficient multimodal sentiment analysis model for the social media contents during public emergencies.


Originality/value
We consider the images posted by online users during public emergencies on social platforms. The proposed method can present a novel scope for sentiment analysis during public emergencies and provide the decision support for the government when formulating policies in public emergencies.

Multimodal sentiment analysis for social media contents during public emergencies

Journal of Data and Information Science

This work is licensed under the Creative Commons Attribution 4.0 International License.

Purpose
Nowadays, public opinions during public emergencies involve not only textual contents but also contain images. However, the existing works mainly focus...

Multimodal sentiment analysis for social media contents during public emergencies

Article Category: Research Paper

Published Online: Aug 25, 2023

Page range: 61 - 87

Received: Nov 07, 2022

Accepted: May 05, 2023

DOI: https://doi.org/10.2478/jdis-2023-0012

Keywords
Public emergency, Multimodal sentiment analysis, Social platform, Textual sentiment analysis, Visual sentiment analysis

© 2023 Tao Fan et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Multimodal sentiment analysis for social media contents during public emergencies

Article Category: Research Paper

Published Online: Aug 25, 2023

Page range: 61 - 87

Received: Nov 07, 2022

Accepted: May 05, 2023

DOI: https://doi.org/10.2478/jdis-2023-0012

KeywordsPublic emergency, Multimodal sentiment analysis, Social platform, Textual sentiment analysis, Visual sentiment analysis

© 2023 Tao Fan et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Keywords
Public emergency, Multimodal sentiment analysis, Social platform, Textual sentiment analysis, Visual sentiment analysis