Multimodal sentiment analysis for social media contents during public emergencies
, , , oraz
25 sie 2023
O artykule
Kategoria artykułu: Research Paper
Data publikacji: 25 sie 2023
Zakres stron: 61 - 87
Otrzymano: 07 lis 2022
Przyjęty: 05 maj 2023
DOI: https://doi.org/10.2478/jdis-2023-0012
Słowa kluczowe
© 2023 Tao Fan et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Figure 1.
Figure 2.
Figure 3.
Figure 4.
Figure 5.
Figure 6.
Figure 7.
Figure 8.
Figure 9.
Figure 10.
An instance of Weibo dataset
| Images | Texts (translated) |
|---|---|
| Bauhinia in front of the window: “Typhoon Mangosteen” violently cracked its spine, but still couldn’t hide its charm |
The first-round annotation results of Weibo dataset
| 2-person consistency | 3-person consistency |
|---|---|
| 2003 | 1905 |
An example of multimodal posts on SMPs during public emergencies
| Image | Text |
|---|---|
| On the day of the “Bus Crash”, I also took on a bus in the urban area of Chongqing. As long as one passenger on the bus came out to stop that, it wouldn’t happen today. Rest in peace. |
The first-round annotation results of Twitter dataset
| 2-person consistency | 3-person consistency |
|---|---|
| 2,961 | 2,703 |
The results of DMFM_conv11, DMFM and DMFM_conv13
| Model | P(%) | R(%) | F1(%) |
|---|---|---|---|
| DMFM conv11 | 84.701 | 84.742 | 84.695 |
| DMFM | 85.865 | 85.915 | 85.881 |
| DMFM conv13 | 84.266 | 84.272 | 84.118 |
An example of Twitter dataset
| Images | Texts |
|---|---|
| The US and China may be nearing a trade deal. That won’t stop the global economic slowdown |
The results of different multimodal sentiment analysis models
| Model | P(%) | R(%) | F1(%) |
|---|---|---|---|
| Feature-level fusion | 79.903 | 79.288 | 79.496 |
| DMFM | 85.865 | 85.915 | 85.881 |
| Decision-level fusion | 84.010 | 83.960 | 83.936 |
The final annotation results of Twitter dataset
| Positive | Neutral | Negative |
|---|---|---|
| 1,406 | 1,148 | 1,190 |
The results of different textual sentiment analysis models
| Model | P(%) | R(%) | F1(%) |
|---|---|---|---|
| SVM-T | 78.107 | 77.494 | 77.465 |
| TSAM | 82.760 | 82.864 | 82.750 |
| BERT-T | 81.333 | 81.153 | 81.227 |
The final annotation results of Weibo dataset
| Positive | Neutral | Negative |
|---|---|---|
| 712 | 768 | 649 |
The results of different visual sentiment analysis models
| Model | P(%) | R(%) | F1(%) |
|---|---|---|---|
| UFT | 57.051 | 57.042 | 56.357 |
| Conv11 | 58.906 | 58.227 | 58.228 |
| Conv12 | |||
| Conv13 | 57.861 | 57.512 | 57.625 |
The results of Twitter public emergency dataset, where T represents text and V represents image_
| Model | P(%) | R(%) | F1(%) |
|---|---|---|---|
| T | 83.407 | 83.482 | 83.386 |
| V | 62.336 | 61.867 | 60.934 |
| T+V | 86.463 | 86.400 | 86.401 |