Uneingeschränkter Zugang

Bridging the computational and visual turn: Re-tooling visual studies with image recognition and network analysis to study online climate images


Zitieren

Introduction

Images hold certain qualities that make them powerful in communicating issues like climate change: making visible, tangible, and relatable what is otherwise complex, abstract, and distant (O’Neill, 2013; Rose, 2016). However, while seemingly presenting the real-world one-to-one, images are neither objective nor neutral, but play an important role in framing an issue. While this has made climate images a subject of growing scholarly interest, most studies have focused on iconographic motifs (Manzo, 2010; O’Neill & Hulme, 2009), as well as images from established news media (Ahchong & Dodds, 2012; Kangas, 2019; O’Neill, 2013), nongovernmental organisations (Doyle, 2007), and scientific publications (Mahony & Hulme, 2012). Climate images circulating on social media platforms have, with few exceptions, so far been largely overlooked. This is curious, since images shared on social media have a dialectic relation to larger sociopolitical events in society, simultaneously reflecting broader trends and serving as a powerful tool for the public's engagement in these events. They impose particular ways of “seeing the world” (Rose, 2016) and engage audiences by evoking emotions, facilitating memory, and transmitting cultural meaning (O’Neill & Nicholson-Cole, 2009).

Combining visual studies and digital methods

While some visual studies of climate change have examined social media images (Hopke & Hestres, 2018), they have done so mostly from a single-platform approach, while looking at a snapshot moment in time. With rare exceptions (Niederer & Colombo, 2019; Pearce et al., 2020), this leaves both a gap of cross-platform research on how climate change is visually debated on different social media platforms, and a gap of temporal frameworks that study how the visual social media debate develops over time. Moreover, a majority of earlier studies have used qualitative, small-sample, or manual approaches that are not well equipped to navigate the large number of digital images that are produced and shared online. This has been problematised by, for instance, visual scholar Gillian Rose (2016), who calls for development of novel methodologies that incorporate the computational into visual studies.

Meanwhile, scholars in digital research have for years pioneered the use of digital tools to study large-scale patterns in climate change communication on social media. A valuable example can be found in Munk (2014), who uses Facebook data to map how climate disputes about wind energy unfold online. This study is inspirational in showcasing the potentials of using digital methods together with network analysis for “controversy mapping” (Latour, 2005a; Marres, 2015; Rogers, 2013). Another project by Marres and Gerlitz (2016) studies the liveliness of the climate debate on Twitter, demonstrating the potential of using digital tools to capture its changing dynamics. Both of these studies – along with others from digital social science – show great potential for using computational tools to study online climate communication. Meanwhile, these and other digital studies (Jang & Hart, 2015; Veltri & Atanasova, 2017) have primarily analysed the climate debate through social media texts, since tools for processing images – compared with natural language processing – have taken longer to refine. This has created a bias towards Twitter studies and textual data, and left visual platforms like Instagram critically understudied, as highlighted by Highfield and Leaver (2016). The lack of big-scale studies of visual content from social media, we argue, makes it crucial to start bridging the computational turn and visual turn of social science, exploring how tools like image recognition can expand digital social science to include visual data.

The article at hand takes up this challenge of developing a digital-visual methodology that opens up opportunities for the study of large-scale visual data with cross-platform and temporal sensibilities. We develop and explore the application of such a methodology through a two-fold case study. First, we set out to generate insights on what visual motifs are mobilised in climate images on Twitter and Instagram, focusing on how the human–nature relation is visually depicted on the two platforms, following recurring scholarly attention to this divide in shaping the climate issue (Latour, 2004; Morton, 2007). Second, we explore temporal patterns in how the debate has changed on Twitter from 2015 (when the Paris Agreement was established), through 2016 (when former President Trump was elected and popularised the notion of “climate hoax”), until 2017 (when Trump announced the US withdrawal from the Paris Agreement). It is imperative, we think, to ask how online climate communication has responded to and changed in light of these significant political events.

Method and data

The methodological anatomy of this project is an exploratory and descriptive, rather than explanatory, social science, following Latour's (2005b) call for shifting focus away from underlying causes and all-explaining theories. Taking a methodological path, this project situates itself within digital methods, an emerging field in social science concerned with re-purposing digital platforms for studying the social through the growing availability of large-scale digital data (Marres & Gerlitz, 2016; Rogers, 2013). As a framework, digital methods does not prescribe a certain tool or method, but implies redirecting focus away from methodological separations of the qualitative and quantitative, which have been heavily problematised within empirical social science (Savage & Burrows, 2007). Instead, digital methods urge us to leverage advances in computational tools and the granularity and scalability of digital data to take a “quali-quantitative” approach that moves between micro and macro levels in data, reconciling the quantitative dimensions of large-scale data with the qualitative sensibilities needed to understand it (Lindgren, 2020; Ruppert et al., 2013; Venturini & Latour, 2010). Adhering to this, we assemble a combination of digital tools to collect, code, and visualise data in a way that empowers us to map cross-platform and temporal dynamics in climate images.

Data for the project was collected in December 2018. Netlytic (Gruzd, 2016) was used to capture Instagram data and Twitter Capture Analysis Tool (TCAT) (Borra & Rieder, 2014) to collect from Twitter,

For a longer debate on public/private in post-API Internet research, we refer to Freelon (2018) and Perriam and colleagues (2019), but rely here on the fact that at the time of collection the data used in this project was made public both by the users and platforms through open APIs.

using search words #globalwarming and #climatechange. Noticeably, these hashtags are not used in every post that talks about climate change on social media; thus, we only study parts of the online debate. We have three reasons for focusing on the two hashtags. First, “global warming” and “climate change” are the two most frequently used keywords to refer to the overarching issue of climate change. Focusing on them is a deliberate strategy that allows us to map more openly and bottom-up what issues are mobilised by users within the climate debate, without deciding a priori what issues (like #windenergy, #biodiversity, etc.) might be important in the debate. Second, focusing on these two hashtags is part of a pragmatist Dewey-inspired issue-specific approach to charting the public's involvement in the climate debate. In this perspective, a public is not construed as an a priori entity existing independently; rather, “issue publics” are seen as ontologically emergent formations that are sparked into being through people's concern for and engagement with an issue (Dewey & Rogers, 2012; Latour, 2005a; Marres, 2005). Digital methods scholars like Birkbak (2013), Marres (2012), Munk (2014), Venturini (2010), and others have brilliantly shown how we can think of online networked media as arenas where issue publics emerge, while Bruns and Burgess's (2015) notion of “hashtag-publics” helps us frame hashtags as socio-technical objects that organise such a public's involvement in an issue. We use this to make operational our study of issue publics on Twitter and Instagram who organise around #climatechange and #globalwarming.

To enable a temporal analysis of Twitter images in relation to COP21 (United Nations Climate Change Conference) in 2015, COP22 and the simultaneous election of Trump in 2016, and the US withdrawal from the Paris Agreement in 2017, we collected data from Twitter in 30-day periods around these events (see the exact dates in Table 1). Data collection was done in collaboration with Digital Methods Initiative at University of Amsterdam, who kindly gave access to their TCAT server with historic Twitter data on the climate debate. To enable a cross-platform comparison of the 2017 debate, we collected a 2017 sample from Instagram. Due to Instagram's application programming interface (API) not allowing historic data collection, the two platform samples are not from the same dates. While this reduces the 1:1 comparability of our data samples from Twitter and Instagram, we still find them useful for our purpose, since they are both collected in a 30-day period, and both from periods after the US withdrawal announcement, enabling us to study how this event influenced the debate on both platforms. This matter of collecting historic data, however, points to a larger challenge in all digital research, where scholars increasingly experience restricted access to data from the big-tech platforms, making it difficult to investigate any part of public life that takes place online. This emphasises how big an influence tech giants behind these platforms have in shaping our epistemic processes and what can be known (Ben-David, 2020) – an issue made only more pressing by the recent closure of APIs, which has made it even more difficult to curate data from social media platforms (Bruns, 2019; Freelon, 2018). As seen in Table 1, the data collection results in a total dataset of 262,093 image posts.

Overview of data sample from Twitter and Instagram

Platform Year Date Related political events Posts with images
Twitter 2015 29 November–28 December COP21: 30 November–12 December 169,220
Twitter 2016 29 October–27 November COP22: 7–18 November 31,931
Twitter 2017 18 May–16 June US withdrawal from Paris Agreement: 1 June 46,567
Instagram 2017 4 October–2 November US withdrawal from Paris Agreement: 1 June 14,375

Comments: Twitter data is collected through Twitter Capture Analysis Tool (TCAT) and Instagram data is collected through Netlytic. All four data samples are collected from the criteria that all posts must contain either #climatechange or #globalwarming. Posts without images or other media content like video have been excluded from the datasets.

To analyse the content of these images, we used the image recognition software of Google Vision AI to annotate all images. Specifically, we used the AI's object detection feature to identify visual content (Google Cloud Platform, 2018), returning a list of objects detected for each image. For simplicity, we call these “image-objects” and reference them with the “ ” symbol. The Vision AI further supplies each image-object with a confidence score from 0 to 1, which we use to filter out objects with a confidence score of less than 0.75. This does not mean we should blindly trust the algorithm to correctly annotate all images. Even though Google's AI has been tested to be more accurate than others (Oberoi, 2016), it has also appropriately been criticised for having racial biases, such as annotating Black people as “gorillas” (Simonite, 2018) or being systematically biased on gender (Schwemmer et al., 2020). The confidence score somewhat improves reliability by giving us a mechanism through which to rule out the most uncertain coding by the AI. It is crucial, however, to remark that this does not remove or lay bare all potential biases. Because of that, we propose that the image-objects should never be self-explanatory. The AI does not deliver the analysis; rather, we used the AI's coding to point us in the direction of large-scale trends in the data, which we explored in depth with a qualitative look at images that have been annotated with certain content. Thus, reliability of our results is not based on assumed algorithm accuracy, but from taking responsibility for interpreting patterns in the coded data via a qualitative look at image samples.

With inspiration from scholars like Ricci and colleagues (2017) and Niederer and Colombo (2019), we use visual network analysis (Venturini et al., 2019) to map how image-objects appear together in climate images, leveraging network analysis both as a heuristic tool to get an overview of our datasets, and to offer an illustration of network analysis findings (Bastian et al., 2009; Jacomy et al., 2014). With this method, we operationalise a re-orientation of visual analysis away from looking at individual images as stand-alone, confined entities (Rose, 2016), and on to seeing images as assemblages of visually related objects. Rather than analysing visual content of images one by one, we look across all our images and analyse how visual objects appear across them. Figure 1 exemplifies how we use network analysis to compute how image-objects can be seen as related to each other if they appear in the same images.

Figure 1

Creating a network graph with images annotated by Google Vision AI

Comments: The figure shows how Google AI might annotate image-objects in two images and recognise “Earth” in both them. If we map this as a network (as seen on the right side of the illustration) it becomes easier to grasp that across these images, the motif Earth is a shared visual object. Qualitative examination shows it is framed differently in the two images: in one, Earth is related visually to “sky” and “energy”, causing a frame of a sustainable world with clean wind energy; in the other, it is visually depicted as a burning world, creating a more dystopian framing. The network helps us see overall patterns in how these image-objects appear together and structure our analytical gaze to, for instance, zoom in on “Earth” as a potentially disputed object. Our quali-quantitative analysis is thus enriched and informed by how the network makes legible the appearance of image-objects across multiple images.

Source: Authors’ conceptual illustration

This relational network approach is key to the first part of the case study, where we explore the following question: How are motifs of humans and natural environments depicted together in Instagram and Twitter images? In the second part, we use streamgraphs to map a temporal question: How have visual genres of climate images changed on Twitter from 2015 to 2017?

In the case study, we situate image recognition within a quali-quantitative approach (Venturini & Latour, 2010), which entails a fluid analytical movement between charting large-scale patterns and quantified data visualisations and zooming in on examples of visuals with specific image-objects. Specifically, we display 71 images through the analyses. The selection of images is based on a combined quali-quantitative logic: First, we use network analysis to quantify occurrence and relations of the visual content and identify image-objects that spark analytical curiosity. Then, we qualitatively investigate how the given image-object appears in images, looking at all or up to 100 images in which the Vision AI has identified the image-object of interest. From the sample, we select a handful to display in the analysis, with the same logic that a researcher selects quotes from interviews to represent a common or important viewpoint in the interview.

More specifically, the selection of images is throughout the project based on qualitatively exploring a multiplicity of images that have been annotated with certain image-object of interest to our analysis. The process involves us looking at the networks in Figure 2 and Figure 3 and finding image-objects that seem central to a visual theme. In the Nature cluster on both Twitter and Instagram networks, we for instance see that sky is a central and big node, indicating that among motifs of nature, a sky has often been detected in climate images by Google Vision AI. To explore examples of such images, we access the coded data and filter it for all images that contain sky. Looking at up to 100 of these manually, we select and put into hexagonal figures a hand-picked selection of image examples that illustrate the typical tendencies or themes we find in this type of image.

It should be mentioned that our aim is not to analyse each image in depth – as is the tradition in most visual research (Rose, 2016) – but to give examples of the cross-platform and temporal patterns at scale. Images are therefore shown in hexagonal series to standardise their presentation and put analytical focus on multiple images that share a motif or genre. We anonymise metadata and usernames, but not the images themselves.

This project only uses photos already made public by the users themselves, meaning that we do not publish any information that is not already public.

Cross-platform analysis

In the first part of the case study, we investigate how image recognition and a network approach can be used as a way of analysing and comparing visual content across social media platforms. We compare 14,375 images from Instagram and 46,567 from Twitter, exploring how the human–nature relation was depicted in the climate debate after the US announcement of withdrawal from the Paris Agreement in 2017. To do so, we built two network graphs – computed with Gephi (Bastian et al., 2009) – that display as nodes the image-objects detected by the AI, while drawing connections between objects if they co-appear in images. The layout is spatialised with the ForceAtlas2 algorithm (Jacomy et al., 2014), which pulls image-objects that often appear together in images closer to each other in the map, hereby giving analytical meaning to node positions and topology of the networks. Node sizes represent the quantified frequency of image-objects: how often an image-object is featured in the overall sample. Finally, a modularity class algorithm is run on both networks, dividing it into clusters of nodes that are more strongly connected to each other, and hence represent image-objects that more often appear together in images. Figures 2 and 3 show the networks for each platform dataset (high resolution images are provided in the supplementary file).

Figure 2

Network of image-objects and their co-appearances in Instagram images

Comments: The network consists of 1,607 nodes and 24,211 edges, where the nodes represent image-objects detected in the dataset of 14,375 Instagram posts. The network is computed in Gephi and spatialised with the ForceAtlas2 algorithm, filtered by setting occurrence count to minimum 2 and adding the Giant Component filter. Sizes of nodes reflect the frequency of the image-objects in the Instagram data, and nodes are clustered by a Modularity Class algorithm.

Source: Instagram data, visualised with Gephi

Figure 3

Network of image-objects and their co-appearances in Twitter images

Comments: The network consists of 1,582 nodes and 24,142 edges, where the nodes represent image-objects that have been detected in the 2017 dataset of 46,567 Twitter posts. The network is computed in Gephi and spatialised with the ForceAtlas2 algorithm, filtered by setting occurrence count to minimum 2 and adding the Giant Component filter. Sizes of nodes reflect the frequency of the image-objects in the Twitter data, and nodes are clustered by a Modularity Class algorithm.

Source: Twitter data, visualised with Gephi

Looking at the overall structure of the networks, the 1,582 image-objects on Twitter and 1,609 on Instagram are subdivided into seven clusters in each network. When investigating the image-objects in each cluster, we find that the seven clusters identified in the Twitter network are fairly similar to the ones identified in the Instagram network, both containing clusters of visual objects related to people, natural environments, urban environments, fauna, flora, food, and graphic elements. Figure 4 shows an interpretative diagram of the networks, outlining the clusters that we name Human, Nature, Urban, Fauna, Food, Flora, and Graphic.

The networks show that Fauna, Flora, and Food are clusters of visual content important to users on both platforms when discussing climate change. Of interest to our study, however, Figure 4 reveals that the three biggest clusters in both networks are Human, Nature, and Urban, indicating that these visual themes are central to the debate on both platforms. Since this methodological experiment focuses on exploring the human-nature relation, we zoom in on these three clusters: Interestingly, we see that Human is positioned opposite of Nature and Urban, preliminarily suggesting that on both platforms, visual motifs of humans do not often co-appear with motifs of natural environments.

While the thematic similarity of clusters in the two networks initially suggests that the visual debates on Instagram and Twitter are similar in content, a closer investigation reveals how several image-objects only appear on one platform, or appear very often on one but rarely on the other. Looking at the Human cluster, we find that some of the most frequent image-objects in the cluster on Twitter are protest and demonstration, which are not detected in a single image from Instagram. These motifs typically appear in Twitter images together with motifs of public-speaking, speech, profession, and official. As exemplified in Figure 5, it is often political agents of some kind that we find on such Twitter images. In the Instagram network, on the other hand, we find a different visual frame of humans oriented towards social life with motifs of fun, vacation, and leisure – objects that are not present in images on Twitter. Photos with these objects are exemplified on the right side of Figure 5, suggesting that humans depicted on Instagram are typically the users themselves, shown in selfies, holiday pictures, or everyday situations.

Twitter images thus frame human engagement in the climate issue as a primarily political one, while Instagram images frames human engagement more in social terms, with more imagery tied to everyday life. This confirms what others have shown in describing the identity of these platforms (Hu et al., 2014; Mislove et al., 2011). A common feature across platforms, however, is that people are rarely depicted in direct relation to nature or to the climate causes or consequences, thus communicating climate change as a seemingly remote issue.

Figure 4

Twitter and Instagram network clusters

Comments: The figure illustrates the clusters identified in the Twitter and Instagram networks recognised with the Modularity Class algorithm in Gephi, also shown in Figures 2 and 3. The illustration shows the seven main clusters in each network, and how they are positioned relative to each other. Clusters are highlighted by coloured polygons placed over the nodes, outlining each cluster. Cluster titles are based on qualitative readings of the visual theme of each cluster.

Source: Authors’ interpretative illustration

Figure 5

Twitter images annotated with demonstration, protest, speech, profession, official, and diplomat (left) and Instagram images annotated with fun, eisure, vacation, and community (right), 2017

Unfolding the differences between the platforms further, we zoom in on the Urban and Nature clusters in both networks. The Nature cluster makes up 21 per cent of the Instagram network and 19 per cent of the Twitter network. The largest image-object in the Nature cluster in both networks is sky. If we qualitatively investigate a sample of images that contain this motif, it becomes clear that sky is depicted very differently on the two platforms, as Figure 6 illustrates.

Figure 6

Twitter images (left) and Instagram images (right) annotated with sky, 2017

On Twitter, users mobilise frames of a polluted nature, putting sky in relation to smokestacks and industrial buildings, displaying the urban environment as a source of destruction of nature. Contrary to this, sky visuals on Instagram are dominated by beautiful and idyllic images of nature as un-touched by humans with no visible traits of urban society. The difference in depiction of nature becomes even more evident when looking at images of disaster, wildfire, natural-disaster, and earthquake, which are detected in several hundreds of images on Twitter, but only in four images from Instagram. Looking at Twitter disaster-images in Figure 7, we see they consistently show urban environments destroyed: crushed, drowned, or burned by earthquakes, hurricanes, and wildfires.

Figure 7

Twitter images annotated with disaster, natural-disaster, wildfire, or earthquake, 2017

It is noteworthy that in the disaster-images from Twitter, humans are very rarely depicted. Instead, it is the built environment which is shown in relation to climate destruction. As such, we find two parallel visual framings of the relation between nature and urban; on one hand, the build environment is causing climate change through pollution, on the other, it is also urban environments – and not humans themselves – that are visually depicted in relation to the consequences of climate change, for example, with floods, fires, and earthquakes. The urban hereby acts as a stand-in symbol of people in depictions of the human-nature relationship.

Zooming in further on image-objects at the border between the Urban and Nature clusters in the two networks, we identify a motif of tourism, which might indicate a specific form of human agency in relation to climate change. Figure 8 displays a sample of tourism-images from both platforms and speak further to the visual framing of human–nature relation across platforms.

Figure 8

Instagram and Twitter images annotated with tourism, 2017

The images of tourism in Figure 8 suggest a rare but specific context where humans are depicted in direct relation to nature – on both platforms. In these images, nature is shown as an attraction that humans visit. Here, then, we finally see a direct visual link between humans and nature. But, it is one that frames humans as tourists, who are by definition visiting a place they do not inhabit, sustaining the frame of the human–nature relation as dichotomist and alienated.

Summarising this analysis, our approach has produced multiple insights on the human–nature relation that demonstrate the potentials of the methodology to open up opportunities for visual cross-platform analysis: First, the macro structure of the Instagram and Twitter networks showed the Human cluster positioned on the opposite side of the map from Nature, indicating that image-objects from these clusters are not often depicted together. Second, a look into Human cluster images revealed that while different frames of human agency proliferate on the two platforms, humans are rarely depicted in direct relation to climate causes and consequences on either of them. Similarly, explorations of Nature cluster images revealed platform-specific differences in an idyllic (Instagram) versus dystopian (Twitter) frame of nature, meanwhile revealing a consistent frame across platforms of an alienated human–nature relationship, with few people displayed in images of nature or images depicting climate causes and consequences. The method has thus made platform differences legible, while revealing a consistent cross-platform visual frame where climate change is depicted as a remote issue. Finally, tourism-images from both Instagram and Twitter confirmed this further by showing humans as visitors in nature, sustaining an alienated imaginary.

These findings add to existing discussions of the human–nature divide that can be found in a wide range of literature, such as Latour (2004, 2012), Ricci and colleagues (2017), Morton (2007), and many others. For the agenda on climate communication specifically, these findings could add to studies of problematising climate visuals, where authors like O’Neill and Nicholson-Cole (2009) have shown that although fearful images attract attention to climate change, fear leaves audiences overwhelmed by the issue, concluding that climate images can either make people feel the issue is important or make them feel they can do something about it. Our analysis could provide a fresh empirical perspective on this saliency/efficacy trade-off. But importantly for this article, the analysis has demonstrated the methodology's potentials for making new empirical routes possible in the study of online visual debates, through combining image recognition and network analysis to map images in large-scale data across platforms. Here, an important move has been to use a quali-quantitative lens to make sense of the image recognition annotations in networks, making overall patterns quantifiable and navigable in a way that structures our analytical gaze and informs where to make targeted qualitative deep dives. Meanwhile, the image recognition annotations do not speak for themselves, and the analysis underlines the importance of investigating them qualitatively, since it was through investigation of image samples, such as images of sky, that we could unpack how the same motif is mobilised in different framings across platforms.

Temporal analysis

In the second part of the case study, we explore how image recognition can be used to trace specific image-objects over time and provide a temporal perspective on the climate debate. From interpreting the Twitter network in Figure 3, we were particularly curious about the Graphic cluster, which contained image-objects such as photography, calligraphy, logo, diagram, poster, comic, drawing, advertising, and art – objects that could be indicative of visual genres of images (Rose, 2016). There was also a particular big node – cartoons – which, although placed in the Human cluster, was strongly tied to nodes in the Graphic cluster, like comic, comicbook, and drawing. To investigate this further, we explored selected genre-related image-objects over time, focusing our temporal analysis on three historic, climate-related political events: COP21 in 2015, COP22 and the election of Trump in 2016, and the US announcement of withdrawal from the Paris Agreement in 2017. Figure 9 shows the amount of Twitter posts collected around each event in 2015 (169,220), 2016 (31,931), and 2017 (46,567).

Figure 9

Twitter data samples divided in eight periods, 2015, 2016, and 2017

Comments: The data is the three Twitter samples from Table 1 mapped in an area graph displaying the number of daily tweets with images containing either #climatechange or #globalwarming. The figure illustrates how the three data samples from 2015, 2016, and 2017 have been divided in eight sub-periods (A–H) around the climate-related political events.

Source: Twitter data, plotted in Excel

Divided into eight periods (A–H), this data allows us to investigate the visual debate before, during, and after each political event. We analyse four selected image-objects connected to the Graphic cluster – cartoon, advertising, diagram, and art – which seem to represent four different visual genres used to communicate the climate issue. We tracked how often these image-objects appeared in Twitter images in each period – before, during, and after an event. To compare the different periods, we normalised the number of image-objects in a period against the total amount of tweets in that period. Figure 10 visualises the result with a sorted streamgraph, which shows proportional occurrence of image-objects across time as streams, ranking the most-occurring on top.

Figure 10

Streamgraph of four genre-related image-objects on Twitter, 30-day intervals 2015–2017

Comments: Data is all images from Twitter containing one of the four image-objects cartoon, advertising, diagram, or art from all data samples. The sorted streamgraph is made with Raw-Graph, which normalises the number of images in each period to make time periods comparable, and places the image-object that appears most in each period on top.

Source: Twitter data, plotted via RawGraphs (Density Design Research Lab, 2013)

What is immediately noticeable in the graph is that advertising and diagram were detected in a large proportion of images in 2015, while their relative share decreased dramatically during 2016 and 2017. With our quali-quantitative approach, this points us in a direction of a larger trend that we unpack by qualitatively studying examples of these visual genres, starting with climate imagery from 2015 of advertising, seen in Figure 11.

Figure 11

Twitter images annotated with advertising, two 30-day periods 2015

The images which the algorithm characterises as advertising are typically cropped photographs with big letters, banners of text, and logos. As seen in Figure 11, this visual genre uses a lot of text – from photographs of protesters holding up signs, to photos of people passing by a huge banner, to digitally produced images that advertise the COP21 event. They are not necessarily commercial, but share some of the same visual traits as advertisements. The prominence of advertising reveals that in 2015, climate imagery was characterised by promotion of the climate agenda through text- and information-heavy visuals containing motivational quotes and insights along with calls for action. In 2016 and 2017, however this genre of image is not as prominent as in 2015.

In 2015, we also see the prominence of climate imagery labelled with diagram. The relative proportion of diagram images, however, decreases during 2016 and falls dramatically in the middle-period of 2017 during the US withdrawal from the Paris Agreement. Interestingly, the declining trend of diagram coincides with a growing use of cartoon and art images in 2017. Comparison of images with diagram and cartoon is seen in Figure 12.

Figure 12

Twitter images annotated with diagram (left), 2015, and cartoon (right), 2017

Figure 12 exemplifies what the uptake of cartoons and downfall of diagrams means for how the visual genre has changed on Twitter from 2015 to 2017. Where scientific diagrams are earlier used to communicate complex numbers and graphs that are hard to read, cartoons are used to communicate in a more political, emotional, and satirical way that is easy to understand. Also, the subject of these two forms of communication is different, with diagrams typically debating climate-specific science, while the favourite subject of cartoons is political figures such as Donald Trump and Barack Obama.

Similar to cartoons, the image-object art goes from being rarely detected in climate imagery from 2015 to being more often detected in images from 2017. The visual genre of these images is characterised by colourful paintings with no text. While the art images are more abstract and surrealist than the cartoon images, as seen in Figure 13, they use a similar symbolism and politicised motifs like smokestacks and Trump.

Figure 13

Twitter images annotated with art, 2017

Interestingly, we see that the art and cartoon genres, which proliferated in 2017, is used by both supporters and deniers of climate change. This is visible in in Figure 12, where cartoons both make fun of and celebrate Trump, indicating that cartoons open for a high level of conflict in the debate. While scientific diagrams are technical and speak to objectivity and reason, cartoons speak to the emotions of the viewer and are known for using irony, sarcasm, and exaggeration as a subversive form of communication (Baym, 2005). Because of this, the cartoon genre has historically been used as a means of political resistance and critique of the existing social order (Rodrigues & Collinson, 1995), something we see here, too. But what our analysis suggests is that the established order subverted by cartoons is the logic of communication itself, where the scientific communication proliferating in 2015 is challenged as the dominant visual genre for debating climate change in 2017. This is made very clear in images of cartoons in Figure 12, which depict Barack Obama as a fortune teller and as blindly giving money to climate scientists. This subversive use of cartoons could indicate that science itself has become politicised: rather than simply disagreeing on climate change, it seems to increasingly be science itself which is turned into a matter of political disagreement in 2017.

In summary, this analysis has highlighted a shift in how climate change is visually debated on Twitter, where, from 2015 to 2017, the scientific visual genre of technical and fact-oriented diagrams lost its dominant position. The shift towards the use of cartoons suggests a change in the dominant types of engagement in the climate issue: where diagrams visually construct climate change as a scientific issue that can be technically measured and rationally debated, cartoons frame climate change as an emotional and political issue: as something you can believe in, make fun of, or have subjective opinions on. In showing this, our findings contribute to a lot of ongoing discussion in both the public and academic literature, where attention is placed on investigating the rise of right-wing populism, spread of fake news, and nature of political participation on social media (as discussed in, e.g., Effing et al., 2011; Rogers, 2018). Most pressingly, perhaps, our findings raise questions about what the consequences are for public participation and deliberative democracy, when scientific visuals lose territory in the online climate debate: how can people, organisations, or societies come to consensus or mutual understanding on the climate issue if the very concept of scientific facts is destabilised as a communication form? While cartoons have historically been effective in opening up controversies, as seen with the 2006 Muhammed drawings, they are equally inefficient in closing conflicts down (Müller et al., 2009), calling for further investigation of how cartoons shape the climate debate.

In empirically opening up opportunities for such research trajectories, the method applied demonstrates a tangible way of tracing how visual content changes over time, and it provides a tool for making legible how genres of visual communication dynamically lose and gain superiority in online debates. Again, we suggest that the quali-quantitative approach was key to fruitfully bridging the visual and computational capabilities of social science: While Google Vision AI helps quantify occurrence of visual genres, the AI-powered annotation is not self-explanatory but demands contextualisation from attentive researchers, as we saw with the advertising images, while also a closer examination of cartoon images was crucial to discover that this genre is not just used by climate sceptics, but also climate advocates, indicating that this genre is used on both sides.

Conclusion

To conclude, our study makes contributions on both the empirical and methodological level. Empirically, the study delivered a cross-platform analysis of Instagram and Twitter climate images from 2017, showing both platform differences and similarities, especially revealing a consistent dichotomist and alienated depiction of the human–nature relation across both platforms that frames climate causes and consequences as a remote, distant issue. Second, a temporal analysis of Twitter showed how the dominant genres of visuals has changed from 2015 to 2017, where diagrams are substituted with cartoons as the most-used type of visual genre, replacing scientific images with a more politicised, sarcastic, and emotional genre. To advance these results further, a priority could be to extend the cross-platform approach by including other platforms, while it would also be useful to track recent developments of the climate debate from 2018 to the present moment, where, among other events, the US has re-joined to the Paris Agreement, continuously altering the debate.

Methodologically, we first argued that to capture the liveliness of complex visual debates, we need a method that can map and combine both cross-platform and temporal dynamics in visual data. Second, we used the two-fold empirical case study to demonstrate how such a method might be assembled as a digital methods approach, where the combination of image recognition and network analysis within a quali-quantitative lens opened up opportunities to study a quarter-million images from two different platforms and three different years. This, we hope, showcases some of the promises in bridging the computational and visual turn in social science, while many more should also be explored.

On the technological level, this study has only just scratched the surface of what is possible to do with image recognition. Digital and media scholars would do well to experiment more with training and building image recognitions, as well as examining the biases and built-in epistemologies of existing algorithms to show how these visual technologies shape our epistemic processes (Ihde, 2000) – experiments which have been beyond the scope of this article. Other analytical potentials should also be explored, including methods for studying relations between textual and visual data, or use of image recognition to study visual properties of image to ask questions like, what is the colour of climate change? Finally, images transcend language barriers, affording scientists an opportunity to break with the Anglo-Saxon bias – seen in our own and most other studies – and study cultural differences in how climate change is debated globally.

eISSN:
2003-184X
Sprache:
Englisch