Swedish teenagers’ difficulties and abilities to determine digital news credibility
Published Online: Feb 19, 2019
Page range: 23 - 42
DOI: https://doi.org/10.2478/nor-2019-0002
Keywords
© 2019 Thomas Nygren et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
New media and modern journalism can, in many ways, facilitate the spread of exaggerations and lies (Del Vicario et al., 2016; Silverman, 2015). The viral spread of disinformation may foster mistrust and disconnect people (Vosoughi et al., 2018; Wardle & Derakhshan, 2017). However, digital media may also connect people across cultural and ideological borders and digital news media is pivotal to stimulate conversations and active citizenry, not least among young citizens (e.g. Carlsson, 2018; Ekström et al., 2014; Lee et al., 2013). Today, digital news from established news media and other sources are mixed and shared in social media, making it difficult to assess credibility (Fletcher & Park, 2017). This implies new requirements for both readers and society (Flanagin & Metzger, 2008; McGrew et al., 2017; McGrew et al., 2018; Silverman, 2015).
Scholars argue that teaching and learning to use online news in critical and constructive ways are absolutely essential to informed and engaged citizenship (e.g. Kahne & Bowyer, 2017; Wardle & Derakhshan, 2017). A central part in active citizenry in a digital world is
The aim of the current study was to investigate Swedish teenagers’ ability to assess the credibility of online news and how this relates to personal characteristics and beliefs, which in turn will enable us to suggest educational measures to support and scaffold young people’s abilities to assess online information.
In order to navigate online information in constructive ways, people seem to need a mix of content knowledge and digital skills. Scholars have carefully outlined different aspects of disciplinary literacy, both in theory and practice. For instance, theories of disciplinary thinking, stemming from the critical literacy of history, underpin findings of how experts and novices make sense of texts and evaluate information in many disciplines (Shanahan et al., 2011; Wineburg, 1991 Wineburg, 1998). This perspective on disciplinary literacy has also been valuable in studies of, for instance, pupils’ critical thinking across disciplines, the use of digital databases and media narratives in human rights education (Nygren, 2014; Nygren et al., 2018; Nygren & Johnsrud, 2018; Nygren & Vikström, 2013).
However, theories and methods, which have proven useful to understand and promote intellectual skills within academic disciplines, have only been used to a limited extent to study what students do when they encounter information online. Recent research indicates that people of all ages and education, also elite students and professors, may struggle to understand trustworthiness when facing online news (McGrew et al., 2017, 2018; Wineburg & McGrew, 2017). Although historians are experts at sourcing, close-reading, and corroborating information from the past, they are not necessarily experts on sourcing and corroboration in contemporary digital environments. Professors of history are experts of historical literacy, but this skill seems to be a disciplinary literacy that differs from that of fact-checkers (Wineburg & McGrew, 2017).
Drawing on these recent findings, scholars underscore the importance of updated heuristics when navigating information like professional fact-checkers do (McGrew et al., 2017, 2018; Nygren, 2018; Wineburg & McGrew, 2017). Specifically, fact-checkers note the source of information, scrutinize the evidence of the information, and corroborate information with other online sources in updated ways. This indicates a special type of disciplinary literacy that is perhaps linked to their journalistic education and experience from working with feeds of fake, biased and credible information (Wineburg & McGrew, 2017). People without this disciplinary literacy also use cues and heuristics (intellectual rule of thumb) to estimate the credibility in online environments. The heuristics may be helpful in some situations, but it is not obvious that intellectual rules of thumb from other disciplines facilitate educated people to separate false and biased news from authentic information regarding debated topics (e.g. Wineburg & McGrew, 2017). Given that critical thinking among pupils seems to be primarily a subject-specific disciplinary literacy (Nygren et al., 2018), we need to consider the possibility that pupils’ ability to evaluate digital news may be part of a disciplinary literacy, which has not been specifically addressed in school.
It is evident that this is a special challenge for education within the international framework of media and information literacy (UNESCO, 2011). Research concepts such as digital literacy, digital critical literacy, media literacy and digital competence have been used to describe many aspects necessary to access, evaluate, analyse and create information online. Clearly, it is a great challenge, both in theory and practice, for citizens to navigate online information in critical and constructive ways (Hatlevik & Christophersen, 2013; Hatlevik et al., 2015; Hinrichsen & Coombs, 2014; Kahne & Bowyer, 2017; Kirschner & De Bruyckere, 2017; Kirschner & van Merriënboer, 2013; Livingstone, 2004; van Laar et al., 2017).
We perceive civic online reasoning as a limited but central part of the complex field of media and information literacy. Civic online reasoning includes knowledge, skills and attitudes found in the social sciences and journalism in a digital world (McGrew et al., 2017, 2018). A pivotal part of civic online reasoning is the ability to determine the credibility of true, false and biased news. With an abundance of more or less biased information citizens must be able to make judgments about the credibility of that information. Citizens need to be able to identify and determine whether they trust different sources of information. Everybody needs, for instance, to be able to discern whether information is vetted by a credible journalist or comes from a source with a commercial interest (McGrew et al., 2017, 2018). Further, citizens must be able to see how texts and images based upon scientific evidence are more reliable than biased information, intended to manipulate their opinions (Wineburg & McGrew, 2017). Such skills enable citizens to engage with political and social topics in critical and constructive ways. Thus, civic online reasoning is a central part of a civic literacy, necessary for all citizens to better understand and participate in democratic conversations and decision-making in a digital world.
But this is a complicated matter and we note that prior beliefs, coherence of the message, and cognitive ability may affect people’s ability to assess the credibility of information (Lewandowsky et al., 2012). This is true not least in digital environments where, for instance, professional design, functionality, and manipulative strategies may seduce users (Francke et al., 2011; Hilligoss & Rieh, 2008; Metzger et al., 2010). Moreover, research in cognitive psychology points at difficulties in processing information correctly when it is presented sequentially, as is the case on a webpage, (Basu & Savani, 2017) and that processing is affected by the framing of the information, especially in fast decisions (Guo et al., 2017) and when the information triggers emotion (Pachur et al., 2014).
It has also been noted that attitudes to news, scientific curiosity and socio-economic factors may influence how people navigate new information, facts and digital environments (Hatlevik et al., 2015; Kahan et al., 2017; Strömbäck et al., 2013). Today, researchers find a second-level digital divide where productive ways of using digital media seem to mirror levels of education and societal inequalities (Deursen & Dijk, 2014; Hargittai, 2001 Hargittai, 2010; Hatlevik & Christophersen, 2013; Hatlevik et al., 2015; Min, 2010). Hence, we need to map out challenges and potentials for more people to navigate better in a digital news world.
Finally, recent research indicates that pupils in Sweden may be quite good at finding and sharing credible news online. Most pupils seem to find and share news from established media written and vetted by journalists (Nygren et al., 2018). When young people in Sweden self-rate their abilities to determine credibility online in questionnaires, most say that they are good at this (Davidsson & Thoresson, 2017) and approximately 80 per cent of the students in secondary and upper-secondary schools self-rate themselves as good or very good at fact-checking online (Skolverket, 2016). However, it has not previously been tested, in a Swedish context, to what extent Swedish pupils are able to determine the credibility of the source, the evidence presented and corroborate trustworthy and biased information. To better understand how to support civic online reasoning in education we need to delineate pupils’ abilities and inabilities to determine the credibility of digital news – credible, biased and false.
To assess the abilities of pupils to determine credibility of online news, researchers in education and psychology designed and piloted, in close collaboration with in-service teachers, an online survey with test items. Ideas for test items, the design of the survey, and assessment came from researchers and teachers who together scrutinized the items in relation to theories of civic online reasoning (McGrew et al., 2017). To safeguard validity and reliability we designed items in line with previous research A manipulated photograph on daisies in Fukushima was also used in McGrew et al., 2018. In Sweden, 2017, 20 per cent of the pupils in upper-secondary schools had a family with a foreign background (SiRiS).
The survey contained questions regarding the participants’ background, education, political party sympathies, media habits, attitudes to information and some questions from previous questionnaires conducted by Skolverket (2016) and Internetstiftelsen (Davidsson & Thoresson, 2017), in combination with multiple-choice questions. The questions relating to attitudes to information asked pupils to: (1) self-rate their ability to find information online; (2) self-rate their fact-checking ability; (3) rate how reliable information on the Internet is; and (4) rate how much source critical evaluation [källkritik] they have had in school. They were also asked to rate the importance of having access to credible news. All self-reported variables were on a 5-point scale, with the exception of source critical evaluation in education, which was on a 10-point scale (see Table 1 for a more detailed account).
Overview of questions and measured constructs in the online survey
| Background variables | Gender (girl/boy/other identity) Since participants were allowed to choose more than one party, we calculated a normalized mean score that spans from 0–1, where 0 represents the most right-winged party (SD) and 1 represents the most left-winged party (V). All parties in between those are given a score based on their position (0–7) on the spectrum (SD-V) divided by 7. |
| Preferred news | Preferred news format(s) (e.g. paper news, radio etc.) |
| Self-rated abilities | Source criticism on the Internet (fact-checking ability) |
| Information credibility | Importance of news credibility (credibility importance) |
| Sourcing in education | Source critical evaluation in education (sourcing in education) |
| Detecting sponsored material (sourcing) | Screenshot from evening paper 1 ( |
| Comparing articles (corroboration) | Two articles about weight loss (weight loss) |
| Scrutinizing comments and images (evidence) | Manipulated photograph of a smoking girl (smoking) |
The assessed variables are native advertisements, unknown commenters, and scientific evidence. Participants were asked to identify the source of information on screenshots from three online newspapers (
We asked participants to compare the credibility of an article based upon current research about weight loss with an article about a surgeon interested in selling surgery for weight loss, to test their abilities to corroborate information (weight loss). We also asked them to corroborate the credibility of a balanced text from public service with a right-wing text on the government’s new policies regarding hate crime (racism).
To test their abilities to evaluate evidence we asked them to rate the credibility of three articles: a credible news article about the government’s energy goals (credible); a reader’s subjective comment on income differences (biased); and an article from a climate change denier (false). We also asked them to evaluate whether manipulated images could be used as evidence (smoking and Fukushima).
The variables reflect three basic skills for assessing credibility of information on the Internet: 1) sourcing – identify where the news come from; 2) corroborate information – what other sources say about the news; 3) evidence – evaluate the presented evidence (see Table 1). To assess the skills three different tasks with multiple test items were used: 1) detecting sponsored material in newspapers; 2) comparing articles with credible and biased information; and 3) scrutinizing credible and biased comments and authentic and manipulated images.
Pupils self-reported that they were quite competent at finding and evaluating online information: 68 per cent rated their fact-checking ability as good (55%) or very good (13%) and 79 per cent rated their search ability as good (57%) or very good (22%). The self-rated abilities were noted on a scale from 1–5: fact-checking ability (
On average, most pupils rated it important to have access to credible news (
To investigate the performance we first computed the total number of correct answers for all question items that could be classified as correct or incorrect, where the questions with embedded advertisements (see Table 1) were coded as correct if all answers were correct. The mean number of correct answers from a maximum of 8 for all participants was
While most pupils rate themselves as quite skilled at finding and scrutinizing online information, most of them (88%) could not separate news from advertisements in Sweden’s most-read newspaper. They performed better at debunking manipulated images: 75 per cent and 81 per cent identifing manipulated images as poor evidence. Still, a majority struggled to separate balanced from biased information.
In order to relate performance to background, variables and self-rated abilities, we conducted a number of regressions. For detecting sponsored material (e.g. Not all predictor variables were used in all models: For some of the models we only used a subset, in order to get a better model fit. For a detailed specification of what variables we used in each regression we refer to Appendix A.
For racism, weight loss, income, smoking, and Fukushima we conducted logistic regressions, with correct/incorrect as dependent variable and the same predictor variables as in the Poisson regressions. The results describe the probability for a correct answer, hence a positive coefficient indicates that the predictor variable increases the probability for a correct answer, and inversely, a negative coefficient indicates that the predictor variable decreases the probability for a correct answer. For detailed information on the regression coefficients and model fit we refer to Appendix A, Tables A6–A9. Finally, for analysing media habits, we conducted Poisson regressions, with the incidence for each participant of a specific media source or media format as dependent variable, and form, sourcing in school, search ability, credibility importance, and party score as predictor variables. The results describe the probability of indicating a specific media source/format, where a positive coefficient indicates that the predictor variable increases the probability for the incidence of a certain source/format, whereas a negative coefficient indicates that the predictor variable decreases the probability. For detailed information on the regression coefficients and model fit we refer to Appendix A, Tables A2–A5.
There were no significant associations between self-rated abilities and background variables for the number of correct items for detecting sponsored material or news articles for either
Less than half of the pupils (44%) identified a text from Swedish public radio as more credible when asked to corroborate this text with a right-wing populist text from
For identifying racism, there was a significant increased probability for a correct answer when being in the aesthetics programme compared with the social science programme, being a girl as compared with indicating other identity, and a marginally significant increased probability for a correct answer for a unit increase in sourcing in school. Even if this is a politically charged topic we did not find a link to the pupils’ self-rated political orientation.
Many pupils believed in the authority of a surgeon selling surgeries when asked to compare an article in which a surgeon states that the only option for weight loss is surgery with an article reporting the latest research on weight loss, diets and exercise. Forty-seven per cent of the pupils found the interview with the surgeon to be more credible than the article reporting research findings. Due to large residual variance, we could, unfortunately, not specify the regression model for weight loss.
The overall high self-reported abilities of finding and evaluating online information can be contrasted with the mean credibility ratings for credible, biased, and false information on a scale from 1–10. The false information on climate change was given a mean credibility rating of 6.02 (
A correct identification of a reader’s comment on income distribution (income) as poor evidence was associated with a higher rating on credibility importance (illustrated in Figure 1), whereas a higher rating on internet info reliability was associated with a decreased probability of a correct answer.
Figure 1
Predicted probabilities of a correct answer on the comment on incomes as a function of internet info reliability and credibility importance
As mentioned above, many pupils were able to debunk manipulated images. In the case of identifying the manipulated image of a smoking girl as poor evidence, a higher rating of fact-checking ability was associated with a lower probability of a correct answer and a higher rating on credibility importance was associated with a marginally significant increased probability of a correct answer. For Fukushima, being in the aesthetics programme compared with the social science programme was associated with a higher probability of answering correctly.
To sum up, a higher rating on credibility importance was associated with an increased probability of correct answers. The inverse is true for self-reported fact-checking ability and internet info reliability – here a higher rating was associated with a decreased probability of a correct answer. Studying in the aesthetics programme, speaking Swedish at home were also associated with correct answers.
Key components to high performance are, hence, high ratings on credibility importance and low ratings on internet info reliability and fact-checking ability, and to shed some light on the concept we investigated the youths’ media habits. As previously described, we regressed the incidence of each media source (e.g. local newspaper, public service radio, etc.) and media format (e.g. computer, radio, etc.) For a complete list of media format and media sources we refer to Appendix B.
In sum, credibility importance and fact-checking ability are strong predictors for preferred media format and source, the same variables that are associated with performance on the test items. The data did not permit a direct analysis of the association between media format/source and performance on the tested abilities. Hence, we cannot say anything about how credibility importance, fact-checking ability and media format/source are linked to performance; further investigation is needed to uncover how these variables are related to each other.
In this paper we present three findings from our survey of pupils’ abilities and problems in assessing credibility in online news: first, the performance on all items was relatively poor – most pupils got less than half of the items correct; second, the credibility ratings on three articles, one credible, one false news and one reader’s comment were relatively high and did not differ much; and, last, but not least, regressions showed that self-rated importance of credibility of news and being in the aesthetics programme were associated with better performance, whereas higher self-reported ratings on fact-checking ability and reliability of news on the internet were associated with a worse performance. Our findings are in line with previous research on civic online reasoning (McGrew et al., 2017, 2018) and we add new dimensions of mindsets and background variables to this previous research, highlighting the complexity of evaluating digital news.
Although pupils in Sweden see themselves as good or great at searching and evaluating online information (Davidsson & Thoresson, 2017; Skolverket, 2016), most pupils in our performance test (88%) struggled to separate news from ads in a common digital newspaper. Importantly, claiming to be good at finding information online is associated with the opposite. Our findings indicate that self-reported surveys regarding media and information skills do not provide us with a better understanding of people’s actual abilities. Self-reported skills of fact-checking and search ability are inversely associated with performance.
Further, our findings show that pupils, like other people, have a hard time determining the trustworthiness of credible, biased and false information online. Pupils may be good at finding and sharing credible news (Nygren et al., 2019) but they are not very skilled at determining credibility in critical and constructive ways. Teaching students to find and use credible sources may be easier than teaching them to scrutinize digital information. The fact that pupils are not very good at determining credibility highlights the importance of access to credible news.
Our findings indicate the existence of subgroups, which in turn implies that there is a digital divide between the young who are skilled at determining credibility, and those who are not. In the group with great problems determining credibility we find pupils who see themselves as skilled at searching and fact-checking online information, and searching for information online, and do not rate access to credible news as important. A speculative account of the low-performing pupils’ attitudes is that this reflects a mindset of overconfidence and ignorance, enhancing confirmation bias. The fact that they do value access to credible news may indicate that the pupils are “news-avoiders” who do not read or seek credible news, and therefore cannot differentiate between true, biased, and fake information. This reasoning is supported by research showing that lack of knowledge in a domain results in overconfidence in one’s own ability and an incapacity to judge others’ performance. A vast amount of research in the field of overconfidence shows an overestimation of performance among incompetent performers and underconfidence in high performers, called the Kruger-Dunning effect (e.g. Kruger & Dunning, 1999; Dunning et al., 2003; Dunning, et al., 2004; Kruger & Dunning, 2002; Ehrlinger et al., 2008). The effect has been replicated in different areas, from logical reasoning and grammar competence to ratings of humour and knowledge of new financial markets. The reason for overconfident ratings among low performers has been attributed to their lack of metacognitive skills; that is, since they have no competence in the area they are unable to gauge their own as well as others performance on the task.
In contrast, pupils doing well on our test seem to have a mindset of scientific curiosity and an openness to consider biases, including their own (Kahan et al., 2017). The fact that they find it important to consume credible news from trustworthy sources may indicate that pupils in this group are aware that there is a lot to learn from news and people with better knowledge on different topics. Further, our findings indicate that there is, possibly, a gap between news-seekers and news-avoiders (Strömbäck et al., 2013). If this is the case, there might be a link between news habits and skills at determining credibility, which may be associated with socio-cultural patterns, separating young people in society (Lindell & Hovden, 2018). At present, we do not know in any detail what characterizes individuals who rate credibility importance highly. Nor do we know what characteristics are embedded in the problematic high ratings of fact-checking ability and reliability of information on the internet. In any case, education can play a vital role in bridging the gaps. The fact that pupils from the aesthetics programme are better than other pupils at debunking the racist narrative of
In sum, our findings indicate that teaching and learning source critical evaluation is important to support a critical and constructive treatment of digital news. Civic online reasoning may be linked to appreciating the importance of reliable information and understanding the difficulties associated with finding and evaluating online information. We understand this as a call to make a more detailed investigation of how education may support this reflective, humble and curious approach to news, as well as how to best support students to become active citizens in a digital world.