How Has Covid-19 Affected Published Academic Research? A Content Analysis of Journal Articles Mentioning the Virus

Covid-19 has impacted many areas of life in addition to the health challenges faced. Lockdowns and other social restrictions have influenced the economy and society by blocking or changing many daily activities, including business, lifestyle, and culture. In this context, it is reasonable to expect the pandemic to be influencing many non-health areas of research, particularly in the social sciences, as researchers attempt to develop solutions to the societal challenges faced. Identifying such changes can help researchers and managers to understand any important developments in the research landscape.

Whilst there are many literature reviews of scientific Covid-19 publications that synthesise key findings of relevance to specific disciplines (e.g. Checchi, 2021; Kutela, Novat, & Langa, 2021), there are also bibliometric studies about overall publishing patterns that give a broader picture. Previous investigations into Covid-19 research have often either focused on medical issues or have reported the main topics of Covid-19 research satisfying some criteria (e.g. Chen, 2020), as the following list demonstrates. Arabic research in Scopus before 8 March 2021 mentioning Covid-19 has been clustered by topic, finding four health-related clusters (Sa’ed, 2021; see also: Verspoor et al., 2020). Lists of terms associated with Covid-19 research have also been discovered (Doanvo et al., 2020). As part of the bibliometric contribution, the CORD-19 database of Covid-19-related publications has supported multiple studies, including a map of their key terms (Colavizza et al., 2021). Individual areas have been investigated for the broad nature of publishing. For example, most Covid-19 diabetes scholarship is not published in high Impact Factor journals (Corrales-Reyes, Hernández-García, & Mejia, 2021) and a key area of aquatic research is about wastewater safety (Ji et al., 2021).

Other previous bibliometric studies about Covid-19 have investigated wider issues relating to the speed of research. Based on an analysis of tweets of academic publications, scholarly interest in Covid-19 may have shifted during 2020 from a virological focus to a wider interest in societal and economic factors (Fang & Costas, 2020). Covid-19 research has been published more quickly than comparable studies, and with less international collaboration (Aviv-Reuven & Rosenfeld, 2020), with perhaps lower quality control and greater dissemination (Gai et al., 2021). Coronavirus research published before Covid-19 was drawn upon heavily in Covid-19 research published by May 2020, with the aid of rapid reviews that contextualised prior research for Covid-19 needs (Haghani & Bliemer, 2020; Thelwall, 2020). Discovery-based literature analysis has also been used to quickly identify promising drugs that might treat Covid-19 (Wang et al., 2020).

Researcher gender during the pandemic has been investigated from the perspective of publishing, suggesting a non-topic influence of the pandemic. In some contexts, female researchers have produced fewer preprints or journal articles compared to male researchers, perhaps due to taking on additional unpaid caring work or health-related jobs (e.g. nursing lecturers vaccinating) (Krukowski, Jagsi, & Cardel, 2021; Squazzoni et al., 2020; Vincent-Lamarre, Sugimoto, & Lariviere, 2020). Female researchers have increased their share of bioRxiv preprints since the start of the pandemic, however (Abramo, D’Angelo, & Mele, 2021), perhaps through being at the forefront of advances in developing technologies for treatment, epidemiology, or vaccines.

From the above summary, little is known about the overall influence of Covid-19 on academic publishing, other than for topics, journals, publishing speed, international collaboration, and gender effects. The goal of this article is to understand the influence of Covid-19 on the broad research landscape so far (March 2021), going beyond the topics investigated or the researchers investigating them. As in all the above-reviewed studies, the focus is on academic contributions in the form of journal articles rather than other forms, such as short letters or editorials. The research question is: What have been the main types of influence of Covid-19 on published academic journal articles by March 2021?

The research design was to obtain a large random sample of journal articles mentioning Covid-19 and then apply inductive content analysis to classify the influence of Covid-19 on the publications. Dimensions were chosen as the source of articles because it has wider coverage of Covid-19 literature than Scopus, Web of Science and PubMed (Kousha & Thelwall, 2020). It includes almost all of Scopus (Thelwall, 2018) and so should not have a large bias against any type of article (e.g. medical). Google Scholar probably has greater coverage than Dimensions (Martín-Martín et al., 2020), but lacks features for automated access. Queries of the form below were submitted to Dimensions on 19 March 2021 through its Applications Programming Interface (API) in the free software Webometric Analyst.

search publications for X where year >= 2019 return publications [basics + extras]

The text match component X above was replaced with each of the following separately: “COVID-19”, “Novel coronavirus”, “2019-nCoV”, “SARS-CoV-2”, “coronavirus 2”, “Coronavirus disease 2019”, “Corona virus disease 2019”. The text match components were designed to capture common ways of referring to Covid-19. The term “pandemic” could also have been used but this is not specific to Covid-19 so it may generate false matches.

The r esults from the seven queries were merged and duplicates removed, resulting in 163,290 records. All 126,193 documents of type “Article” were then extracted and sorted in random order with a random number generator and the first 500 were manually coded, as described below. During the manual coding, documents were removed if they did not mention Covid-19 in any form (e.g. the mention was in an advert, not the article text), or if they were not academic journal articles or similar documents (e.g. full conference papers, extensive letters to the editor, medical case reports) (n=93 altogether). An additional 29 paywalled articles were excluded because their full text could not be accessed. Articles not in English were translated with the free Google Translation service. When an article was removed, it was replaced by the next document in the randomly sorted list.

Inductive content analysis (Neuendorf, 2017) was used to code the influence of Covid-19 on each paper in the random set of 500. Inductive content analysis is a human classification method, where the categories are not decided on in advance but are designed after contact with the data. An initial set of codes was developed by the first author after reading the first 100 articles and then extended and revised as necessary when coding the rest. Each paper was coded by the first author by reading how Covid-19 was mentioned (with any name) in the article and then characterising this influence with a suitable category. This produced 20 categories that seemed to characterise the main types of Covid-19 influence. During this process, a codebook was constructed describing each category. The 500 articles were then coded again by the first author, hiding the original codes, with the results compared. The codebook was refined to clarify ambiguities found and two categories that were difficult to separate (Application to Covid-19; Increased importance due to Covid-19) were merged. Finally, the codebook and list of 500 documents were passed to the second author to code independently, to check the consistency of the coding and the informativeness of the codebook.

The level of agreement between the coders was checked using Cohen's kappa (Cohen, 1960), which calculates the level of agreement above chance. This yielded a score of 0.554, which is a moderate level of agreement and enough to validate the results as reliable enough to report (Landis & Koch, 1977). Cases of disagreement were re-examined and a consensus was reached. A common cause of disagreement was that an article could reasonably fit multiple categories (e.g. a paper about the diagnosis and treatment of Covid-19, or an article about the epidemiology of Covid-19 from a safety perspective). Review and introductory papers typically covered multiple topics, for example, with none being dominant.

The codebook and full results are available in an online supplement (https://doi.org/10.6084/m9.figshare.14599908). Many of the journal articles and categories developed were biomedical or otherwise health-related (Table 1), for research about dealing with the health threat of Covid-19. Extensive category descriptions were needed because there were often overlaps between categories. For example, some articles discussed diagnosis and treatment, or safety precautions and epidemiology. Similarly, articles about the virus often mentioned potential treatments that might be derived from them. These categories seem to be straightforward influences of Covid-19 in the sense of covering the health-related types of research needed for dealing with it.

The second set of categories covered topics that were not necessarily directly related to Covid-19 (Table 2), although there was some overlap for the first four categories. For instance, the research might be classified as having increased importance due to Covid-19 in health or non-health contexts. None of the papers in this group were primarily about the health impacts of the Covid-19 virus, however. These categories are much broader in terms of subject because there were too few papers to form useful narrower subject categories.

The first three categories and last one do not map to academic fields and so are useful to explain in more detail. Many authors explicitly mentioned or implied that their research had increased or decreased importance due to Covid-19. Such claims were sometimes made in introduction or discussion sections. For example, “In the face of COVID-19, this study is especially important in bringing awareness to the possible use of salivary biomarkers in determining potential risk for T2D and metabolic syndrome” in an article entitled, “Escalating risk of metabolic syndrome imparted by salivary biomarker in obese children” was interpreted as a claim that the paper had become more important due to Covid-19. In this case, although the article is about health, it was not about attempts to address Covid-19 in a health context. In contrast, the paper “Chinese outbound tourism: An alternative modernity perspective”, admitted that pandemic travel restrictions would have consequences for the study in a way that could not be analysed in the article. This was coded as an instance of decreased importance. As an example of a paper coded as Covid-19 context, the article, “Defining the older patient population that requires, but do not undergo emergency laparotomy: an observational cohort study protocol”, recorded information about the impact of Covid-19 on decision making as a relatively minor detail. In another example, a paper about H7N9 mentioned that Covid-19 showed the need for pandemic preparedness. For the final category, an academic mentioned that his partner helped him revise his paper during the pandemic, which was coded as Irrelevant. In another example, the paper “Evidence of polyphenols efficacy against dry eye disease” cited a reference for medical safety, where Covid-19 was in the title of the cited article, despite being cited for something else.

The content analysis results show that different medical topics account for 61.6% of the journal articles mentioning Covid-19 (Figure 1). Despite this dominance, 18.2% covered other subject areas (society, health, education, economy, art) and in 15.2% of cases the article was not about Covid-19 or the pandemic, but it formed or altered the context of the article. In 5% of cases Covid-19 was mentioned in an article despite being irrelevant to it. Thus, according to Dimensions, almost two thirds of articles about Covid-19 that had been published by 19 March 2021 were medical in nature (including safety precautions and wider health service provision but excluding wider health effects of the pandemic).

Figure 1

The largest group of articles about addressing the health effects of Covid-19 covered potential medical treatments for it, although not always with positive results. Presumably this large body of research has collectively led to better outcomes for patients that can access high-quality medical treatment. This research also includes many papers about the challenges of detecting and tracking Covid-19 and of organising safe and effective health services for patients. It is perhaps surprising that there is relatively little research about vaccines. This reflects several factors. Presumably, the lack of research into delivering vaccines in the early days of the pandemic contributes to this number. Vaccine-related research may also take longer to publish because of the time scale of large vaccine trials and may be primarily applying existing knowledge and technology rather than developing a line of publishable findings that could lead to a vaccine.

Many of the papers that were not about dealing with Covid-19 from a health perspective addressed the health, education, or economic impacts of social distancing, such as by charting its influence or by reporting solutions designed for the new situation. For example, “Effects of COVID-19 epidemic lockdown on postpartum depressive symptoms in a sample of Italian mothers” was coded as Pandemic health, “COVID-19: Are you satisfied with traveling during the pandemic?” was coded as Pandemic society, “towards ushering in a resilient [microfinance] eco-system in the battle against corona pandemic” was coded as Pandemic economy, and “Virtual role-playing simulation as clinic: A model for experiential learning of critical thinking skills” was coded as Pandemic education because the article reported the initiative as a pandemic distance learning adaptation.

The results are limited by the subjective nature of the content analysis coding, and especially the construction of the initial set of categories. They are also limited by the scope of the database used, and the data collection date. In particular, the share of medical research may be exaggerated if it had been published more quickly than other research (Horbach, 2020). The sample size of 500 is also a limitation because the percentages reported are approximate. Perhaps most importantly, the study does not capture influences that do not detail the influence of Covid-19 in a published paper. For example, someone's research being delayed (e.g. laboratory samples that could not be maintained) or stopped by Covid-19 would not result in a journal article and is ignored. Another issue is that nations have been unequally impacted by the pandemic. There may therefore be less need to investigate the pandemic in countries that have controlled it relatively well by March 2021, such as China, New Zealand, and Vietnam.

Some of the codes reflect the themes found in a study of tweets about Covid-19 academic research (Fang & Costas, 2020), but do not map easily onto the topic-based information or keyword-based maps of previous studies (e.g. Chen, 2020; Colavizza et al., 2021). Most prior research has focused on medical studies, but one analysis has noted the existence of research from other areas, accounting for a third of publications (Aristovnik, Ravšelj, & Umek, 2020). These studies have not used content analyses to investigate the role of Covid-19, however, but have instead analysed the topics or words used in papers matching a Covid-19 search. Thus, whilst the results confirm from a different perspective to prior research (Aristovnik, Ravšelj, & Umek, 2020) that medical Covid-19 research (as defined above) is almost twice as prevalent as other types, these other types are not rare and are characterised above (health, education economy, society, art). Moreover, the main novel finding is that Covid-19 is influencing a minority of research that is not directly about Covid-19 or the pandemic. Whilst some authors claimed that their research became more important because of Covid-19, a few admitted that Covid-19 undermined their study methods or findings. For the first time, this points to the deeper influence of Covid-19 across academia going beyond research topics, although impact on research productivity has previously been found in terms of gender differences (Abramo, D’Angelo, & Mele, 2021). Nevertheless, the relatively small numbers point to this wider influence not being dominant, especially given that medical research forms a small proportion of all academic research.

Author claims that their research had become more important due to Covid-19 were not evaluated for accuracy in any case. Such a claim could be a simple statement of fact (e.g. a paper evaluating the effectiveness of masks for influenza transmission clearly has value for Covid-19 safety measures). Nevertheless, it could also represent wishful thinking from the author, either reflecting their own belief of the potential importance of their own work or as part of a persuasion strategy targeting editors, reviewers, or potential readers. Journal articles are not simple descriptions of a research project but are artefacts of persuasion, following disciplinary norms (Hyland, 2004). One standard task when writing a journal article is to establish the importance of a problem (Swales & Feak, 2012), which might be enhanced by a connection with Covid-19. Thus, an author might seek to make such a connection with Covid-19 when writing their paper, and such connections might not be plausible or important. In this context, the pandemic may give an advantage to authors and research topics that can make plausible claims of Covid-19 relevance for their findings, even when not directly about the virus.

The results confirm that whilst approximately three fifths of research mentioning Covid-19 is just about providing medical treatments or health care for those infected by Covid-19, the remainder covers other pandemic-related topics or is influenced by the pandemic but is not about it. Added to prior findings of pandemic-related gender shifts in research, this suggests that the pandemic is altering the academic research landscape more broadly than just through the emergence of a large body of scholarship aimed at controlling the pandemic.

The rapid adaption of sections of academia to new pandemic-related challenges is a welcome sign that researchers are trying to adapt their skills or focus on supporting society. Nevertheless, the relatively small amount published so far suggests that wider academia may not be reacting quickly enough to the new challenges, assuming that fast reactions do not compromise quality (Dick et al., 2020). In this context, it is important that research managers and funders recognise that scholars may have pandemic-related contributions to make even if they are not in the health field. More concretely, funders and managers may need to ensure that researchers are able to react rapidly to the arguably unprecedented societal changes and are supported through funding when necessary. They should also support researchers that need the flexibility to cope with the pandemic-related pressures on their existing research.