An Automatic Method to Identify Citations to Journals in News Stories: A Case Study of UK Newspapers Citing Web of Science Journals

Journal names were extracted from the Journal Citation Reports (JCR) Science (9,639) and Social Science (3,412) editions. Journal names were downloaded separately for each year 2006–2015 and duplicate titles were removed. In a few cases, extra abbreviations or subtitles from the original JCR Full Journal Titles were deleted to increase the power of the subsequent ProQuest citation searching, as shown in square brackets in the following examples. For example, “JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION” gave eight UK newspaper ProQuest matches, whereas over 15,000 were found when “JAMA-” was removed. The abbreviated journal name “JAMA” was not used alone because it could retrieve many false matches by capturing names such as “Mustaf Jama”, “Yusuf Jama”, “Mohamed Jama” or “Ibrahim Jama”. However, we used journal titles containing the term JAMA with other medical specialities such as JAMA Psychiatry, JAMA Surgery, JAMA Oncology, JAMA Internal Medicine, and JAMA Dermatology. Moreover, the main English language journal names in JCR were used to generate the ProQuest citation queries for higher recall (e.g., “Journal of Orofacial Orthopedics” instead of “Journal of Orofacial Orthopedics-Fortschritte der Kieferorthopädie”).

UK newspapers sometimes use British English spellings of journal names in American English. For instance, “AMERICAN JOURNAL OF OBSTETRICS AND GYNECOLOGY” retrieved 73 UK newspaper ProQuest matches but the British English spelling “GYNAECOLOGY” gave 120 more. Scientific terms that could be in journal names were therefore identified using a list of British English words that have different American English spellings (https://en.wikipedia.org/wiki/Wikipedia:List_of_spelling_variants) such as Behavior/Behaviour, Pediatric/Paediatric/, Signaling/Signalling, and Labour/Labor, as the examples below show. For these cases two searches were conducted to cover both British and American spellings and the results were combined.

For journals with “&” in their JCR titles, additional searches were conducted to retrieve newspaper mentions with “and” instead. For instance, the JCR-indexed journal “LANCET DIABETES & ENDOCRINOLOGY” had 141 UK newspaper matches, whilst the same search with “AND” (“LANCET DIABETES AND ENDOCRINOLOGY”) retrieved 116 extra results.

For JCR-indexed journals with generic names, such as Science, Nature, Cancer or Cell, the term “journal” was added before and after their names as a practical method to reduce false matches during ProQuest full text searching. From initial investigations, news stories typically add the term “journal” before, and occasionally after, journal names when reporting research from journals with names that are not clearly academic, such as “published in the journal Science”, “reported in the journal Nature”, or “a finding in the medical journal Cancer”. For instance, there were 66 ProQuest hits for “Nature journal” in the eight UK newspapers during 2006–2015 and 4,259 for “journal Nature”. Additional examples are shown below.

The term “journal” was not added to publications containing the similar terms Journal, Quarterly, Annual, Proceedings, Bulletin, Letters or Archives, or the distinctive terms Acta, ACM, ACS, IEEE, BMC, JAMA, BMJ or Revista or the small set of journals PLOS Biology, LANCET Infectious Diseases and Harvard Business Review because their names were unique enough for effective searches. For instance, the ProQuest query “journal PLOS Biology” gave 558 newspaper matches, missing 4,600 relevant results from the query “PLOS Biology”.

To locate mentions of the 13,051 scientific journals in (full text) news stories, the above journal queries were submitted separately in the full text field (“FT”) of ProQuest “Command Line Search” interface as phrase searches in October 2016. Extra search commands were added to the query to limit the results to each of the eight selected UK newspapers and to each year during 2006–2015, as in the example below.

Because the maximum number of Boolean operators in a single ProQuest search was 1,000, fourteen separate searches were conducted for each newspaper/year combination to cover all journals in the data set. Irrelevant publications, such as the Sunday editions Independent on Sunday and Sunday Herald, were filtered out with the ProQuest “Narrow results” menu. The full text ProQuest records for the search results were downloaded to generate a set of news stories potentially mentioning one or more of the 13,051 academic journals.

The Stage 2 method returns some false and ambiguous results. For instance, the query FT(“JOURNAL OF CANCER”) matched multiple journals (e.g., EUROPEAN JOURNAL OF CANCER, BRITISH JOURNAL OF CANCER). Adding the term “journal” after journal names (“Science journal” or “Nature journal”) also sometimes retrieved many false matches for journals with generic names (e.g., “published in the science journal Advanced Materials Interfaces”). To automatically identify journal names in the full text of each news story, a program was written and added to the free Webometric Analyst software (http://lexiurl.wlv.ac.uk Services option, “News: Extract journal mentions from ProQuest records”) to extract correct mentions of journals. This program scans news stories for mentions of one or more of the terms, “journal”, “bulletin”, “acta”, “annals”, “transactions”, “proceedings”, “zeitschrift”, or “lancet” and then uses heuristics to identify if the text before, after or surrounding is a journal name. The heuristics rely upon journal names being in title case (initial capital letters except for small words) and incorporate a list of rules to eliminate common text segments that these rules often mistakenly add (e.g., “Writing in the”, “Reporting in the”, “Dr”, “Prof”, “Professor”, “University”).

The list of journal names extracted was matched against the 13,051 pre-selected journal names using Webometric Analyst (see the Tab-sep option, “Split file”> “Split file 1 based upon…”) to exclude non-JCR publications (e.g., Indian Journal of Scientific and Industrial Research) and other noun phrases (e.g., American Heart Association) that were incorrectly extracted by the software. This gave a list of newspaper stories with at least one citation to a Science Citation Index (SCI) or Social Science Citation Index (SSCI) journal.

There are duplicate news stories from different editions of the same newspaper in ProQuest. To avoid counting duplicate citations, stories with duplicate titles in different editions (e.g., Scotland, Eire or Ulster regions) were deleted.

Some news stories had the same contents but slightly different headlines, as shown in the Daily Mirror example below. Hence, news stories with the same abstract as a previous story were removed (in the example below: “Bosses at the Prostate Cancer Charity point out that men who ate the most eggs in the study were also more likely to smoke, be overweight, take less exercise and have a poor diet - all risk factors for the disease which kills 10,000 in the UK annually.”). Such title changes are presumably minor copyedits by regional editors.

A content analysis was conducted on a random sample of 360 newspaper stories (45 from each of the eight UK newspapers) mentioning a WoS science journal to assess whether newspapers mention scientific journals in their articles to communicate research results and, if so, how newspapers discuss research. The content analysis also estimated the accuracy of the method for capturing journal names in the text of newspapers. Three independent coders with social science PhDs conducted the content analysis after a pilot study on a random sample of 40 newspaper stories (not in the main sample of 360 news stories) to refine the classification procedure. Cohen’s kappa values were calculated between the three coders for all classes and compared with standard heuristic guidelines (at least 0.21+ indicates fair agreement, 0.41+ indicates moderate agreement, 0.61+ indicates substantial agreement and 0.81+ indicates almost perfect agreement) (Landis & Koch, 1977). The following classes were used.

Only 5.2% of Science and 5.7% of Social Science WoS journals were cited at least once by a daily UK newspaper 2006–2015. More WoS Science journals were cited by the Daily Mail (Science: 8.8%; Social Science: 9.1%), Daily Telegraph (7.6%; 8.9%) and Times (7.3%; 8.1%) than the Sun (2.1%; 1.4%), Daily Mirror (3.4%; 1.7%), and Herald (3.9%; 2.9%) (Figure 1), showing large differences between newspapers. No major UK newspaper restricts its coverage to a small number of elite journals, however. Perhaps surprisingly, the tabloid Daily Mail has the widest coverage of academic journals.

Figure 1

Although more Science journals were cited than Social Science journals (Figure 2), in comparison to the number of journals in WoS, a slightly higher proportion of Social Science journals were cited by the UK newspapers (the fourth column). It is therefore possible that the lower total Social Science press coverage is due to the smaller amount of published Social Science articles in WoS or overall.

Figure 2

In order to estimate the main academic fields of the cited journals, each was assigned to one of 22 Essential Science Indicators (ESI) research fields

http://ipscience-help.thomsonreuters.com/incitesLiveESI/8289-TRS.html

Figure 3

From the Social Sciences category (Figure 4 and Table B in the online appendix), journals with at least one newspaper citation were mainly from Psychiatry/Psychology (43%), Social Sciences (37%), and Economics & Business (10%) based on their ESI fields. Psychiatry/Psychology is particularly interesting to the tabloids: Sun (54%), Daily Mail (50%), and Daily Mirror (49%). The slightly left-leaning broadsheets Guardian and The Independent cited more Social Science, General journals. These included Foreign Affairs, British Journal of Sociology, Antiquity (a journal about world Archaeology) and British Journal of Political Science. A few (10%) of the cited journals have a science classification (e.g., Clinical Medicine, Environment/Ecology, and Engineering) due to the ESI classification scheme, such as Addiction, Alcohol and Alcoholism (classified as Clinical Medicine in ESI) or Energy policy and Applied Ergonomics (classified as Engineering in ESI).

Figure 4

Within Science, British Medical Journal (now officially called The BMJ) had the most citations from all newspapers, ranging from 12% in the Daily Mail and Daily Telegraph to 22% in the Sun (Figure 5). The multi-disciplinary journals Nature, Science, and PLOS ONE were also frequently cited. Table C in the online appendix (https://doi.org/10.6084/m9.figshare.4796548.v4) shows the additional general medical journals New England Journal of Medicine (NEJM), Lancet and Journal of the American Medical Association (JAMA), and specialist scientific journals, like British Journal of Cancer, British Journal of Sports Medicine, Current Biology, and Neurology. The overall results confirm the importance, but not dominance, of a few prestigious journals for UK newspapers. However, there are large differences. For instance, half of the citations to science journals from the Independent (52%) and Guardian (48%) were to seven world-leading science and medical journals (BMJ, Nature, Science, NEJM, Lancet, JAMA and PLOS ONE), in comparison to less than a third from the Daily Mail (29%), Sun (32%), and Herald (33%).

Figure 5

In Social Sciences, the WoS journals most frequently cited by UK newspapers were about psychology, psychiatry, epidemiology, public health, medical ethics and climate change, such as British Journal of Psychiatry, Psychological Science, Journal of Epidemiology & Community Health (a BMJ publication), Journal of Medical Ethics (a BMJ publication), and Nature Climate Change. Figure 6 and Table D in the online appendix (https://doi.org/10.6084/m9.figshare.4796548.v4) show that there are also some social science journals from other specialisms with many citations from UK newspapers, such as Journal of Consumer Research, The Economic Journal, and British Journal of Sociology. This suggests that most UK newspapers were most interested in reporting public health (e.g., psychology and psychiatry) but also had some interest in the environment, economics and business.

Figure 6

There are some differences in reporting social science research between newspapers. For instance, 14% of the citations from the Times were to Journal of Archaeological Science (9%) and Antiquity (3.6%) and more tobacco research in the journal Tobacco Control was cited by the Sun (5.5%) and Daily Mirror (3.6%) than by other newspapers.

There has been no universal and systematic overall increase or decrease in terms of the numbers of journals reported in the UK press, although there are patterns for individual newspapers (Figures 7 and 8). The Times seems to have decreased its coverage in both broad areas 2009–2013 and the Daily Telegraph seems to have increased 2010–2014.

Figure 7

Figure 8

The coverage of newspapers by ProQuest was checked to assess whether changes over time could explain the citation variations. There were five times more news stories from the Times indexed by ProQuest during 2006 (about 117,000), 2007 (108,000) and 2008 (105,000) compared with 2009–2015 (19,000 to 20,000), indicating that the fewer indexed news stories during 2009–2012 was the cause of fewer citations to Science and Social Science from the Times during this period but it does not explain the increase 2013–2015 (Figures 7 and 8). This seems to reflect deliberate policy shifts or staff changes within ProQuest rather than an 80% reduction of news content in The Times in 2009.

To get a simple overview of the topics of news stories citing at least one journal, all news headline terms were extracted and the Porter algorithm (Porter, 1980) was used to convert plural English terms to singular and to remove verb suffixes (e.g., ing, ed and tion). Term frequencies were calculated for each newspaper and for Science and Social Science journals separately. Small function words, such as “the”, “of”, “to”, and “in”, were then manually removed and common irregular nouns were combined (e.g., woman and women or child and children).

For the news headlines of stories citing WoS Science journals, medical terms are the most frequent, but the Guardian and Independent also had terms relating to climate change, and global warming, as well as dinosaur and fossil (Tables E and F in the online appendix, https://doi.org/10.6084/m9.figshare.4796548.v4) Medical terms described common major diseases, including cancer (e.g., breast and prostate cancers), heart (e.g., heart attack), brain (e.g., stroke), Alzheimer’s, diabetes, and dementia. These are probably mentioned in the contexts of warnings rather than cures. For instance, Daily Mail terms risk (frequency: 552), warn (70), death (66), kill (56), harm (53) and danger (53) are more common overall than cure (87), hope (63) and help (221). Medical issues concerning children and women are more apparent than those for men. For instance, child (205 including children), baby (151), woman (176 including women) and mother (65) were more frequent in the headlines of Daily Telegraph news stories than man (111 including men), although these may include some non-medical stories. For Social Science journals, women, children, men and mental health issues such as depression and suicide were common.

Following the Landis and Koch (1977) guidelines, inter-coder agreement between the three coders for the first category (Research citations, non-research citations and errors) was either substantial (0.61–0.80) or perfect (0.81–1) and for the second (The extent of research coverage) and fourth (Sentiment of research in news) categories the agreement was mainly moderate (0.41–0.60) or substantial (0.61– 0.80). The agreement rates between coders for the third (Sources reported in the news story) and fifth (Research quality judgment) categories include many cases of fair agreement (0.21–0.40) and no agreement (less than 0.20). This is probably due to the unbalanced nature of the categories (most in one class) and the subjective nature of the judgements for these classes and hence the results (averaging the three coders) should be interpreted cautiously (see Table G in the online appendix, https://doi.org/10.6084/m9.figshare.4796548.v4)

Nearly all (94%) of the sampled news stories had cited WoS Science journals to report their research findings, with fewer for the Guardian (86%) and the Independent (88%) (Table 1). About 6% of the journals were mentioned in non-research contexts, such as “Bernard Lagan Sydney The editor of the Medical Journal of Australia has been dismissed after he objected to outsourcing the 101-year-old journal’s production to Reed Elsevier”. No cases of software errors in capturing journal names were identified in the sample.

The descriptions of 59% of the reported journal articles were extensive. The Herald (83%), Independent (74%) and Daily Telegraph (69%) had more extensive reports than the Sun (33%) and Daily Mirror (36%), for example giving more findings or discussing methods. Most newspaper stories had used more than one source (53%) rather than a single journal article (47%). The Guardian (67%), the Independent (62%) and Daily Mail (62%) used multiple sources more often than did the other newspapers.

Similar shares of the newspaper stories reflected good, bad or other aspects of research. However, good news was more common in the Daily Mail (48%) such as “Cherry aid for the heart; A slice of pie could cut cholesterol and reduce your risk of diabetes”. In contrast, the Sun (47%) reported more bad news, such as “A lack of international will means the chances of bringing climate change under control may already be slipping out of reach.”.

Most stories (94%) made no comment about the quality of the research covered. However, 5.5% of the reported research was explicitly judged to be important or high quality, such as “Scientists have created a revolutionary drug that could restore sight to the blind” and “In a landmark study,…”. Less than 1% of the stories clearly referred to faults or weaknesses, such as “The author of an article in the British Medical Journal which incorrectly claimed that statins cause side effects in 20 per cent of patients has hit out at the leading academic who made a complaint to the journal - accusing him of taking “a biased view” on the usefulness of the drugs.”