The need to develop tailored tools for improving the quality of thematic bibliometric analyses: Evidence from papers published in Sustainability and Scientometrics

In the context of the replication crisis in science (Saltelli & Funtowicz, 2017), measuring the methodological quality of studies becomes particularly important. Thus, Lindner et al. (2018) urged conducting assessments of the quality of the methods used in research and of the reproducibility of the results. They also advocate for replication/confirmation studies. Along the same lines, Moher et al. (2020) proposed as one of their five Hong Kong principles for evaluating researchers to “value the accurated and transparent reporting of all research”, and therefore advised the use of reporting guidelines to ensure complete and adequate reporting of the methods used. They also recommended the public availability of raw data, code, and materials that allow research to be reproduced. However, studies in sectors such as Business and Management found that only 4.7% of journals were expressly open to accepting replications of previous works (Tipu & Ryan, 2021), and that only between 0.1% and 1.5% of papers published in top Economics and Business and Management journals were replication studies (Mueller-Langer et al., 2019; Ryan & Tipu, 2022). Very low levels of replication are also reported in other Social Science fields, around 1% in Psychology (Makel et al., 2012), and slightly above 0.1% in Education (Makel & Plucker, 2014), albeit with an upward trend in recent years (Perry et al., 2022).

The field of bibliometrics has not been characterised by promoting replication studies, despite the fact that, as Glänzel (1996) rightly pointed out, “reproducibility is one of the basic criteria for any science. Under identical conditions research results should be reproducible in bibliometrics, too. The reproducibility of results can only be guaranteed, if all sources, procedures and techniques are properly documented in scientific publications”. Authors such as Harzing (2016), also highlighted that replication studies are essential for the advancement of scientometrics, and even proposed the inclusion of a Replication Studies section in scientific journals. Among the few cases found in the scientometric field, we can cite replication studies on nepotism in peer reviews (Sandström & Hällsten, 2008), or on publications of excellence in Nursing (Nicoll et al., 2020). Due to its similarity to our research, we should point out the replication carried out by Liu (2022) of a thematic bibliometric study, in which she identifies different methodological gaps that significantly affect the results and their interpretation.

In early 2023, the first protocol aimed to provide guidelines for the reporting of bibliometric analyses was launched. The Guidance List for the repOrting of Bibliometric AnaLyses (GLOBAL) “will help promote transparency and completeness in reporting bibliometric and related analyses and provide a framework for authors to report methods and results” (Ng et al., 2023). As GLOBAL has just been released at the time of this study, it is not feasible to measure the impact that this tool may have on the bibliometric community.

Until the GLOBAL protocol was released, there were no specific protocols or guidelines that indicated how a study should be conducted in the field of bibliometrics, as there are for systematic reviews or other types of evidence synthesis. However, some existing guidelines may be applicable, at least in part, to identifying the different items that should be reported and how to report them in a bibliometric study. Thus, the Search (S) extension derived from the PRISMA protocol - Preferred Reporting Items for Systematic reviews and Meta-Analyses- (Rethlefsen et al., 2021) identifies up to 16 variables that should be reported in a completely transparent and replicable systematic review, including aspects such as the breakdown of the databases and platforms used, the “copied and pasted exactly as run” search equation, the limits and restrictions used, the search filters, or the date of the search.

Many of these variables can also be used to properly report the designed search equation in a bibliometric analysis, as well as to accurately document the procedures and choices made in the search process and determination of the final analysis sample. Adherence to the PRISMA-S items is linked to the accuracy in reporting the search strategies and procedures employed (Sadeghi-Ghyassi et al., 2022) and is therefore related to a higher methodological quality of the research, as has been exposed in several works assessing the rigour of systematic reviews, mainly in the health field (Biocic et al., 2019; Koffel, 2015; O’Donohoe et al., 2021). The only study located that analyses the quality of search strategies in systematic reviews in the area of Information Science (Salvador-Oliván et al., 2018) concludes that the search methods are poorly reported, and that it is necessary to improve this information in order to replicate such systematic studies. With this objective in mind, a group of specialists in information and library science and evidence synthesis methodology has proposed a standardised data structure to report all the necessary details that allow the reproducibility of database searches, and enable their reuse and interoperability between information systems (Haddaway et al., 2022).

We need to mention that methodological flaws are not exclusive to bibliometrics. Other research techniques, such as clinical interventions (Chou et al., 2007), expert panels (Evans, 1997), or interviews (Alsaawi, 2014) also present shortcomings, which limit the reproducibility and reliability of results. Notwithstanding, the focus of this work is on methodological limitations in bibliometric research.

Sustainability is a journal published since 2009 by the Swiss publisher MDPI, which publishes 418 open-access titles through a golden route. This publisher is questioned for its aggressive editorial practices, for the massive publication of special issues and for the laxity of its peer review processes (Repiso et al., 2021; Siler, 2020), which, together with its high rates of self-citation and intra-citation, has led some authors to describe it as a predatory publisher (Oviedo-García, 2021). Sustainability was listed as a questionable journal in products released by national authorities, such as the Norwegian Register of Scientific Journals, Series and Publishers. It was also part of the initial version of the China Early Warning Journal List, which was created in 2020 by the Chinese Academy of Sciences to combat academic misconduct. Among the 22 MDPI journals included in the list, Sustainability was the only one classified as “medium risk” (Early Warning Journal List, 2020). Nevertheless, it was subsequently removed from the list in 2021 and 2023 releases.

However, MDPI is also a publisher valued by authors, especially for its high bibliometric indexes, and for its very fast review and publication times, with an average of approximately 40 days from manuscript submission to publication date (Csomós & Farkas, 2022; Repiso et al., 2021). These reasons are probably behind the increase in the market share of this publisher, which in some countries, such as Romania or Poland, is above 30% (Csomós & Farkas, 2022). In Spain, MDPI was the publisher with the highest number of works published in 40 out of its 70 universities in 2021 (Delgado López-Cózar & Martín-Martín, 2022).

Scientometrics, started in 1978, is considered the flagship journal of bibliometrics (Chen et al., 2002), and since its inception has been the main publication forum for academics and professionals in this field. Although there are currently numerous options for the dissemination of works in the field of metric studies of science, such as the Journal of Informetrics, Quantitative Science Studies, or Research Evaluation, this outlet is the oldest and most productive in the field of knowledge, and is considered in the academic community to be a prestigious and relevant journal that publishes quality bibliometric studies.

In order to identify bibliometric publications in both journals, we used the search equation designed by González-Alcaide (2021), which, in turn, is based on previous studies. This search equation was:

TS=(bibliometric* OR scientometric* OR webometric* OR altmetric* OR informetrics* OR “citation analysis” OR “citation study” OR “scholarly productivity” OR “publication analysis” OR “scholarly impact” OR “patent citation”) AND PY=(2019-2021) AND SO=(Sustainability OR Scientometrics)^②.

The data source searched was the Web of Science Core Collection, selecting the Science Citation Index (SCI) and Social Science Citation Index (SSCI) databases. The search was carried out on 04/12/2021 for the 2019-2020 period and updated on 09/14/2022 for including the year 2021.

This search yielded a total of 1005 papers, 419 from Sustainability and 586 from Scientometrics. On this set of results, the following inclusion criteria were applied to determine the final sample to be analysed (Figure 2). These criteria were:

C1. It is an article or a review (document types Article or Review, according to the classification made by Web of Science).

Figure 2.

Papers that did not meet criteria C1 or C2 were excluded from the final sample without the need to review their content. Criterion C3 was met in all cases. To determine the suitability for criterion C4, each of the papers was reviewed manually by the authors. To this end, the title, abstract and, when necessary, the full text of the papers were analysed. In the case of Sustainability, most of the papers reviewed were thematic bibliometric analyses (93.5% of the sample after the exclusion of C1-C2 papers). In contrast, the thematic bibliometric analyses published in Scientometrics constituted 21.7% of the results retrieved in the initial search (after the exclusion of C1-C2 papers). Thus, the final sample was composed of 508 papers, 77% from Sustainability (391 papers) and 23% published in Scientometrics (117 papers).

The following analysis variables were defined.

Search strategy: The search carried out by the authors, with an indication of the terms used, the field where the search is carried out, and the filters or limitations indicated. In the case of different searches being indicated, the most general one is taken into account, in line with what is indicated in the Sample variable.

Based on these variables, seven different types of shortcomings were identified for each of the articles.

Shortcoming search string. It is marked as a flaw if the paper does not break down the keywords used or the search fields used. It is not necessary for the paper to indicate the exact search string used as long as it indicates the keywords, how they are combined (Boolean operators) and the search field.

Therefore, each article has a number of shortcomings ranging from 0 (no flaws) to 7 (all flaws).

The results were expressed as medians and interquartile ranges (IQRs) for continuous variables, and numbers and percentages for categorical variables. The comparison of subgroups was done by the Chi-square test or 2 x 2 tables, and the Fisher exact test when the number of events was less than 5. The Mann-Whitney U test was used for quantitative variables with non-parametric distribution. We used the Welch’s t-test to measure equality of variances (Welch, 1947) and the Hedges’ g to compare the magnitude of effects across different groups (Krzywinski & Altman, 2014). R Statistical Software (Foundation for Statistical Computing, Vienna, Austria), using the ggstatsplot approach (Patil, 2021) was employed.

The sample size was n=508, with a median of 846 papers analysed per study (average: 11,875.2). The minimum number of papers analysed was 14, and the maximum was 2,685,356. Notably, the number of papers studied in articles published in Scientometrics was much higher, with 2,009 papers (average: 40,615.4) compared to Sustainability, with 678 articles (average: 4,016).

The periods analysed were also longer in Scientometrics (median: 25 years) compared to a median of 22 years for studies published in Sustainability. In terms of the number of sources used, the median was one in all cases, although the mean was 1.9 for Scientometrics and 1.4 for Sustainability. In other words, in general, the studies published in Scientometrics were broader in their samples and in the years analysed, as well as in the number of sources used (table 1).

Table 2 shows the information on the sources used in the primary studies, as well as the types of documents and languages considered.

In terms of sources, 46.5% of the sample used only Web of Science (WoS) as a data source, and 26.8% used only Scopus. Only 4.1% of the papers analysed did without both data sources, with the use of alternative sources to WoS or Scopus being more frequent in articles published in Scientometrics (9.4%) than in those published in Sustainability (2.6%).

With regard to the document types, in the sample as a whole, the most frequent typology analysed was “article”, together with other specifically indicated types (review, letters, proceeding papers, book chapters, etc.), as this situation occurred in one-third of the works analysed. The next most common situation was, in the case of Sustainability, the analysis exclusively of the “article” typology (31.2%), while in Scientometrics it was the analysis of all document types (19.7%), although with a very similar percentage to that of the “articles only” category.

With regard to the language variable, if we exclude the papers for which this data is not provided, the most frequent was the analysis of papers only in English (33.7%). This is the case for Sustainability, while Scientometrics detects the same number of papers analysing documents only in English as in all languages (17.9%). A tiny number of papers are found that analyse the results of papers in English and some other specific language(s) (2.6%), or that study papers in a language other than English (specifically, Chinese in one case and Ukrainian in the other) (0.4%).

In the sample as a whole, a total of 1,304 shortcomings are found, 1,000 in the case of Sustainability, and 304 in the journal Scientometrics, with an average of 2.6 shortcomings per article in both journals, and a median of 2 flaws in both cases (Figure 3).

Figure 3.

t Welch: test to measure equality of variances; g Hedges: test to compare the magnitude of effects across different groups; CI: confidence intervals; n obs: number of observations.

Table 3 and Figure 4 show for each of the seven variables the percentage of papers with shortcomings for both journals. Sustainability has a slightly higher percentage of flaws than Scientometrics in the variable Search String (38.4% vs. 35%, p = 0.295), which is more pronounced in the parameters Sample (8.7 points difference, p = 0.039), Period (10.1 points difference, p = 0.024) and Sources (7.4 points difference, p = 0.102).

Figure 4.

The bibliometric analyses published in Scientometrics showed a greater number of shortcomings than those published in Sustainability in the criteria Search date (4.3 points difference, p = 0.241), Document types (7.8 points difference, p = 0.039), and Language, with the latter parameter showing the greatest difference between subsamples, with more than 20 points difference between Scientometrics (61.5%), and Sustainability (41.2%) (p < 0.001).

Finally, the analysis of the frequency of articles according to the number of flaws shows similar profiles for both journals, with 2 being the most common number of shortcomings for each of the subsamples. Up to 28 papers were detected without flaws, which represents a percentage of 5.1% in Sustainability and 6.8% in Scientometrics (p = 0.404). At the other extreme, there are 127 papers with four or more shortcomings, with a percentage of 23.9% for Sustainability and 29.1% for Scientometrics (p = 0.267).

Figure 5.

Of particular concern is the lack of transparency in the reporting of executed searches. It should be borne in mind that in this variable we have not taken a criterion as demanding as the one proposed by PRISMA-S (Rethlefsen et al., 2021), such as “Include the search strategies for each database and information source, copied and pasted exactly as run”, but that a search was considered reproducible when the keywords, Boolean operators and the search field were specified. The most common reason for this parameter to be qualified as a shortcoming was that the search field was not reported, but only the keywords. This is relevant because the number of results retrieved in the databases differs significantly depending on the field in which the search is executed. In some cases, however, not even the specific terms used were mentioned. It was not the purpose of this paper to determine the quality of the search equation, but a brief review reveals that in many cases the terms used are not the most appropriate for the objectives of the specific study, sometimes because they are too broad, sometimes because they are too specific. In other cases, the use of inverted commas in the transcription of searches is not rigorously employed, or it is not clearly indicated how the terms have been combined using Boolean operators. Finally, we have also found in some of the studies that several searches are broken down, without it being clear which is the one taken into consideration in the work, or how they have been combined.

The development of the search strategy should be a well-documented process, clearly specifying the terms used, the search fields where the query was executed, the operators used to connect the search(s), as well as special characters (such as asterisks or inverted commas), as any small change in the search string may generate different results (Romanelli et al., 2021). The use of incorrect search strategies affects the precision and completeness of results, leading to unreliable conclusions (Salvador-Oliván et al., 2019). Examples of such strategies include utilizing a single keyword to define a field of study, improperly utilizing Boolean operators to combine search terms, using ambiguous terms, or selecting inadequate search fields. In this regard, knowledge of the database syntax, coverage, and changes over time is essential to avoid errors in interpreting results. The cases of thematic analyses in Artificial Intelligence (Liu, 2021) and Climate Change (Liu, 2022) are paradigmatic of the challenges involved in performing a rigorous and error-free bibliometric analysis. In this sense, we recommend the development of more precise search strategies, for which various techniques can be employed, such as using search equations based on the previous literature, deriving terms from artificial intelligence tools, or having committees of experts in the field that may propose/refine strategies (Rethlefsen et al., 2021). Finally, it is highly advisable to include documentalists or librarians to help generate more effective search strategies, as this is associated with higher quality and reproducibility of the research (Koffel, 2015).

In this research, we have set a threshold of 200 papers to consider an analysis as using a small sample, so that the bibliometric approach is probably not the most appropriate for studying the selected topic. While this threshold is derived from the recommendations of Donthu et al. (2021) and has been considered as a minimum requirement in some thematic bibliometric analyses (Gao & Wang, 2022; Kumar et al., 2021; Tamala et al., 2022), it must be acknowledged that the 200-paper threshold is somewhat arbitrary. However, we believe it to be a reasonable threshold, considering that the papers that used a sample of less than 200 articles in our research analysed an average of only 8.2 papers per year. It seems difficult to identify trends or derive robust conclusions on topics with such small numbers.

The fact that as many as 12 of the papers analysed samples of less than 50 papers, or that 57 papers used samples of less than 100 papers, shows that either the topics are so specific that a bibliometric analysis does not make sense, or that the searches are poorly designed. Furthermore, in some cases, the initial search retrieved tens of thousands of papers, which after review by the authors and the exclusion of irrelevant articles remained in the hundreds or tens of papers, which also points to a significant dissonance between the search strategies designed and the final results, impeding the reproducibility of such analyses (Boyack et al., 2022).

It is also striking that up to 42.5% of the papers studied committed some kind of shortcoming in relation to the period analysed, the most common being that incomplete years were studied (the last of the period analysed), which prevents a strict comparison with previous years. Given that one of the main objectives of thematic analyses is usually to verify the quantitative evolution in the number of papers generated this methodological limitation prevents us from reaching solid conclusions about the strength of a given research topic. We are not talking about a few days to complete the year, but rather that in up to 70 of the papers, in the last year of the sample, a period of less than six months was analysed. It is probably the authors’ desire to make their papers more up-to-date and relevant that leads them to include the current year in the analysis.

For its part, the indication of the search date is another essential element to allow the reproducibility of the research, given the dynamic nature of data sources and their changing character. Unless there is an error or omission on our behalf, we have not located other studies that study this variable in the bibliometric analyses. In the only study detected that analyses this variable in systematic reviews in the field of Library and Information Science, the search date was an element that was omitted in 56.5% of the papers (Salvador-Oliván et al., 2018). On the other hand, it is a parameter that has been studied for systematic reviews in different medical fields, detecting between 0% and 12% of papers that did not indicate the search date (Beller et al., 2013; Biocic et al., 2019; Franco et al., 2018; O’Donohoe et al., 2021; Sadeghi-Ghyassi et al., 2022). The specific date of search is an element that needs to be reported in a systematic review, according to PRISMA-S (Rethlefsen et al., 2021) and which we believe should be extended to bibliometric studies, in order to allow reproducibility of research.

In this study, we have verified the limited diversity of the sources used in the bibliometric studies analysed, given the overwhelming dominance of Web of Science and Scopus. Only in a residual 4.1% of the papers was neither of the two sources used. The use of a greater number of local sources is lacking, which would provide a new scenario for comparisons, for example, between the literature circulating at the international level and that which is recorded in local or regional sources.

Another significant conclusion is that 75.2% of the analysed articles used a single data source (WoS or Scopus) in their bibliometric analysis. Although we do not consider this as a shortcoming, current recommendations in a responsible metrics scenario suggest the use of a diverse range of databases, avoiding the use of single sources that may show biased results (Cabezas-Clavijo & Torres-Salinas, 2021). Web of Science and Scopus are high-quality databases, designed, among other purposes, for bibliometric analyses, providing extensive possibilities for data querying and downloading. Their characteristics, strengths, and weaknesses have been repeatedly highlighted in the literature (Aksnes & Sivertsen, 2019; Mongeon & Paul-Hus, 2016; Visser et al., 2021), enabling authors to acknowledge their limitations in their bibliometric studies. It contrasts with the difficulty of extracting structured information from other data sources which are not originally designed for performing bibliometric analyses. We consider that a greater use of alternative sources, such as DOAJ, Redalyc or Dialnet, is directly linked to the fact that the producers of these resources allow a more appropriate use for bibliometric research, for example by facilitating the search by a greater number of fields, support for combined searches, or the downloading of records in standardised formats. This low use of alternative sources may be overshadowing research topics and trends that are locally relevant but not visible in mainstream journals.

On the other hand, the low level of disaggregation of the specific indexes used within Web of Science is certainly surprising. We have identified that, considering only papers using WoS, 65.6% of the studies in Sustainability and 57% of those published in Scientometrics fail to provide this information. In the case of Scientometrics, this figure is slightly higher than reported previously (Liu, 2019). It is advisable to indicate the subset used within WoS, given that institutional subscriptions may include a greater or lesser number of indexes, or differ in the period covered.

Authors should be aware that results obtained from Web of Science depend on the type of access granted to each institution for the database. This implies that the particular indices that an organization can search and the coverage of a certain database might vary from one institution to another (Gusenbauer & Haddaway, 2020). However, these variations are sometimes unknown for researchers, which may affect the accuracy of their reporting and hinder the reproducibility of the research. Likewise, wide-used databases may also present mistakes in their references. These errors could be due to the quality of metadata provided by publishers, such as mistakes in the original paper, or to errors introduced in the data processing stage by the database operators (Pranckutė, 2021). Although the rate of errors for Scopus and Web of Science is relatively low, it may affect outcomes, specially when working with small samples (Franceschini et al., 2016). Therefore, researchers should be familiar with the databases characteristics and limitations before choosing any of them for bibliometric research purposes.

As for the types of documents, 17.9% of the papers did not specify this information. When it is detailed, it is found that almost a quarter of the papers (24.4%) analyse all the document typologies retrieved in their search strategy, while the remaining papers analyse the typology “article”, either exclusively or combined with other typologies. Given that the bibliographic databases include not only research documents, but also other types of materials (editorial material, proceedings papers, meeting abstracts, corrections, book reviews, book chapters, etc.), it is advisable to indicate the exact document types analysed, as this may condition the final results. It is also important to bear in mind that some typologies, such as proceeding papers, can have delays of up to three years in their indexing in Web of Science (Maddi & Baudoin, 2022), which can have a notable effect on bibliometric analyses in certain areas.

Almost half of the papers (45.9%) did not indicate the languages of the documents analysed. Although most of the papers use Web of Science or Scopus, sources that mainly compile documents in English (Vera-Baceta et al., 2019), it should not be forgotten that in up to 18 Web of Science categories, mainly in the Humanities, the percentage of documents in languages other than English exceeds 20%, and in five of these 18 it exceeds 50% (Liu, 2017). The fact that there are almost twice as many analyses of papers in English alone (171) than in all languages (89) highlights the need to study not only the scientific production visible in English in the major databases, but also that produced in other languages, given that the subject matters, methodologies, approaches and findings may differ significantly. In this sense, it would be desirable that the more inclusive perspective advocated by statements such as the Helsinki Initiative on Multilingualism in Scholarly Communication (Federation of Finnish Learned Societies et al., 2019) be more clearly incorporated into thematic bibliometric studies, in order to endow them with greater academic and social value.

As it cannot be assumed that there is a default mode of performing a bibliometric analysis (e.g., considering all document types, or all languages), the reporting of these variables is key to assure the transparency and reproducibility of results (Rethlefsen et al., 2021). Conceptually, it is important to specify which document types are selected, as not all of them may convey relevant scientific information (Glänzel & Moed, 2002). Likewise, the choosing of only one language (e.g., English) can introduce biases through certain topics or research lines, so researchers in the field should always make clear the methodological decisions taken into account.