Gauging scholars’ acceptance of Open Access journals by examining the relationship between perceived quality and citation impact

Quite a few studies have evaluated the relationships between subjective assessments of reputation (or “quality”) and citation impact. These investigations have been undertaken with regard to entire journals (e.g., Haddawy et al., 2016; Hall, 2011; Hodge et al., 2021; Kulczycki & Rozkosz, 2017; Mahmood, 2017; Saarela & Kärkkäinen, 2020; Walters, 2017b, 2024; Walters & Markgren, 2019), individual articles (e.g., Abramo et al., 2019; Weitzner et al., 2024), individual researchers (e.g., Derrick et al., 2011; Guba & Tsivinskaya, 2023), and institutions (e.g., Szluka et al., 2023). However, a comprehensive literature search revealed no prior studies of the impact of OA status on the relationships between expert journal ratings and journal citation metrics. Specifically, there is no clear evidence that OA journals have especially high (or low) citation impact relative to their subjectively assessed quality.

As noted in section 1, skepticism about OA publishing can generate negative biases that affect scholars’ opinions of OA journals (Borrego, 2023; Dalton et al., 2020; Feenstra & López-Cózar, 2022; Halevi & Walsh, 2021; Hayman, 2016; Segado-Boy et al., 2022; Togia & Korobili, 2014). In turn, these negative attitudes may lead expert evaluators to assign lower reputation scores to OA journals than to subscription journals of comparable objective quality or impact. Although this assertion is reasonable, it has not been tested directly. Just one study, of limited scope, has addressed the question. Carvalho Neto et al. (2016) evaluated a random sample of 450 Brazilian journals included in the Qualis assessment exercise, which rates more than 24,000 journals on an eight-point scale based on their citation impact, their index coverage, and the subjective assessments of expert reviewers. Perhaps surprisingly, the OA journals in their sample had substantially higher Qualis ratings than the subscription journals. In fact, the OA journals—68% of the 450—comprise at least 80% of the journals in each of the top four rating categories but fewer than 60% of the journals in each of the lowest four categories. Although this finding is contrary to the expectations that underpin this investigation, it is based on data for just 450 journals published in Brazil—journals that may be systematically different from most international journals.

This study also draws on the idea that the accessibility of OA journals may give them greater citation impact than subscription journals of comparable quality. This assertion, set forth by Lawrence (2001) and subsequent authors (e.g., Copiello, 2019), is supported by some empirical investigations but not by others.

At the level of the individual article, OA status does tend to be associated with higher citation rates. However, this advantage can be fully attributed to factors other than OA status itself. For instance, OA and non-OA articles vary systematically with regard to field of inquiry, external funding, authors’ willingness to pay article processing charges, institutional support for OA publishing, and whether authors believe their findings are especially important or especially likely to appeal to a non-scholarly audience (self-selection). Many well-known studies have reported an OA citation advantage while controlling for just one or two (or none) of these confounding factors (e.g., Antelman, 2004; Dorta-González & Dorta-González, 2022, 2023a, 2023b; Eysenbach, 2006; Gargouri et al., 2010; Hadad & Aharony, 2023; Hadad et al., 2024; Koler-Povh et al., 2014; Sotudeh, 2020).

Studies that gauge the independent effect of OA status at the article level—those that include a full range of control variables or that randomly assign papers to OA or non-OA groups—reveal that all else equal, OA status has no independent impact on citation rates (Davis, 2011; Davis et al., 2008; Gaule & Maystre, 2011; Piwowar et al., 2018). This is not the same as no impact, of course. Moreover, there is some evidence of an independent effect for certain types of articles—those in particular subfields, for instance (Basson et al., 2021; Hubbard, 2017; Wang et al., 2021; Wray, 2016).

It is also possible that the influence of OA status on citation rates may be greater for entire journals than for individual articles. For instance, the title of an OA journal (e.g., PLoS Biology) often signals that the journal content is freely available, so potential readers know in advance that there will be no barriers to access. In contrast, OA papers published elsewhere—in hybrid journals or OA archives, for instance— have no such advantage. Simple comparisons of OA and non-OA journals do not support the idea that OA journals have greater citation impact (Björk & Solomon, 2012; Khan et al., 2023), although most studies of subscription journals that have “flipped” to OA status show that the transition to Open Access does bring a citation advantage (Asai, 2023; Bautista-Puig et al., 2020; Momeni et al., 2021). One recent paper reports mixed results (Wang et al., 2019), and one study of journals that have “reverse flipped” suggests that the transition from OA to subscription status also brings an increase in citations (Ming & Zhao, 2022).

Finally, two recent studies have shown that OA articles in hybrid journals— journals that publish both OA and non-OA articles—have higher citation rates than those published in fully OA journals (Hadad et al., 2024; Maddi & Sapinho, 2022). This challenges the idea that fully OA journals gain a citation advantage by signaling their OA status to potential readers. On the other hand, it can also be attributed to a high degree of self-selection—that authors choose to publish only their higher impact papers as OA articles in hybrid journals.

While at least five expert ratings of business journals are in current use, CABS is perhaps the most widely adopted by universities and other organizations that evaluate research contributions in business and economics. Moreover, CABS is the expert rating most closely related to the other four, based on the correlations among them (Walters, 2024). That is, CABS best represents the set of all five ratings.

The CABS ratings, assigned in March 2021, are based on the opinions of 53 experts appointed by the Chartered Association of Business Schools (UK), who consulted with other researchers as well as representatives of scholarly and professional societies. The rating criteria include reputation, selectivity, perceived scholarly impact, adherence to high methodological standards, and rigor and transparency of peer review. Raters were provided with a range of citation data, but those data were not formally incorporated into the ratings. The final ratings, from 4* (most important) to 1 (least important), were assigned after discussion and consensus among the raters. Overall, 3% of the CABS journals were assigned ratings of 4* (recoded as 5 in the current study). Six percent have ratings of 4, 19% have ratings of 3, 32% have ratings of 2, and 41% have ratings of 1 (Chartered Association of Business Schools, 2021).

Four citation metrics were selected for this investigation. Two of them, 5IF (five-year Impact Factor) and CiteScore, are among the indicators most often reported by publishers and used by research evaluators. Two others, AI (Article Influence score) and SJR (SCImago Journal Rank), are the citation metrics most closely related to CABS (Walters, 2024). 5IF and CiteScore are straightforward; they represent, essentially, the average number of citations accrued in the 1–3 years since publication. In contrast, AI and SJR are weighted and normalized; they account for both (a) the impact of the journals that cite each article and (b) differences in citation rates across fields of study (Bornmann, 2020; González-Pereira et al., 2010; Guerrero-Bote & Moya-Anegón, 2012; SCImago Research Group, 2007; Walters, 2017a; West et al., 2010, 2013). Further details about the CABS ratings and the citation metrics, including frequency distributions and correlations, are presented elsewhere (Walters, 2024).

The journals included in the analysis are those covered by both CABS and either Journal Citation Reports (Web of Science) or Scopus (Chartered Association of Business Schools, 2021; Clarivate, 2023; Elsevier, 2023). The selection standards of Web of Science and Scopus ensure that predatory or marginal journals are excluded, and none of the OA journals in this analysis appear on Beall’s List (Beall, 2021; Clarivate, 2024; Meester, 2021).

Within the set of 1,561 journals, the 85 OA journals are those that make both current issues and back issues freely available online. Journals with optional or hybrid OA status—those in which only some articles are freely available—were not counted as OA, nor were “subscribe to open” journals, where the journal becomes freely accessible only when a particular number of subscriptions have been sold and maintained. Likewise, journals that adopted OA status in 2021 or later were not regarded as OA, since they were still subscription journals at the time of the CABS evaluations. Information on OA adoption dates was obtained from the Directory of Open Access Journals, the journals’ web sites, and other online sources.

Four regressions were conducted for the full set of journals. Each used a single citation metric—5IF, CiteScore, AI, or SJR—and an OA dummy variable (OA or non-OA) to predict CABS. The goal was not to identify the determinants of CABS ratings, but to evaluate the impact of OA status. Likewise, four regressions were conducted for the set of business journals (i.e., excluding those in the CABS economics, general psychology, and social science categories) and four for the set of economics journals.

In each regression, the citation metrics were entered in square root or log₁₀ form. This approach, adopted by several recent authors, was undertaken to maintain linearity, to ensure the best fit, and to account for the strong right skew of the citation distribution (Bradshaw & Brook, 2016; Saarela & Kärkkäinen, 2020; Thelwall, 2022; Thelwall et al., 2023; Walters, 2017b). For this investigation, the substantive results are the same regardless of whether the citation metrics are transformed.

Although we cannot assume measurement invariance across the five CABS ratings—that the difference between ratings of 1 and 2 is equal to the difference between ratings of 4 and 5, for instance—there is good justification for treating CABS as a continuous variable in the regressions. There are three reasons for this. First, the underlying constructs (such as quality or reputation) represented by scores that combine individuals’ ratings may be continuous even when the actual rating categories are not. Second, there is reliable evidence that violations of measurement invariance do not necessarily invalidate the results of techniques such as OLS regression. Finally, the use of methods meant for ordinal variables would result in the loss of valuable information about the shape of the variables’ distributions (Robitzsch, 2020, 2022; Robitzsch & Lüdtke, 2023).

Because the study data include the entire population—all the journals for which both CABS ratings and citation data are available—there is no need to draw conclusions about the population on the basis of sample characteristics (i.e., no need for significance tests). Nonetheless, nearly all the results achieve statistical significance if we assume that our data refer to a larger hypothetical population. The significance levels reported in the regression results are for Student’s t test, two-tailed.

Tables 1–5 present descriptive statistics and regression results for the set of all CABS journals—the journals in all 22 CABS subject categories. The OA variable was coded 1 for OA journals and 0 for non-OA journals.

As shown in Table 1, OA journals in business and economics tend to appear considerably lower in the status hierarchy than conventional subscription journals, in terms of both subjective reputation (CABS) and citation impact (5IF, CiteScore, AI, and SJR). The OA/non-OA differential is more pronounced for CABS than for the citation metrics, when expressed in standard deviations.

Moreover, the CABS ratings of the OA journals are consistently lower than the ratings we might expect based solely on the journals’ citation impact. (See the OA coefficients in Tables 2–5.) This trend can be seen across all four citation metrics. For instance, the OA business and economics journals have, on average, actual CABS ratings about 0.43 points lower than those of non-OA journals with the same 5IF scores. A similar differential can be seen for the CiteScore equation, although the effect of OA status is not as great when AI and SJR are used to predict CABS. Taken together, these results show that expert reviewers evaluating OA journals do assign them ratings that are low relative to their citation impact. That is, OA business and economics journals have high citation impact relative to their CABS reputation scores.

The relationships shown in Table 1 persist when only those journals with a clear business focus are included in the analysis—when the journals in the CABS economics, econometrics and statistics; psychology (general); and social sciences categories are excluded. Like Table 1, Table 6 reveals both low CABS ratings and low citation impact for the OA journals. It also shows an OA/non-OA differential that is greater for CABS than for the citation metrics, in terms of standard deviations. And, as before, the mean CABS rating of the OA journals is consistently lower than the mean rating we would expect based solely on the journals’ citation impact. (See Tables 7–10.) On average, the OA business journals have actual CABS ratings about 0.42 points lower than those of non-OA journals with the same 5IF scores. The OA differential is 0.40 points when CiteScore is used as the citation metric, 0.16 when AI is used, and 0.21 when SJR is used. Again, we can see that the OA journals have low subjective ratings relative to their citation impact, and that this relationship persists across all four citation metrics.

Tables 11–15 present very similar results for the journals in the CABS economics, econometrics and statistics category. On average, the OA economics journals have actual CABS ratings about 0.43 points lower than those of non-OA journals with the same 5IF scores. As the tables show, the disparity between citation impact and reputation is even stronger for OA economics journals than for OA business journals.