Scientific research has entered a new era of Open Science, which aims to make all aspects of the scientific research process open and available by promoting new models for the dissemination of findings and the peer review process. Peer review is one of the cornerstones of scholarly inquiry, where peers assess the merits of research to determine if the claims made, methods used, findings and conclusions adhere to accepted practices in conducting scholarship. Assessment by peers also performs an important gatekeeping function, where poorly conceived research is not recommended for publication. The gold standard for peer review has been blind pre-publication review, where reviewers do not know the identities of the authors and
To date, there has been much research that has investigated different aspects of Open Science such as Open Access (OA) to documents that report research findings in OA journals or publicly accessible repositories, and open data, which make the data collected, generated, or processed for research studies openly available to others. Open Access to scientific journals is growing steadily (Laakso & Björk, 2012). Although many OA journals have adopted single or double blind Closed Peer Review (CPR), a number of more recent OA journals launched in this century have not passed the scrutiny of quality control (Bohannon, 2013), and may be considered predatory journals (Bartholomew, 2014). Open Science cannot succeed if the peer review process is flawed or compromised. One aspect of Open Science that has not been widely adopted and investigated is that of Open Peer Review (OPR), which makes pre- or post-publication reviews of scholarly works publicly available. In recent years, several OA journals have pioneered the OPR process to vet manuscripts for publication. Today, the notion of OPR is still evolving; diverse models of OPR have been implemented from the most open and interactive process, such as by
The idea of OPR is not new, but it has not been widely adopted to date. In an early investigation of this issue, McNutt et al. (1990) examined the quality of submitted reviews to a medical journal based on blind and signed reviews. The authors found there was no association between review quality and signing, but concluded blinding improved the quality of reviews based on human judgments. This latter conclusion has changed over time. Proposals for OPR extend back at least to the early days of Web-based Open Access journals. Sumner and Shum (1996) proposed pre- and post-publication OPR (which they called computer-supported collaborative argumentation) for a newly created electronic OA journal, arguing that OPR was central to the journal’s operation and for opening up scholarly debate. The
The debate whether the traditional peer review model should be modernized to promote transparency in the referee process or to accelerate dissemination of scientific discoveries has been a topic of continued interest (e.g. Taylor & Francis Group, 2015). Scientists, such as Nobel Laureate Harald zur Hausen, openly endorse the OPR model
Critics of CPR have argued that the blind peer review process is, at a minimum, flawed (Smith, 2006) or, at worst, broken (McCook, 2006). Aside from opinions (e.g. Kriegeskorte, Walther, & Deca, 2012; Whither Science Publishing, 2012) or theoretical discussion (e.g. Lee et al., 2013), there is a lack of empirical research examining the complex peer review process to provide fact-based insights. It is obvious that in the blind peer review process the original review reports are only accessible by the editors. As OPR journals are opening the peer review process and sharing review reports, researchers now have the opportunity to study the process and review the discourses. Furthermore, Web 2.0 technology has also provided useful tools to collect data to help in the analysis of these processes.
Led by the European Union, Open Science is moving towards the ambitious goal “to open access to scientific publications as the default option by 2020” (Council of the European Union, 2016, p. 12; Enserink, 2016). Publications for Open Access will still need peer review. In a recent article in
We developed a descriptive model to frame the research questions and guide data collection (Figure 1).
Figure 1
Model of OPR implemented in

As literature reports, OPR may adopt different levels of openness. The most transparent peer review model opens the entire review process, during which both reviewers and authors are known to each other. Several
To study the complex OPR process and understand the nature of OPR as it stands in current practice, this research examines the peer review process adopted by the OPR journal
Nevertheless,
How have scientists accepted principles of Open Peer Review as represented in From which countries do the authors who participate in Open Peer Review originate? Has there been a change over time in the proportion of articles with full peer review histories appearing alongside published articles? Has there been a change over time in the proportion of reviewers who identify themselves (i.e. signed their reports) as represented in the full peer review histories? What does the peer review history reveal about the review process? What is the time frame for completion of peer review and for publication of accepted articles? What is the time taken to submit reports in the first round of review by referees? Is there a difference in the number of rounds of reviews prior to acceptance based on signed and anonymous reviews? Is there a difference between anonymous and signed review reports: in the effort measured by the number of sentences and number of tokens per review? in the time taken to submit the first round of review? Is there a correlation between the length of the report and the time taken to submit the report?
This study adopts a Web mining approach to collect OPR data from
Figure 2

For the articles with a full review history, the review page has a summary of the publication process (dates on initial submission, initial decision, revision, acceptance, etc.) followed by the peer review history, including the editor’s decision (accept, major revision, or minor revision) and review reports (signed or anonymous), all versions of the article, and rebuttals. For the articles whose authors choose not to publish the full peer review history, the review page provides only a summary of the publication process. Because the data relevant to our research purpose were scattered across different pages and in diverse formats, we divided the data collection into two processes: (1) the use of Google Chrome SelectorGadget to collect each article’s data, and; (2) the use of a PHP program developed by one of the authors to crawl the full peer review history Web pages. Fortunately, the URLs for
We extracted three sets of data:
The article data include the article identification (articleID is the sequential number of the URL), authors, publication dates (submission, acceptance, and publication), grants, and affiliations (author’s organization and country). Affiliations are associated with each article instead of individual authors in this study due to difficulties in definitively assigning authors to specific affiliations. The Peer Review History: Summary: All published articles have a summary that includes the article’s identification (articleID is the same sequential number of the URL) and dates for submission, revisions, and acceptance; the summary history does not include the editor’s identity. The Peer Review History: Review Reports: These are available only if the authors have opted to make their reviews public. The full history page includes the summary data as described in 2) plus the dated and named editor’s decision letters (each for a specific version), referees (either signed or anonymized as Reviewer 1, Reviewer 2, etc.), and referee’s review reports (each report is for a specific version). Some review reports also referred to marked manuscripts as being attached, etc. (Figure 3).
Figure 3

The data collected represent accepted papers published by
The available data that were not extracted include: (1) author rebuttals, which are shown in the full peer review history as downloadable files (DOC or PDF); (2) affiliations of the signed referees, which is available only if the referee’s profile is public (but most reviewers set their profiles to private), and; (3) files which reviewers attached as review reports or marked manuscripts.
The data required substantial cleaning and restructuring, for example, where reviews provided no substantive comments (e.g. “N.A.”, “See above”). We did not correct misspellings. The cleaned data were restructured using Java and Python to produce tab-delimited files. Additional data processing consolidated variations in country names. For example, country names were not standardized in
Quantitative data were tabulated for statistical analysis to characterize
This project collected 1,643 peer-reviewed articles, which
The affiliation of authors’ organizations and countries is complicated by the journal format, in which one author may be associated with multiple organizations and several authors may be associated with one organization. Each country, therefore, was counted once regardless of how many author affiliations were from a given country. The contributing authors of the 1,643 articles are affiliated with organizations in 107 countries. The top five countries (Table 1) for the number of articles contributed, which include the USA, United Kingdom, Australia, and Canada, are also the top contributing countries of articles that published full peer review histories. China and Germany switched order in ranking for the number of articles and the number of articles with published full review histories. The top five countries’ articles with full peer reviews are similar to the study of a post-publication OPR journal based in the United Kingdom,
Ranking and percentage of the countries whose authors participating in OPR.Ranking Country of origin Number of articles Country of origin Number of articles with published review reports Percentage 1 USA 700 USA 536 76.57% 2 UK 254 UK 202 79.53% 3 Australia 136 Australia 98 72.06% 4 Canada 100 Canada 69 69.00% 5 96 67 6 91 58 7 Spain 61 Spain 49 80.33% 8 France 59 France 45 76.27% 9 Netherlands 50 Netherlands 41 82.00% 10 Japan 49 Japan 31 63.27%
Of the 1,643 articles, 1,214 articles (73.89%) published a full peer view history (including 10 portable reviews). Figure 4 shows the trend of the published articles over three years as 12 data points. The published articles have been steadily increasing over time, but the proportion of the articles with a published full peer review history ranges between 64.79% and 83.46% without an obvious increase or decrease overall.
Figure 4
A comparison of articles with and without full peer view history.

The 1,214 articles’ full review histories include 2,855 decision letters by the academic editors (including the 10 special cases with portable review histories) and 3,569 peer review reports (excluding 10 portable review reports). Of the 3,569 review reports alongside the 1,204 articles, 1,543 (43.23%) were signed reports and 2,026 (56.77%) were anonymous reports. As depicted in Figure 5, all data points except for one show more anonymous reports than signed reports. Although the 8th point for November 2014 to January 2015 had 27.08% more signed review reports, the subsequent data points have not followed this increase. If this data point and the last data point are excluded as outliers, the proportion of signed reports over the three-month periods ranged from 40.74% to 48.86%. No obvious trend is observed.
Figure 5
Comparison of signed or anonymous review reports over three-month intervals 2013–2016.

Himmelstein (2015) ranked 16 journals by measuring acceptance delays and publication delays.
Figure 6
Delays in peer review and delays in publication.

The first round of peer reviews was mostly conducted on the first version of the manuscripts, except for four articles where it was conducted on the second version. The editors reviewed the original submissions, which the authors revised accordingly. The times taken were calculated based on the difference between the time stamp of the review report and the manuscript submission date. The reviewers may have received the manuscript review request on a later date. The receipt date is not recorded by
Figure 7
Boxplot of the first round of review time taken to submit report.

Acceptance of the types of reviews for articles by versions. Portable review reports were used by the editors.Version Anonymous Both Signed External Total 1 2 5 0 6 13 2 247 423 145 4 819 3 93 182 56 0 331 4 16 18 11 0 45 5 0 6 0 0 6 Total 359 633 212 10 1,214 Percentage 29.82 52.57 17.61 — —
The lengths of reviews were measured by the number of sentences and the number of tokens (i.e. words). The Python text processing module was used to calculate the numbers of sentences and tokens. This analysis focused on the first round of review reports because the versions affect the availability of review reports and the first review reports are presumably the most thorough and detailed. The subsequent reviews were either brief or not submitted. There were a total of 2,720 valid first round of review reports for the 1,204 articles after excluding the reports submitted as attachments or no substantive content (Table 3).
Review length of the first round of all articles (Sentences Tokens Anonymous Signed Anonymous Signed Mean 28.09 28.81 477.16 508.41 Std. Dev. 23.74 24.07 389.87 427.25 Median 22 22 368 394 Min 1 1 8 7 Max 275 167 2,604 3,747 0.432 0.050
There is no significant difference in review lengths using a
There is evidence that the same referee reviewing different articles is likely to submit reports of different lengths. For example, three reviews by one referee for three different articles have lengths of 213, 1,185, and 286 tokens, respectively. Another referee’s two reports for two different articles were 274 and 995 tokens, respectively. The differences in the lengths of the reviews and the reviewer decisions to sign their reviews may be influenced by a number of factors related to the manuscripts such as the nature of the research, the topics undertaken and the quality or rigor of the research being evaluated. It is, therefore, also necessary to analyze the set of articles in which each article had both signed and anonymous review reports. We identified 633 such articles (
First round of review of the articles (each had both signed and anonymous reports).Sentences Tokens Anonymous Signed Anonymous Signed Mean 27.71 27.90 478.93 495.25 Std. Dev. 21.83 24.16 385.02 431.15 Median 22.00 21.00 382.00 372.00 Min 1 1 9 7 Max 143 167 2,604 3,435 0.874 0.436
As mentioned in Section 4.5, the time taken to complete the first round of review shows a wide range. Figure 8 plots the 1,534 review reports from the set of 633 articles that each had at least one anonymous and one signed review report. For this set of articles, we can observe if the distributions of times differ between anonymous and signed review reports. Figure 8 plots the percentage of reports submitted over days. The two plots show similar patterns with a spike for signed reports at 13 days and for anonymous reports at 14 days. For anonymous reports, the plot shows 4 cases at 70 days or more. There were 15 review reports (0.98%) submitted the same day or next day and 280 review reports (18.22%) submitted after more than 30 days. This raises the question if there is an association between the time taken and the length of the reviews (See Section 4.9).
Figure 8
Time taken to submit review reports.

For the 15 quickest turnaround reviews (same or next day), the mean report length is 17.60 sentences or 284.73 tokens. For the much delayed reviews (more than 60 days), the mean report length of 25.75 sentences or 458.41 tokens. Spearman’s rho procedures were applied to test correlations between time taken and the length of the report. There was no significant correlation between the time taken and report length for both the number of sentences (
It must be pointed out as well that the longer turnaround could be influenced by longer periods of time between the manuscript submission date and the date when a reviewer receives her or his invitation. Therefore, additional stamps are needed to provide accurate assessments of the time taken to submit a review.
Although the current study provides insights into author and reviewer acceptance of OPR in
This study has presented a detailed and systematic analysis of an OPR model as implemented in
The vast majority (98.93%) of submissions have undergone at least one round of revision before acceptance, indicating that there is rigor in the review process. The lack of a significant difference between the number of rounds of reviews by signed and anonymous authors provides an indication that reviewer anonymity does not significantly influence when a submission is accepted. The finding that reports by signed reviewers could be longer than by anonymous reviewers based on the number of tokens suggests that signed reviewers may be putting more effort into their reports. However, this simple analysis does not reveal whether there are qualitative differences between the reviews of signed
There is no significant difference in the time taken to submit reviews or the lengths of reports between the signed or anonymous review reports. This finding suggests that reviewers conduct their reviews in similar manner whether or not they sign their reports. Our results corroborate the findings of the experiment by van Rooyen, Delamothe, and Evans (2010). In their experiment, the participants were randomly assigned to a control or intervention group. The participants in intervention group were told that their review reports would be published alongside the paper. The authors found there was no significant difference in review quality between the two groups, although the intervention group took longer to review (mean difference 25 minutes).
The current research represents a first attempt to observe the characteristics of OPR and the peer review process using a Web mining approach to collect all the articles published in
Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Acceptance of the types of reviews for articles by versions.
Version | Anonymous | Both | Signed | External Portable review reports were used by the editors. | Total |
---|---|---|---|---|---|
1 | 2 | 5 | 0 | 6 | 13 |
2 | 247 | 423 | 145 | 4 | 819 |
3 | 93 | 182 | 56 | 0 | 331 |
4 | 16 | 18 | 11 | 0 | 45 |
5 | 0 | 6 | 0 | 0 | 6 |
Total | 359 | 633 | 212 | 10 | 1,214 |
Percentage | 29.82 | 52.57 | 17.61 | — | — |
Review length of the first round of all articles (Na = 1,537; Ns = 1,183).
Sentences | Tokens | |||
---|---|---|---|---|
Anonymous | Signed | Anonymous | Signed | |
Mean | 28.09 | 28.81 | 477.16 | 508.41 |
Std. Dev. | 23.74 | 24.07 | 389.87 | 427.25 |
Median | 22 | 22 | 368 | 394 |
Min | 1 | 1 | 8 | 7 |
Max | 275 | 167 | 2,604 | 3,747 |
0.432 | 0.050 |
Ranking and percentage of the countries whose authors participating in OPR.
Ranking | Country of origin | Number of articles | Country of origin | Number of articles with published review reports | Percentage |
---|---|---|---|---|---|
1 | USA | 700 | USA | 536 | 76.57% |
2 | UK | 254 | UK | 202 | 79.53% |
3 | Australia | 136 | Australia | 98 | 72.06% |
4 | Canada | 100 | Canada | 69 | 69.00% |
5 | 96 | 67 | |||
6 | 91 | 58 | |||
7 | Spain | 61 | Spain | 49 | 80.33% |
8 | France | 59 | France | 45 | 76.27% |
9 | Netherlands | 50 | Netherlands | 41 | 82.00% |
10 | Japan | 49 | Japan | 31 | 63.27% |
First round of review of the articles (each had both signed and anonymous reports).
Sentences | Tokens | |||
---|---|---|---|---|
Anonymous | Signed | Anonymous | Signed | |
Mean | 27.71 | 27.90 | 478.93 | 495.25 |
Std. Dev. | 21.83 | 24.16 | 385.02 | 431.15 |
Median | 22.00 | 21.00 | 382.00 | 372.00 |
Min | 1 | 1 | 9 | 7 |
Max | 143 | 167 | 2,604 | 3,435 |
0.874 | 0.436 |
Editorial board publication strategy and acceptance rates in Turkish national journals Research misconduct in hospitals is spreading: A bibliometric analysis of retracted papers from Chinese university-affiliated hospitals The need to develop tailored tools for improving the quality of thematic bibliometric analyses: Evidence from papers published in Sustainability and Scientometrics The notion of dominant terminology in bibliometric research