Open Peer Review in Scientific Publishing: A Web Mining Study of PeerJ Authors and Reviewers

Scientific research has entered a new era of Open Science, which aims to make all aspects of the scientific research process open and available by promoting new models for the dissemination of findings and the peer review process. Peer review is one of the cornerstones of scholarly inquiry, where peers assess the merits of research to determine if the claims made, methods used, findings and conclusions adhere to accepted practices in conducting scholarship. Assessment by peers also performs an important gatekeeping function, where poorly conceived research is not recommended for publication. The gold standard for peer review has been blind pre-publication review, where reviewers do not know the identities of the authors and vice versa. The longstanding rationale has been that blind review will limit bias in the review process if all parties remain anonymous. There has been growing concern with this traditional approach to peer review. Woosen (2015), for example, reports in an article in The Chronicle of Higher Education that some publishers are now making changes to the peer review process to fix what some call a broken system. Criticisms of the closed peer review process include biases, failure to weed out flawed manuscripts or the rejection of innovative ideas that do not fit believed norms, and long delays in publishing. However, not all scientists see the need for radical changes in the peer review process.

To date, there has been much research that has investigated different aspects of Open Science such as Open Access (OA) to documents that report research findings in OA journals or publicly accessible repositories, and open data, which make the data collected, generated, or processed for research studies openly available to others. Open Access to scientific journals is growing steadily (Laakso & Björk, 2012). Although many OA journals have adopted single or double blind Closed Peer Review (CPR), a number of more recent OA journals launched in this century have not passed the scrutiny of quality control (Bohannon, 2013), and may be considered predatory journals (Bartholomew, 2014). Open Science cannot succeed if the peer review process is flawed or compromised. One aspect of Open Science that has not been widely adopted and investigated is that of Open Peer Review (OPR), which makes pre- or post-publication reviews of scholarly works publicly available. In recent years, several OA journals have pioneered the OPR process to vet manuscripts for publication. Today, the notion of OPR is still evolving; diverse models of OPR have been implemented from the most open and interactive process, such as by F1000Research, to blind peer review followed by optional publishing of review history, such as by PeerJ. A dual model of peer review as used by Papers in Physics allows the author to choose, at the time of submission, either CPR or OPR. Despite the lack of standards or de facto models, OPR journals are growing fast. The number of OPR journals indexed by the Directory of Open Access Journals

https://doaj.org

The idea of OPR is not new, but it has not been widely adopted to date. In an early investigation of this issue, McNutt et al. (1990) examined the quality of submitted reviews to a medical journal based on blind and signed reviews. The authors found there was no association between review quality and signing, but concluded blinding improved the quality of reviews based on human judgments. This latter conclusion has changed over time. Proposals for OPR extend back at least to the early days of Web-based Open Access journals. Sumner and Shum (1996) proposed pre- and post-publication OPR (which they called computer-supported collaborative argumentation) for a newly created electronic OA journal, arguing that OPR was central to the journal’s operation and for opening up scholarly debate. The British Medical Journal (now BMJ) began experimenting with OPR in the late 1990s to determine if peer review quality was different when referees identify themselves. van Rooyen et al. (1999) in studying OPR in BMJ with blind reviews noted that there was no difference in quality between the open and blind reviews based on human judgments. They concluded there was no evidence to support that the traditional blind approach to peer review resulted in superior reviews. They also posited that if reviewers were identifiable, they might put greater effort into their reviews. Although these experiments with BMJ began more than 15 years ago, it was not until September 2014 that the journal fully adopted pre-publication OPR

http://www.bmj.com/about-bmj/resources-authors/peer-review-process

The debate whether the traditional peer review model should be modernized to promote transparency in the referee process or to accelerate dissemination of scientific discoveries has been a topic of continued interest (e.g. Taylor & Francis Group, 2015). Scientists, such as Nobel Laureate Harald zur Hausen, openly endorse the OPR model

https://peerj.com/about/endorsements

Critics of CPR have argued that the blind peer review process is, at a minimum, flawed (Smith, 2006) or, at worst, broken (McCook, 2006). Aside from opinions (e.g. Kriegeskorte, Walther, & Deca, 2012; Whither Science Publishing, 2012) or theoretical discussion (e.g. Lee et al., 2013), there is a lack of empirical research examining the complex peer review process to provide fact-based insights. It is obvious that in the blind peer review process the original review reports are only accessible by the editors. As OPR journals are opening the peer review process and sharing review reports, researchers now have the opportunity to study the process and review the discourses. Furthermore, Web 2.0 technology has also provided useful tools to collect data to help in the analysis of these processes.

Led by the European Union, Open Science is moving towards the ambitious goal “to open access to scientific publications as the default option by 2020” (Council of the European Union, 2016, p. 12; Enserink, 2016). Publications for Open Access will still need peer review. In a recent article in Nature, Rennie (2016) states that OPR provides a more ethical approach to peer review and calls for rigorous studies to compare various models (open, blind, pre- and post-publication, portable and so on).

We developed a descriptive model to frame the research questions and guide data collection (Figure 1). PeerJ uses a single blind, CPR process. As the established model for journal publishing, CPR assigns an editor to handle a submission and solicit peers to review the manuscript before making a decision. During the review process, the editor communicates with the corresponding author on review results and the decision. A manuscript may be resubmitted after revisions and go through one or more rounds of peer review before a final decision. In a published article, two types of review history can be accessed: for all articles, a summary of the processing dates (submission, revision, and acceptance), and for some, if the author chose this option, a full peer review history. The full peer review history includes: (1) all previous versions of the article; (2) all decision letters corresponding to the versions; (3) all review reports corresponding to the versions (the reviewers may have signed the reports or remained anonymous), and (4) all rebuttal letters as attachments (See the dashed objects depicted in Figure 1).

Figure 1

As literature reports, OPR may adopt different levels of openness. The most transparent peer review model opens the entire review process, during which both reviewers and authors are known to each other. Several BMJ journals moved from CPR to this OPR model; F1000Reserch adopted OPR with a post-publication peer review implementation. Transparent peer review has one more decision element: whether to make the signed review reports public alongside the articles. A less rigid model allows reviewers the option of anonymity or signing their identity, and authors the option of publishing their reviews or keeping them private. A variable level of transparency exists between CPR and OPR. It is the combination of a CPR model with an OPR model that is of particular interest in the current research. Do authors choose to make their reviews available? How willing will reviewers be to identify themselves? Answers to these questions will inform decisions on how to adopt OPR.

To study the complex OPR process and understand the nature of OPR as it stands in current practice, this research examines the peer review process adopted by the OPR journal PeerJ. PeerJ

https://peerj.com

Nevertheless, PeerJ has succeeded as a fast growing OPR journal for research articles since its inception and gained its first (partial) Impact Factor of 2.183 (2015). The journal is of great interest because of its peer review policies and processes. PeerJ consists of three publications: PeerJ (Life, Biology, & Health Science), PeerJ Computer Science since May 27, 2015, and PeerJ Preprints (not peer reviewed). This research collected data from PeerJ (Life, Biology, & Health Science) and addresses the following specific research questions:

How have scientists accepted principles of Open Peer Review as represented in PeerJ?

This study adopts a Web mining approach to collect OPR data from PeerJ and applies several analytical techniques and tools. The use of Web analytics to study OA online journals faces unique challenges due to the complexity of the articles and associated objects as well as the lack of standards for OPR platform design. No well-developed computational tools exist for data collection, and quantitative and qualitative analysis except for a few project-based experimental tools that need to be modified for use. This Section describes the tools used to manage data collection, extraction, cleansing, restructuring, and analytical mining for the present study.

3.1

Web Scraping and Data Extraction

PeerJ publishes all articles as Web pages with a right sidebar for hyperlinks to a downloadable PDF file, a peer review history page, article level metrics, and sections of the article (Figure 2).

Figure 2

For the articles with a full review history, the review page has a summary of the publication process (dates on initial submission, initial decision, revision, acceptance, etc.) followed by the peer review history, including the editor’s decision (accept, major revision, or minor revision) and review reports (signed or anonymous), all versions of the article, and rebuttals. For the articles whose authors choose not to publish the full peer review history, the review page provides only a summary of the publication process. Because the data relevant to our research purpose were scattered across different pages and in diverse formats, we divided the data collection into two processes: (1) the use of Google Chrome SelectorGadget to collect each article’s data, and; (2) the use of a PHP program developed by one of the authors to crawl the full peer review history Web pages. Fortunately, the URLs for PeerJ articles are well structured with a domain and a sequential article number.

We extracted three sets of data:

The article data include the article identification (articleID is the sequential number of the URL), authors, publication dates (submission, acceptance, and publication), grants, and affiliations (author’s organization and country). Affiliations are associated with each article instead of individual authors in this study due to difficulties in definitively assigning authors to specific affiliations.

The Peer Review History: Summary: All published articles have a summary that includes the article’s identification (articleID is the same sequential number of the URL) and dates for submission, revisions, and acceptance; the summary history does not include the editor’s identity.

The Peer Review History: Review Reports: These are available only if the authors have opted to make their reviews public. The full history page includes the summary data as described in 2) plus the dated and named editor’s decision letters (each for a specific version), referees (either signed or anonymized as Reviewer 1, Reviewer 2, etc.), and referee’s review reports (each report is for a specific version). Some review reports also referred to marked manuscripts as being attached, etc. (Figure 3).

Figure 3

The data collected represent accepted papers published by PeerJ between its first published article in 2013 and February 4, 2016. This resulted in 1,643 articles with ID numbers from 1 to 1,676. The missing numbers were likely due to cancelled manuscripts or retracted papers. Of the 1,643 articles, 1,214 (73.89%) published full review histories. Ten of the 1,214 articles were accepted by the academic editors based on portable reviews submitted by the authors with the manuscripts. These portable reviews were review reports from other journals that had rejected the manuscripts. Excluding these ten articles with portable peer reviews, there were 3,569 PeerJ peer review reports for the 1,204 articles’ full review histories. Each review history page (Figure 3) includes time stamps, versions, the editor’s letters, reviewer reports, and author rebuttals as attached PDF files. Each report includes four sections: basic reporting, experimental design, validity of the findings, and comments for the author. Of the 3,569 review reports, 85 were submitted as attachments in various formats (comments inserted to the manuscripts, PDF or Word DOC) or the reviews contained no substantive content such as “no further comments.” There were 3,484 valid reviews after removal of the reviews without any content.

The available data that were not extracted include: (1) author rebuttals, which are shown in the full peer review history as downloadable files (DOC or PDF); (2) affiliations of the signed referees, which is available only if the referee’s profile is public (but most reviewers set their profiles to private), and; (3) files which reviewers attached as review reports or marked manuscripts.

This project collected 1,643 peer-reviewed articles, which PeerJ published from February 12, 2013 to February 4, 2016. The time period was further grouped into 12 data periods consisting of three-month intervals in some analyses. As noted above (see Section 3.1), 10 of the 1,643 articles were found to be special cases of portable reviews. The ten articles were included in the descriptive analysis (such as article’s country of origin, versions, and author’s decisions) but their external review reports were not included in text analysis of the PeerJ review reports.

4.2

The Proportion of the Articles with a Full Peer Review History

Of the 1,643 articles, 1,214 articles (73.89%) published a full peer view history (including 10 portable reviews). Figure 4 shows the trend of the published articles over three years as 12 data points. The published articles have been steadily increasing over time, but the proportion of the articles with a published full peer review history ranges between 64.79% and 83.46% without an obvious increase or decrease overall.

Figure 4

4.3

Comparison of Signed with Anonymous Peer Review Reports

The 1,214 articles’ full review histories include 2,855 decision letters by the academic editors (including the 10 special cases with portable review histories) and 3,569 peer review reports (excluding 10 portable review reports). Of the 3,569 review reports alongside the 1,204 articles, 1,543 (43.23%) were signed reports and 2,026 (56.77%) were anonymous reports. As depicted in Figure 5, all data points except for one show more anonymous reports than signed reports. Although the 8^th point for November 2014 to January 2015 had 27.08% more signed review reports, the subsequent data points have not followed this increase. If this data point and the last data point are excluded as outliers, the proportion of signed reports over the three-month periods ranged from 40.74% to 48.86%. No obvious trend is observed.

Figure 5

4.4

Peer Review Process — Time Delays

Himmelstein (2015) ranked 16 journals by measuring acceptance delays and publication delays. PeerJ had the shortest period for peer review with 74 days median time from receipt of a manuscript to acceptance for publication based on 777 articles, but ranked only 8^th with a median value of 22 days from acceptance for publication to actual publication. A time series analysis was performed to depict the time lags for acceptance delay and publication delay. Figure 6 plots the means and medians for the delays over the three years in a three-month interval. Publication delays remain close to 25 days over time, whether measured by mean or median number of days. Acceptance delays, that is the peer review time, have increased from a mean of 55 days to a mean of 106 days over the past three years. The mean and median values both show the same trend, although the median value is the measure used by publishers because the data usually are skewed and may contain outliers. Following a pre-publication CPR process, PeerJ faces the same challenges as most print CPR journals: time for peer review increases as the number of submissions increases. Our results are similar to the findings of Björk and Solomon (2013). They investigated scholarly peer reviewed journals for publishing delays using a stratified random sample of 2,700 papers from 135 journals. They found that delays ranged from nine months to 18 months. They also found that the online OA journals showed a shorter time from acceptance to publication. The various factors contributing to the journal publishing delays include disciplines, size of the journal, and authors’ revision time.

Figure 6

4.5

Time Taken to Submit First Round of Peer Reviews

The first round of peer reviews was mostly conducted on the first version of the manuscripts, except for four articles where it was conducted on the second version. The editors reviewed the original submissions, which the authors revised accordingly. The times taken were calculated based on the difference between the time stamp of the review report and the manuscript submission date. The reviewers may have received the manuscript review request on a later date. The receipt date is not recorded by PeerJ, so it is unknown. The descriptive statistics for the first round of review reports (N = 2,720) are Mean = 20.36 days (SD = 11.57) and Median = 18.00 days. This set was partitioned into two sets to compare differences between anonymous reports (N_a = 1,537) and signed reports (N_s = 1,183). The descriptive statistics are as follows: Mean_a = 20.70 days (SD = 11.75), Median_a = 18.00 days, ranging between 0 to 77 days; Mean_s = 19.90 days (SD = 11.31), Median_s = 18.00 days, ranging between 0 to 69 days (Figure 7). Although no significant statistical differences were shown across the two sets, the means and ranges show differences. We found that 15 review reports were submitted the same day or next day. In contrast, 280 review reports were submitted after more than 30 days. Further analysis is needed to compare the distributions of the time taken and the review text lengths (Section 4.8).

Figure 7

4.8

Comparisons of Time Taken to Submit Anonymous vs. Signed Reports

As mentioned in Section 4.5, the time taken to complete the first round of review shows a wide range. Figure 8 plots the 1,534 review reports from the set of 633 articles that each had at least one anonymous and one signed review report. For this set of articles, we can observe if the distributions of times differ between anonymous and signed review reports. Figure 8 plots the percentage of reports submitted over days. The two plots show similar patterns with a spike for signed reports at 13 days and for anonymous reports at 14 days. For anonymous reports, the plot shows 4 cases at 70 days or more. There were 15 review reports (0.98%) submitted the same day or next day and 280 review reports (18.22%) submitted after more than 30 days. This raises the question if there is an association between the time taken and the length of the reviews (See Section 4.9).

Figure 8

Although the current study provides insights into author and reviewer acceptance of OPR in PeerJ, the study of a single OA journal over a relatively short period of time does not permit the prediction of long-term acceptance of OPR or to generalize our findings to other OPR journals. Given the recent adoption of different OPR models by current or new OA journals, there is a lack of substantial datasets across large numbers of journals or over long periods to analyze and compare. Furthermore, the data are not always readily accessible using current computational tools. In addition, the peer review history data are not available for the articles for which authors chose not to publish full peer review histories. Finally, the anonymity of reviewers prevents an accurate count of how many reviewers contributed to the review process. To reduce the effect of these limitations, the analysis in this study has been at the level of the articles or review reports rather than at the level of authors or reviewers when appropriate.

This study has presented a detailed and systematic analysis of an OPR model as implemented in PeerJ, an OA journal that straddles the world of CPR and OPR. PeerJ article Web pages were systematically crawled to collect relevant article metadata and peer review history data. PeerJ is an international journal, with contributed articles by authors from 107 countries, but is still dominated by contributions from North America and Western Europe, with some representation from countries in East Asia and Oceania. A majority of articles (73.89%) have publicly available full peer review histories alongside the articles. The percentage of published full peer review histories has remained stable over the three-year period analyzed; the ratio of anonymous/signed reviewers has also remained stable during this period of time. Given a choice, the majority of authors publishing in PeerJ will make the peer review histories of their articles public. However, when it comes to reviewers signing their reports, the majority of reviewers still prefer to remain anonymous. As a CPR OA journal, the delay in peer review for PeerJ submissions has been increasing, from less than 50 days (Median) to 88 days (Median), despite the publication delay from acceptance being stable and less than 30 days. The volume of the journal has also grown, having experienced a four-fold increase in the number of articles published between the first time period and the most recent time period studied.

The vast majority (98.93%) of submissions have undergone at least one round of revision before acceptance, indicating that there is rigor in the review process. The lack of a significant difference between the number of rounds of reviews by signed and anonymous authors provides an indication that reviewer anonymity does not significantly influence when a submission is accepted. The finding that reports by signed reviewers could be longer than by anonymous reviewers based on the number of tokens suggests that signed reviewers may be putting more effort into their reports. However, this simple analysis does not reveal whether there are qualitative differences between the reviews of signed vs. anonymous reviewers and the statistical outcome was not definitive. This result is similar to the findings in Bornmann, Wolf, and Daniel (2012) that the comments in public peer review are much longer than the comments in closed peer review.

There is no significant difference in the time taken to submit reviews or the lengths of reports between the signed or anonymous review reports. This finding suggests that reviewers conduct their reviews in similar manner whether or not they sign their reports. Our results corroborate the findings of the experiment by van Rooyen, Delamothe, and Evans (2010). In their experiment, the participants were randomly assigned to a control or intervention group. The participants in intervention group were told that their review reports would be published alongside the paper. The authors found there was no significant difference in review quality between the two groups, although the intervention group took longer to review (mean difference 25 minutes).

The current research represents a first attempt to observe the characteristics of OPR and the peer review process using a Web mining approach to collect all the articles published in PeerJ up to February 4, 2016. One aspect of the reviews that the current research did not assess was the quality of the reviews based on semantic analysis of the reviews. Although text length provides an indication of the efforts by reviewers, a higher level of analysis is needed to focus on concepts that reviewers have identified to be of concerns or need additional work. Some of the texts show affective responses in the review process as well. For example, in a comment in the Section of “Comment for the author” the reviewer wrote “Cool paper.” As we move forward with more in-depth semantic analysis, we will investigate the conceptual and affective level of the reviews. Specifically, the analyses will focus on a comparison of the texts of signed and anonymous review reports using sentiment analysis and other natural language processing techniques, along with other text mining tools. The research will also be expanded to include additional OPR journals that use different models of OPR (e.g. pre-publication, post-publication, multistage) to provide further insights into OPR trends. Are scientists ready to accept a completely transparent peer review model? Which models will be adopted widely by different scientific disciplines? Our future research will address these important questions.