Peer Review: How We Found 15 Million Hours of Lost Time

Lost time in the current peer review process

In the current journal submission process, rejection is common, yet reviews are rarely shared from one journal to the next. Even when reviews are passed along, the lack of an industry-wide standard means that each journal in the chain solicits its own reviews before making a decision. All of this leads to reviewers repeating work that has already been done on the same manuscript by other colleagues. We estimate that over 15 million hours are spent on redundant or unnecessary reviews – every year.

How did we get to 15 million hours each year?

The two key metrics for finding wasted time are **quantity** (how many manuscripts are reviewed and then rejected?) and **time** (how long does each submission one take to be reviewed?). While there are well over 28,000 peer reviewed journals, we only use 12,000 in our calculations since that is roughly the number of journals that are included in Thomson Reuters' prestigious Web of Science. The figure below shows how we calculated both quantity and time, and the descriptions and citations for the key steps in the process follow:

Calculation and source details:

1. 3,360,207 (English-language, STM) submissions per year

• Although the 2012 Mark Ware STM report1 showed that there are over 28,000 peer-reviewed journals, we focused our scope within just the 12,000 English-language STM journals identified in that same report, as they represent the most established and highest-volume journals.
• The average submissions per journal were shown in the Thompson Reuters data2 as 280 (total ScholarOne submissions divided by the count of ScholarOne journal sites). Calculating 280 submissions for each of the 12,000 journals equals 3,360,207 submissions per year.
• Note that this is submission-based data, not paper-based. A single manuscript that was rejected by one journal but then accepted by another within the same year would go through two review cycles and thus be recognized as two separate submissions.

2. 1,344,099 (40%) accepted submissions per year

• Thompson Reuters data2 reports 37% acceptance based on all submissions received and accepted within their system, but the Mark Ware PRC report3 estimated an average of 50%.
• We feel the Thomson Reuters data is more accurate than PRC data based on how the information was collected and how calculations were made. Combined with our own internal data and personal interviews with some of the largest STM publishers, we selected 40% as the best representation for this group of journals. 40% of our total submission number equals 1,344,099 accepted papers.

3. 705,652 (21%) submissions per year rejected WITHOUT Review

• The MarkWare PRC report3 stated **21%** as its estimate for submissions that are rejected without going through peer review, also known as a “desk rejection.”
• Although there is time lost and an opportunity cost to the author when this occurs and they have to try again with another journal, we are currently only focused on time spent on peer review. We do not factor this group in with our calculation of lost time.

4. 1,310,496 (39%) submissions per year rejected WITH Review

• The number of submissions that are sent to peer review but are then rejected is our key starting metric for calculating lost hours. We use the two preceding calculations to find this number.
• If 21% were rejected without review, and 40% were accepted, then the remaining submissions were rejected after the peer review process. Applying 39% to our total gives us 1,310,496 submissions.

5. 11.5 average reviewer hours spent per submission

• Data from the Mark Ware STM report4 provided us with an average (median) of five hours spent per review.
• The Mark Ware PRC report3 states that an average of 2.3 reviewers is used for each submission.
• Five hours * 2.3 reviewers equals 11.5 average review hours per submission.
• Note that this number only takes into account the time spent per submission by reviewers – it does not include time spent by the journal or publisher in coordinating the review process (e.g., recruiting reviewers, editorial check of reviews, review software costs) or other time spent processing these papers (e.g., screening, editorial review, technical check, other operational time).

6. 15,070,706 hours per year spent on redundant reviews

• Assuming 11.5 hours per submission and 1,310,496 submissions that were reviewed but then rejected leads to over 15 million hours -- every year.
• Since there are only 8,760 hours in a year, you can also think of it as equaling 1,720 years (if it was all one reviewer working 24 hours per day).

Have other questions? Found a better number with your own calculations? Feel free to write us and let us know!

This article originally appeared on the Rubriq blog. Special thanks to the Rubriq peer review team for contributing this research.

References

1. M. Ware, M. Mabe. The STM Report: An overview of scientific and scholarly journal publishing. International Association of Scientific, Technical, and Medical Publishers, Oxford, UK, 2012. Link.

2. Thomson Reuters. Global Publishing: Changes in submission trends and the impact on scholarly publishers. April 2012. Link.

3. M. Ware. Peer review: benefits, perceptions, and alternatives. Publishing Research Consortium, London UK, 2008. Link.

4. M. Ware. Peer Review: Recent Experience and Future Directions, _New Review of Information Networking_, 16:1, 23-53, 2011. Link.