Standardization of Scientific Peer Reviews
Inter-Rater Reliability (IRR): new initiative aims to bring standardization to ORISE scientific peer reviews
In some peer reviews, clear instructions on the basis for rating proposals may not be given to reviewers. As a result, the reviewers' final ratings can leave much room for interpretation thanks to varying degrees of reviewer knowledge and experience… all of which contribute to the final rating number.
For instance, the rating scale for a particular review is 1 to 5 (with 1 being the highest). One reviewer rates the proposal a 2, and a second reviewer rates the same proposal a 4 but notes “this is the best proposal” he’s ever seen. If the proposal was the best ever, why didn’t he rate the proposal a 1? And, was the range in their scores due to a difference in professional opinion or something else?
The answers to this question and others like it are what the Oak Ridge Institute for Science and Education’s (ORISE) Scientific Peer Review Program is trying to uncover through a new initiative termed “inter-rater reliability” or IRR. Although IRR may have a different meaning in another industry, for ORISE the goal of IRR is to establish standards or a level of agreement among reviewers when rating a proposal.
The need for IRR proves even more important for ORISE’s customers that must award/fund proposals in “rank order” or based on the highest ranking. The goal is to increase the likelihood that the reviewer’s score for each proposal in a review has the same meaning.
To discuss and test the IRR concept, ORISE recently conducted a working meeting with experts in a range of disciplines including statistics, data analysis and human factors.
The meeting, funded by ORISE’s internal research and development, incorporated questions like these as part of the discussion: “Should the rating numbers include a middle, neutral number such as 3?” or “Should only even numbers be used which ‘forces’ reviewers to give a more focused rating?”
At the meeting’s conclusion, ORISE walked away with a report from the experts that included recommendations and suggested next steps.
Although there appears to be no definitive answer to implementing IRR, the concept was eagerly received by the experts. ORISE is currently studying the outcomes report to determine the next best course of action.