How to build a better peer reviewer: An exploratory study

Jan. 8, 2019

Subject matter experts participate in a panel peer review

The exploratory study “Technology and Peer Review Panel Skills” examines how the use of technology in conducting panel reviews for research-funding agencies affects the development and improvement of skills needed by effective reviewers.

Recent research has focused on how the use of technology in conducting peer reviews affects review outcomes. Can technology also make peer reviewers themselves better at what they do?

In 2018, a group of ORISE researchers—Miriam L. E. Steiner Davis, Ph.D., T. Reneau Conner, Ph.D., and Leslie Shapard, Ph.D.—sought to answer this question by conducting an inquiry about “Technology and Peer Review Panel Skills.” This exploratory study examined how the use of technology in conducting panel reviews for research-funding agencies affects the development and improvement of skills needed by effective reviewers.

ORISE researchers considered two specific review formats: in-person and virtual video conference. After interviewing program officers and expert reviewers, they found that most high-quality reviewers possessed common skills, such as subject matter expertise, broad scientific understanding and communication skills.
Researchers also explored the relationship between skill development, review format and technology. Interview responses formed the basis of a quantitative survey. The results identified three additional panel review skills.

Researchers found that 1) modeling competencies improves panelists’ skills more than peer review guidance and training; 2) general academic training improves panelist competencies the least of all experiences measured; and 3) being the chair/running a panel improves skills more than any other experience measured.

In the final report, ORISE researchers recommended more analysis on reviewer skills and informational needs in relation to peer review software and processes, and development of training that focuses on important panel skills.

Download the Report (.PDF, 462 KB)

Media Contacts

Pam Bonee
Director, Communications
Phone: 865.603.5142
pam.bonee@orau.org

Wendy West
Manager, Communications
Phone: 865.207.7953
wendy.west@orau.org

The Oak Ridge Institute for Science and Education (ORISE) is a U.S. Department of Energy (DOE) asset that is dedicated to enabling critical scientific, research, and health initiatives of the department and its laboratory system by providing world class expertise in STEM workforce development, scientific and technical reviews, and the evaluation of radiation exposure and environmental contamination.

ORISE is managed by ORAU, a 501(c)(3) nonprofit corporation and federal contractor, for DOE’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit science.osti.gov.