Recent research has focused on how the use of technology in conducting peer reviews affects
review outcomes. Can technology also make peer reviewers themselves better at what they do?
In 2018, a group of ORISE researchers—Miriam L. E. Steiner Davis, Ph.D., T. Reneau Conner,
Ph.D., and Leslie Shapard, Ph.D.—sought to answer this question by conducting an inquiry about
“Technology and Peer Review Panel Skills.” This exploratory study examined how the use of technology
in conducting panel reviews for research-funding agencies affects the development and improvement of
skills needed by effective reviewers.
ORISE researchers considered two specific review formats: in-person and virtual video conference. After
interviewing program officers and expert reviewers, they found that most high-quality reviewers possessed
common skills, such as subject matter expertise, broad scientific understanding and communication skills.
Researchers also explored the relationship between skill development, review format and technology. Interview responses formed the basis of a quantitative survey. The results identified three additional panel review skills.
Researchers found that 1) modeling competencies improves panelists’ skills more than peer review guidance and training; 2) general academic training improves panelist competencies the least of all experiences measured; and 3) being the chair/running a panel improves skills more than any other experience measured.
In the final report, ORISE researchers recommended more analysis on reviewer skills and informational needs in relation to peer review software and processes, and development of training that focuses on important panel skills.