Evaluations

Scope Note

The Center for Scientific Review (CSR) routinely performs evaluations to assess the quality, effectiveness, and integrity of the NIH peer review process. This includes studies and initiatives related to review operations, study sections, reviewers, and other aspects of scientific review.

Effect of Meeting Format on Review

CSR has conducted multiple evaluations, with large sample sizes, to understand the effect of meeting format on the peer review process. CSR has conducted review with all participants joining remotely, with all participants meeting in-person, and in a hybrid format where some attendees join remotely and others attend in-person.

  • The majority of reviewers prefer in-person meetings.
  • Reviewers rated the overall quality of both in-person and virtual meetings to be high.
  • Meeting format does not affect score distribution.
  • Meeting format does not appear to affect the roster, in terms of career stage or geographic distribution of reviewers.
  • Engagement appears to suffer in virtual meetings with reviewers reporting a shorter attention span during virtual meetings.

Posted Reports

ENQUIRE

Science changes rapidly. Making sure that study sections change with the science is an ongoing challenge. CSR ENQUIRE integrates data and input from multiple stakeholders to determine whether changes in study section focus or scope are needed to facilitate the identification of high impact science, with special consideration of emerging science.

ENQUIRE Overview

Science-based
Clusters of study sections are formed based on scientific topics (instead of CSR managerial units) for review via ENQUIRE

Systematic, data-driven, continuous process
~20% of CSR study sections are evaluated each year, every study section evaluated every 5 years

Stakeholder input and involvement
External scientific community, extramural programs at NIH, CSR staff

Iterative Approach
Continuous refinement of the process based on experience

Critical to success
Matching referral of applications and reviewer expertise to redefined scientific scope of study sections

Multiple Possible Actions Follow

  • Change in scientific guidelines
  • Merge study sections
  • Create new study sections
  • Redistribute areas across study sections
  • Add emerging areas of science
  • Eliminate study sections

CSR Scientific Publications

  • Richard K Nakamura, Lee S Mann, Mark D Lindner, Jeremy Braithwaite, Mei-Ching Chen, Adrian Vancea, Noni Byrnes, Valerie Durrant, Bruce Reed (2021) An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes. eLife 2021;10:e71368. DOI: 10.7554/eLife.71368
  • Erosheva EA, Grant S, Chen M, Linder MD, Nakamura RK, Lee CJ (2020) NIH peer review: Criterion scores completely account for racial disparities in overall impact scores. Science Advances 6(23): eaaz4868. DOI: 10.1126/sciadv.aaz4868
  • Lindner MD, Nakamura RK (2015) Examining the Predictive Validity of NIH Peer Review Scores. PLoS ONE 10(6): e0126938. doi:10.1371/journal.pone.0126938
  • Lindner MD, Vancea A, Chen M-C, Chacko G (2015) NIH Peer Review: Scored Review Criteria and Overall Impact, American Journal of Evaluation online first, 2015
  • Boyack KW, Chen M-C, Chacko G (2014) Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health. PLoS ONE 9(8): e104244. doi:10.1371/journal.pone.0104244
  • Martin MR, Kopstein A, Janice JM (2010) An Analysis of Preliminary and Post-Discussion Priority Scores for Grant Applications Peer Reviewed by the Center for Scientific Review at the NIH, PLoS ONE 5(11): e13526
  • Martin MR, Lindquist T, Kotchen TA (2008) Why are peer review outcomes less favorable for clinical science than basic science applications? Amer J Med 121:637–641
  • TA Kotchen, T. Lindquist, A. Miller Sostek, et al. (2006) Why are peer review outcomes less favorable for clinical science than basic science applications? J Invest Med, 54 (1):13–19
  • TA Kotchen, T. Lindquist, K. Malik, E. Ehrenfeld (2004) NIH peer review of grant applications for clinical research, JAMA, 291 (7):836–843

This page last updated on: November 13, 2026
For technical issues E-mail OER Webmaster