Evaluation Reports

Current Reports

NIGMS’s Institutional Clinical Postdoctoral Research Training T32 Grants

Evaluation of Postdoctoral NRSA T32 Institutional Training Grant Program [PDF, 890KB]

June 27, 2018

Program Type: T32

Pre Evaluation FOA: T32 Training Grants

Evaluation of Postdoctoral NRSA T32 Institutional Training Grant Program, PDF, 890KB  
  1. The panel recommends that NIGMS continue its support of the postdoctoral NRSA CPRT T32 programs and re-assess the emphasis of the programs in regard to changes in clinical research and practice, as well as changes in financial obstacles faced by clinician scientists that have occurred since the inception of the programs (e.g., income differential compared to clinical pursuits).
  2. NIGMS should define clear goals for its clinically focused research training programs (CPRT T32, MSTP, K08/K23) and develop metrics that can be routinely collected by each CPRT program and collectively by NIGMS for all CPRT programs. Such data could provide a basis for measuring success and continuing enhancement for each program and for CPRT programs overall. The committee felt that, if required, support for such data collection would be beneficial for all institutional and individual training grant mechanisms.
  3. NIGMS should reexamine the focus research areas of Medicine and Clinical Research of these programs to ensure they are best serving the clinical research workforce needs such as : exploring Medical Genetics, identifying NHGRI areas of commonality with the possibility of co-funding or the transfer of medical genetics training to NHGRI. Broaden Trauma, Burn, and Injury to include emergency and critical care medicine. Examine Clinical Pharmacology to see what it encompasses today (for example, pharmacogenetics, systems biology, mathematical models, big data), current workforce needs, and how this fits within the NIH. Consider incorporating pain medicine more clearly within the Anesthesiology focus area.
  4. Examine the feasibility of expanding loan repayment eligibility for clinical investigators in the 4 NIGMS focus areas, once updated.

First evaluation of this program. Evaluation was first to use medical certification records (Elsevier) and Census Bureau information for every applicant. During NIGMS Advisory Council meeting in September 2018, Training and Workforce Diversity Division Director Alison Gammie, outlined steps to address the recommendations of this analysis. Including continued support these programs through additional portfolio analyses (monitor income differentials etc.), establish workforce needs, and in collaboration with NIH Office of Extramural Research and eRA Commons, collection of outcome data including on-line data entry and the establishment of a centralized location with trainee information. Findings from this evaluation were presented during the May 2018 NIGMS Council Meeting.

June 15, 2018

Program Type: R25

Pre Evaluation FOA: PAR-17-209

Evaluation of the NIGMS R25 Bridges to the Doctorate Program, PDF, 391.7KB  
  1. To benchmark the NIGMS B2D program against the NIGMS Post Baccalaureate Research Education Program (PREP) and NSF Louis Stokes Alliance for Minority Participation (LSAMP) Bridge to the Doctorate programs to identify the most successful and cost-effective mechanism(s) to increase UR Ph.D. graduates, with the goal of reducing, remodeling, or phasing out the NIGMS B2D program; and,
  2. To increase resource flow toward (centralized) data collection mechanisms that improve the longitudinal tracking of trainees and trainee outcomes.
  3. Analyze successful programs and use elements to inform writing the next FOA.
  4. Conceive strategies to enhance research activity through partnership
  5. Consider NIGMS research capacity building programs.

From the analysis, programs that had longer funding periods were found to have more established information about students. During NIGMS Advisory Council meeting on September 2018, Training and Workforce Diversity (TWD) Division Director Alison Gammie outlined steps to address the recommendations of the PREP and T32 analyses. This included analysis of outcome data within renewal applications and students at various stages of scientific training (NIGMS supported pre and post matriculation into Ph.D./Clinical programs). Similar to the T32 Training grants, TWD will, in collaboration with NIH Office of Extramural Research and eRA Commons, improve collection of trainee outcome data by requiring on-line data entry and establishing a centralized database with trainee information. Findings of this evaluation were presented during the May 2018 NIGMS Council Meeting.

Models of Infectious Disease Agent Study (MIDAS) Program

MIDAS Program Evaluation Report (STPI) [PDF, 2.3MB]

September 2016

Program Type: U01

Pre Evaluation FOA: RFA-GM-14007

MIDAS Program Evaluation Report (STPI), PDF, 2.3MB  
  1. If additional analysis of MIDAS outreach is desired in the future, NIGMS should mandate that awardees give additional care to the outreach activities sections of their annual reports.
  2. Many investigators stated that they had proposed outreach programs, especially short courses and conferences, which were not implemented due to limited funds, and that cuts in funding would affect outreach activities first.
  3. To build on MIDAS’ record of policy engagement and improve its impact on public policy decisions, policy makers and researchers suggested that MIDAS support further engagement including, creative initiatives that allow researchers to work more closely within government offices.
  4. Data access policies and data sharing practices presented significant barriers for some MIDAS investigators who wanted to provide policy relevant modeling information to federal officials during disease outbreaks.
  5. It is important to develop clear standards and shared expectations to foster successful collaborations among an array of modelers and policy makers. Consensus data sharing agreements, standards, and policies are necessary to expedite time-sensitive collaborations and enhance the utility of modeling outputs for policy decisions.
  6. MIDAS investigators suggested the following to enhance relationships with policy makers: deployment of modeling research for epidemic support should be an “end-to-end data to decision support” paradigm. Modelers should become an active part of public health teams to provide data-driven input regarding policy decisions at all times. MIDAS could have a more formal internal system to identify and match policy makers with the appropriate researchers given the nature of expertise and type of model desired.
  7. The complexity and diversity of MIDAS infrastructure presents challenges in meeting the programmatic goals of sharing and dissemination of MIDAS supported models and tools. NIGMS may need to develop sharing metrics that are appropriate for the specific type of resource.
  8. NIGMS may need to reconsider its goals for sharing and dissemination of MIDAS resources given the current issues and future shifts in information technology, data security and public policy.

Following this report, NIGMS initiated a funding announcement for an MIDAS Coordination Center PAR-17-267 and RFA-18-003 which acts as a focal point for collaboration and training, testing and dissemination of MIDAS research projects. First by providing the infrastructure and resources for evaluating and disseminating software, collecting and managing data relevant to MIDAS systems. Providing support for dissemination and outreach and providing logistical support for MIDAS activities such as scientific and Educational coordination and outreach. Findings of this report were presented during the May and September 2017 NIGMS Council Meetings.

Institutional Research and Academic Career Development Award (IRACDA) (K12)

Institutional Research and Academic Career Development Awards (IRACDA) (K12) Outcomes Assessment [PDF, 2.3MB]

June 21, 2016

Program Type: K12

Pre Evaluation FOA: PAR-13-290

Institutional Research and Academic Career Development Awards (IRACDA) (K12) Outcomes Assessment, PDF, 890KB  
  1. IRACDA scholars have research doctorates in a broad array of scientific disciplines and represent a diverse group, with 63% female, 17% Hispanic and 19% African-American participants.
  2. 73% of IRACDA alumni are in academic research and/or teaching positions.

As identified by the IRACDA evaluation, there was a need to develop grantee training to fit where grantees begin their careers (i.e. grant writing for other types of funding mechanisms for those who are in small liberal arts colleges). Awardees are encouraged to highlight this for grantees. IRACDA program was reissued PAR 16-103. Findings of the report were presented during the May 2016 NIGMS Council, and published in a June 2016 Feedback Loop Post.

May 5, 2016

Program Type: R24

Pre Evaluation FOA: PAR-13-324

Outcomes Evaluation NIGMS Legacy Community-Wide Scientific Resources Pilot Program, PDF, 438KB  
  1. Strengthen the language in future FOAs, particularly pilot programs, to further emphasize the importance of metrics and evaluation. Resources developed as part of a grant should be encouraged to find ways to track users and should encourage the appropriate citation of the resource by its users/user community in publications, track the number of such publications/citations in which the resource is actually referenced, and find additional methods of measuring productivity and impact.
  2. Any concerns noted by reviewers regarding lack of adequate evaluation plans or insufficient metrics should be addressed with PIs upon award to ensure that progress reports contain the required data and require documentation that the deficiencies have been addressed.
  3. Program officers (POs) should review progress reports to ensure that the required information is included or that there is an explanation why it is not included, and should address any concerns with PIs.
  4. Consider developing a reporting template or standard reporting format for capturing utilization rates in progress reports and monitor compliance through regular review and follow-up by POs.
  5. Encourage principal investigators (PIs) to conduct and share results of internal evaluations.

The Legacy program had a sunset mechanism. To continue support for these vital resources, a separate funding opportunity announcement was created. See PAR-16-104. The recommendations of this report were published on the NIGMS OPAE website.

May 2016

Program Type: P41

Pre Evaluation FOA: PAR-14-021

Report of the Biomedical Technology Research Resources External Review Committee, PDF, 155KB  
  1. Although the panel felt the BTRR program has been successful in its mission, there was broad agreement that it could nonetheless be improved.
    • Duplication of Resources: There are some technologies that are duplicate. There should be more assessment of the overall portfolio;
    • Missing Technologies: It was felt that there were some missing technologies that would meet existing needs within the research community;
    • Low Rate of Program Turnover: Overall size of the BTRR program is essentially fixed, however, the low rate of turnover (1 per year) prevents new technologies from being introduced;
    • Better Documentation of BTRR Outcomes: More details of how novel technologies resulted in new science was needed. As such, need better documentation of patents, patent citations, and commercialization of technologies;
    • Better Highlighting of BTRRs: Raise the visibility of the BTRR program by promotion and highlighting of their capabilities.
  2. The panel thought the overall five-component structure for a BTRR was reasonable, they felt there should be flexibility and modularity in how these components are implemented; Multiphase Grant Mechanism: The panel envisioned three “phases” of support (initial phase supporting higher risk technology development, middle phase, fund continued efforts, final phase support established technologies) using different grant mechanisms, perhaps something along the lines of the R21/R33 or SBIR/STTR pathways; Integration of Technology Programs: effectively integrated with existing technology-focused R01s within NIGMS. Similarly, R21s could allow for initial support for development of more novel and risky technologies ultimately leading to new BTTR formation.
  3. The panel saw significant merit in the NIGMS BTRR program, as well as the technology-focused R01s in NIGMS. These technological advances have direct impact on biological and biomedical problems relevant to the mission of NIGMS. The panel had several ideas on how BTRRs and technology-based grants could be better integrated into NIGMS:
    • Using Technology R01s to Feed BTRRs: The range of technology based R01s that could be surveyed to determine new techniques that might be developed into BTRRs;
    • Coordination of Reviews: The panel felt there should be better overall coordination in the review of P41s and technology-based R01s. This may include a common review panel with research resources and technology development grant applications.
  4. Additional suggestions included:
    • The need for comparative reviews: There is a clear need to modify the procedures so that most reviewers would see multiple P41 applications, and over time participate in multiple site visits;
    • Joint Review of NIGMS BTTR and NIBIB Biomedical Technology Resource Center [BTRC] programs: This might prove effective in evaluating the merit of the programs and enhancing coordination of the two programs and the range of centers they support;
    • Enhanced Program Flexibility: In the BTRR requirements, flexibility in the terms of the types and number of projects (and components);
    • Recognizing Differences in BTRRs: Acknowledge the fundamental differences in technologies and their ability to be disseminated;
    • Potential Merits of a BTRR Sunset: The panel opinions were split on the potential merits of this provision and felt it not be wise to introduce a sunset mechanism at this current stage;
    • Mechanisms for support adjustments over time: Transition to other support mechanisms is appropriate for mature technologies and critical site-specific resources.

This report provided an Initial use of comparative programs for evaluation analysis. In looking for ways to show return on investment output, NIGMS OPAE expanded network analysis for collaborative mechanisms that demonstrate connections between individuals. First NIGMS analysis to use at patent information within analyses, now a regular suite within NIGMS analyses. Findings of this report were presented during the May 2016 NIGMS Council meeting.

Program Project Grants (P01s)

Project Grant (P01) Analysis

April 20, 2016

Program Type: P01

Pre Evaluation FOA: PAR-13-280

Program Project Grants (P01s)  
  1. P01s are awarded 4x as much in Total Costs compared to R01s and Multi-PI R01s, per project, this disappears.
  2. Multi-PI R01s out perform R01s and P01s in terms of the cost effectiveness of publication production.
  3. No statistically significant difference exists in the cost effectiveness of citations for these mechanisms.
  4. No statistically significant difference is present between mechanisms in terms of Relative Citation Ratio.
  5. Although both mechanisms publish in similar fields, Multi-PI R01s have more articles in the Top 1% and Top 10% of cited papers than P01s.
  6. Little difference in the size of the collaborative network between P01 and Multiple R01 cohorts.

This analysis was a primer to identify how NIGMS incentivizes team science. Post-assessment, NOT-GM-17-016 encouraged those interested in collaborative research that requires multidisciplinary teams to apply to PAR-17-340 "Collaborative Program Grant for Multidisciplinary Teams" (RM1). This analysis was an opportunity to clarify and communicate OPAE's role in NIGMS evaluation operations leading to the development of OPAE Standard Operating Procedures. This was the first analysis to contribute to the development of cost effective measures for evaluations; Collaborative opportunity with the NIH Library to create a standardized Extramural Investigator network analysis from Profile Person Identification (PPID)s and PubMed IDs. The findings of this analysis were presented during the January 2016 NIGMS Council meeting and published in an April 2016 Feedback Loop post.

Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) program

Educational Outcomes from the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) Program [PDF, 625KB]

March 14, 2016

Program Type: T34

Pre Evaluation FOA: PAR-16-113

Educational Outcomes from the Maximizing Access to Research Careers Undergraduate Student Training in Academic Research (MARC U-STAR) Program, PDF, 625KB  
  1. Some 70% of recent MARC U-STAR alumni have enrolled in graduate programs or earned subsequent graduate degrees in science or health, with almost one-third in recent years earning a Ph.D. or M.D.-Ph.D.
  2. The overall rate for Ph.D. attainment (29%) observed for recent MARC U-STAR alumni is almost twice that seen for undergraduates supported by NIGMS supplements to enhance diversity, and four times higher than that modeled for biology baccalaureates in general.
  3. The present data also indicates that, for MARC U-STAR alumni who matriculate in a doctorate program, two-thirds complete the Ph.D., a rate that is higher than the national average in the sciences (DIMAC; Ph.D. Completion).
  4. Future steps to align program activities with research on science education may increase the impact of MARC U-STAR. In particular, efforts at earlier stages to foster development of a large pool of undergraduates who go on to science careers may be effective.
  5. This report provides support that the MARC U-STAR program meets the educational outcome goals outlined in the FOA. Future studies may explore any additional benefits from the MARC USTAR program to students or faculty.

This analysis highlighted that identifying program outcomes beyond scope of program funding is very challenging. This was another study where NIGMS took the opportunity to form collaborations with the Census Bureau and combine databases in order to analyze student outcomes longitudinally. The findings of this analysis were presented during the May 2016 NIGMS Council meeting and published in a May 2016, NIGMS Feedback Loop post.

January 2016

Program Type: P50

Pre Evaluation FOA: PAR-14-319

Report of the National Centers for Systems Biology External Review Committee, PDF, 1MB  
  1. NCSBs greatly contributed to the origins of Systems Biology and continue to uniquely contribute to its development (Systems biology has grown in parallel with the center initiative, Systems Biology extends beyond the centers themselves, Systems Biology is increasing interdisciplinary).
  2. Some reductions in overall funding to the NCSB program may be achieved by narrowing the focus and limiting the number of awards during each funding cycle.
  3. Systems Biology remains a potentially transformative field.
  4. NCSB Program excels at integrating the diverse elements of research, training, and outreach, but should evolve in a structure appropriate to current opportunities and challenges and the pool of potential outstanding applications.
  5. Center-like funding is needed to develop the team of interdisciplinary team-orientated scientists to design, implement, and analyze screens, using computation, statistical analyses, bioinformatics, for discovery bases and tool-oriented projects.
  6. NCSBs are not as cost-efficient as P01s and R01s in the production or publication of some kinds of data and therefore are not the best mechanism to support every kind of Systems Biology research.
  7. The prospect for broadening opportunities for applications of Systems Biology emphasizes the catalytic role NCSBs will have in influencing the rate at which the discipline penetrates the practice of science and medicine.
  8. NCSBs represent a 15-year sustained effort in training of junior investigators skilled in Systems Biology and the panel recommends that this training mission be sustained and possibly enhanced by encouraging additional innovative combinations of training and outreach in the context of necessary workforce development.

The NCSB Program was allowed to sunset (see NOT-GM-14-120). The NCSB analysis created an opportunity to further develop standard procedures for NIGMS evaluations. The findings of this analysis were presented during the January 2016 NIGMS Council Meeting.

August 28, 2015

Program Type: R25

Pre Evaluation FOA: PAR-14-076

Analysis of Scholar Outcomes for the NIGMS Postbaccalaureate Research Education Program, PDF, 205KB  
  1. Ph.D. Matriculation Rate: Several data sources (Competing renewal applications, Cohort 1 scholars currently in graduate school, Cohort 2 in graduate training or completion) suggest the current PREP matriculation rate into Ph.D. programs is about 65%.
  2. Ph.D. Degree Completion and attainment: PREP scholars overall are highly dependent upon appropriate preparation for the Ph.D. by PREP In cohort 1, at least 38% had earned a Ph.D. (191 earned Ph.D./501 total PREP), at least 15.6% had earned an M.S., 12% earned an M.D., and 6.8% were still in graduate training; for 27% no further degree beyond the bachelors could be confirmed.
  3. Where Ph.D. is earned; The vast majority of PREP Ph.D.'s are earned at research-intensive universities and medical schools.
  4. Renewal reports suggest that about 11% of the Ph.D. enrollment at PREP institutions is composed of individuals from underrepresented racial and ethnic groups (range 6-16%; 2011-2012). PREP scholars who have now completed their doctoral and postdoctoral training are predominantly entering the research workforce (43%) and science-related non-research (36%) categories.

During NIGMS Advisory Council meeting in September 2018, Training and Workforce Diversity (TWD) Division Director Alison Gammie outlined steps to address the recommendations of the PREP and T32 analyses. This included analysis of outcome data within renewal applications and students at various stages of scientific training (NIGMS supported pre and post matriculation into Ph.D./Clinical programs). Similar to the T32 Training grants, TWD will, in collaboration with NIH Office of Extramural Research and eRA Commons, improve collection of trainee outcome data by requiring on-line data entry and establishing a centralized database with trainee information. The results from this analysis were published in a September 2015 Feedback Loop post.

NIH-Wide Supplements to Promote Diversity in Health-Related Research (DSP)

NIGMS Analysis of Supplements to Enhance Diversity (1989-2006) [PDF, 205KB]

May 28, 2015

Program Type: Supplement

Pre Evaluation FOA: PAR-12-149

NIGMS Analysis of Supplements to Enhance Diversity (1989-2006), PDF, 205KB  
Archived Reports