National Institute of
General Medical Sciences

Findings

1. Has the INBRE program been effective in meeting its objectives?
   • Yes, long-term support for the INBREs has significantly changed institutional culture, leading to increased biomedical research capacity within the IDeA states and Puerto Rico.
   • Yes, the INBRE program is well coordinated – the networks share practices with each other and publish peer-reviewed analyses of their programs. The entire INBRE program could benefit from more systematic information about the effectiveness of different approaches within each state, to allow more states to benefit from the findings.
2. Are the objectives of the INBRE program appropriate for its intended impacts?
   • Yes, given the diversity of the states and institutions involved, the flexibility of the program is an asset and a factor in its success and should be maintained.

Recommendations

1. For future assessments of trend data, take care to consider the impact of COVID in the 2020-2022 period.
2. Include the explicit goal of enhancing diversity in the biomedical workforce and collect appropriate data on metrics to help examine progress toward that goal.
3. Consider the possible benefits of shifting to virtual meetings for states with geographic limitations, and collect more systematic information on participation in these meetings, particularly by faculty.
4. Consider the possible network-wide impact of research-based courses to engage and prepare students in biomedical research, at primarily undergraduate institutions (PUIs) and community colleges, on the INBRE research network, infrastructure, faculty and student development.

Broader Recommendations

1. Examine qualitative and anecdotal impact testimonials, including progress reports, to understand the impact of INBRE on the research culture at partner institutions.

During the NIGMS Advisory Council Meeting in February 2024, the Acting Director of the Division for Research Capacity Building (DRCB) outlined the following actions NIGMS is taking to address the recommendations presented by the Working Group (NIH Videocast @ 1:11:30).

Recognizing the Pandemic's Impact on the Program

1. Being mindful of the impacts of the pandemic when assessing progress reports, including factors such as:
   • a. Impacts on research productivity, including changes to research focus or regular duties.
   • b. Impacts on faculty recruitment
   • c. Potential delay in publications
   • d. Potential effects from faculty leaving research or academia.

Enhancing Biomedical Workforce Diversity

1. Continuing to encourage close partnerships with HBCUs and TCUs.
2. Emphasizing inclusion when conducting outreach for undergraduate awareness of research opportunities.
3. Developing strategies to advance the scientific and technical merit of the INBRE network through expanded inclusivity (illustrated by the requirement for a Plan for Enhancing Diverse Perspectives (PEDP) in the current Notice of Funding Opportunity (NOFO)).
4. Exploring methods to capture changes in student demographics over time.

Expanding Communication Practices

1. Considering both the positive and negative aspects of virtual meetings implemented during the pandemic.
2. Providing new suggested tables for progress reports to capture quantitative data on virtual activities as well as encouraging PIs to describe their experiences in progress reports.
3. Encouraging discussion of virtual, hybrid, and in-person meetings and courses as part of INBRE PIs’ regular sharing of best practices.

Developing Research-Based Courses

1. Encouraging innovation and integration of research into curricula in the current NOFO (PAR-23-100).
2. Including updated guidance for progress reports with suggested data tables to better track student numbers and course-based research experiences and educational activities offered.
3. Releasing a funding opportunity to expand the repository of Cloud-based learning modules in the NIGMS Sandbox (NOT-GM-24-006) with an emphasis on modules’ suitability for integration into undergraduate or graduate curricula in addition to self-learning by researchers.

Examining Qualitative Impacts of INBRE

1. Encouraging INBRE PIs to share examples of INBRE impact on research culture through testimonials at INBRE meetings, emailing NIGMS Program Officers, and including examples in the progress reports.
2. Disseminating impact testimonials and success stories through NIGMS media outlets.

Findings from this evaluation were presented during the May 2023 NIGMS Council Meeting (NIH Videocast @ 1:39:23).

Findings

1. Is the B2B program meeting its stated objectives?
   • Yes, both student transfer rates to 4-year institutions and Bachelor's degree completion rates exceeded historical targets and national benchmarks.
2. Are the goals of the B2B program the correct ones?
   • There is room for current goals to be expanded to be more explicit (e.g., specifying acquisition of a degree in a STEM field) and further advance the long-term goal of enhancing diversity in the biomedical workforce.
3. B2B has transitioned from the Research Education (R25) to the Institutional Training Grant (T34) program structure, but what lessons learned from the R25 program might be applied to the T34 program?
   • Given the variability in outcomes, the analysis could not pinpoint specific factors or optimal sizes for program success; however, such variability can be examined to identify and communicate effective practices to inform additional/future programmatic success.
   • Some T34 changes, such as student support and stronger data reporting requirements/tools, are both helpful and positive. Other changes may require additional analysis at a future point given the recency of conversion to the T34.

Recommendations for the B2B Program

1. Identify successful practices from B2B R25 data through case studies or qualitative analysis and from new B2B T34 data collected through new reporting requirements. Track how these practices are affected by the transition from the R25 to the T34 and use this information to inform the administration and implementation of the program.
2. Communicate any identified successful practices and allowable budgetary approaches to applicants and current awardees; this information could be beneficial given the budget structure of the T34 mechanism.
3. Include the explicit goal of enhancing diversity in the biomedical workforce and collect data on metrics that will help examine progress toward that goal, such as STEM degrees earned and students' activities after graduation (further degrees or entry into the biomedical workforce).

Broader Recommendations

1. Assess the feasibility of conducting a broader examination of the biomedical research workforce and need for community college-prepared employees (i.e., a Skilled Technical Workforce) to generally inform NIGMS diversity efforts.

The items listed in this section represent impressions and suggestions derived from the NIGMS Advisory Council Working Group on NARCH and the NIH Tribal Advisory Council as well as feedback derived from Tribal Leaders through formal Tribal Consultation [PDF].

Questions the Working Group Considered

1. Is the NARCH program meeting its stated objectives?
   • The working group agreed the current NARCH objectives are being met.
2. Are the current program objectives the correct objectives?
   • While the current goals and objectives are appropriate overall, some goals and objectives could benefit from modification.
3. Can certain areas of the NARCH program be optimized, improved, or strengthened?
   • The working group agreed that certain areas could be modified, optimized, or strengthened.

Impressions

1. The NARCH program has changed over time.
   • Administrative Core became a required component and Capacity Building Projects were added as of NARCH VIII.
   • All of NARCH funding and management was moved to NIGMS.
2. The NARCH program supports health research projects prioritized by AI/AN communities and works toward reducing health disparities.
   • NARCH projects must be supported and prioritized by the AI/AN organization.
   • Research projects funded by NARCH have examined the most significant health disparities experienced by AI/AN communities (e.g., substance abuse, diabetes, mental health, cancer, etc.).
3. The NARCH program has promoted a cadre of scientists and health research professionals interested in AI/AN health research and enhanced partnerships between AI/AN communities and research institutions.
   • NARCH programs have funded 53 student and faculty enhancement subprojects from the high school to faculty level.
   • NARCH grantees have partnered with 22 research-intensive institutions (92% of NARCH awards).
4. The NARCH program has likely reduced AI/AN communities’ distrust of research. (Note: difficult to empirically assess; a modified RPPR using a specific data collection item could help)
   • Almost 60% of the research projects have implemented Community Based Participatory Research (CBPR) methods, which intimately involves the AI/AN community and has been shown to enhance relationships and build trust.
     

Summary of feedback from NIH Tribal Advisory Committee

1. Shift the focus away from academic institutions doing the research to the Tribes doing the research.
2. Increase opportunities for/emphasis on capacity building, including helping AI/AN students develop into researchers.
3. Provide more support for Tribes and Tribal Organizations in applying for and managing grants.
4. Reduce the complexity of NARCH grant applications and awards.
   • Although funding rates of first time NARCH proposals is high (75% or 21/28), only 2 of a total of 18 unfunded applicants have applied for funding a second time.
     
5. Better align peer review with the goals of the NARCH program.
   • Ensure that reviewers follow guidelines in “American Indian and Alaska Native Research in the Health Sciences: Critical Considerations for the Review of Research Applications”.
6. Ensure that reviewers follow guidelines in “American Indian and Alaska Native Research in the Health Sciences: Critical Considerations for the Review of Research Applications”.
7. Put reviewers on NARCH study sections from AI/AN communities.
8. Ensure we are looking beyond traditional academic indicators (papers, citations, etc.) in evaluating success and progress.
9. Shift away from a reliance on traditional western academic values and norms.

Summary of feedback from formal Tribal Consultation

1. Ensure that both research projects and research infrastructure/capacity building are primarily conducted by Tribal Nations and communities rather than by academic partner institutions.
2. Remove barriers to application by reducing complexity of submission requirements, particularly for Tribal Nations or Organizations that do not already hold NIH grants.
3. Ensure that reviewers understand the importance of culture and language as research elements for examining health disparities, resilience, and dealing with trauma in Indian Country.
   • Ensure that NIH utilizes the American Indian and Alaska Native Research in the Health Sciences: Critical Considerations for the Review of Research Applications [PDF] as part of the application review process.
4. Support AI/AN trainees earlier in the training pathway (starting in high school) and provide greater support to AI/AN institutions of higher learning (i.e., TCUs).
   
5. Provide stable Administrative Core funding and increase overall program funding to support new grantees alongside existing ones.
6. Make funding announcements continuous rather than on the current skipped year cycle.
7. Create opportunities for new applicants to learn from existing grantees through mentorship or collaboration on proposals.
8. Conduct regular Tribal Consultations, including consultations related to programmatic changes and future program evaluations.

Commensurate with information contained in the Tribal Consultation Report [PDF], NIGMS plans to consider the following actions which were presented during the September 2021 NIGMS Advisory Council Meeting by the NIGMS Institute Director. In June 2023, NIGMS released an updated NARCH NOFO: PAR-23-166 incorporating some of the findings and recommendations from the evaluation and tribal consultation, with other specific recommendations described below along with programs launched in response.

Tribal Technical Assistance and Resource Center(s)

1. Provide support for Tribes in applying for and administering NIH grants.
2. Training and consulting to help Tribes establish their own Sponsored Programs Offices.
3. Training and consulting to help Tribes develop capacity for financial administration of NIH (or HHS) grants.
4. Similar to concept for SuRE Resource Center (PAR-21-227).

Grants to Tribes to help them build or enhance their own Sponsored Programs Offices

1. Develop tribal capacity to apply for and administer grants.
2. Could support, e.g., staff training, systems purchases/upgrades, consulting services.
3. Similar to NIH Sponsored Programs Administration Development (SPAD) program: RFA-RM-19-004.
4. NIGMS released the NOFO Biomedical Research Environment & Sponsored Programs Administration Development (BRE-SPAD) Program (UC2- Clinical Trial Not Allowed): PAR-24-268, which will support Sponsored Program Offices at eligibile institutions, including TCUs.

Grants to Support the Establishment or Enhancement of Tribal Institutional Review Boards (IRBs)

1. Could support staff time, consultation, training, systems procurement, or development, etc.
2. NIGMS released the NOFO Tribal Institutional Review Board Establishment and Enhancement (TIRBEE) (R24 - Clinical Trial Not Allowed): PAR-24-260 to address this recommendation

Planning Grants for Tribes without NARCH Awards

1. Provide time and resources to allow Tribes to develop their own applications.
   • Time and support for Tribes to define research questions and plan how to answer them.
   • Capacity-building and training needs could be assessed.
   • Time to build administrative capacity for application support and grants management.
   • Time to identify any needed consultants, contractors, or collaborators.
2. Could support meetings, travel, staff time, consultation needed for writing NARCH applications.
3. Goal would be to increase number of competitive NARCH applications to support Tribally-driven and conducted research.
4. NIGMS released the NOFO Native American Research Centers for Health (NARCH) Planning Grants (R34 - Clinical Trial Not Allowed): PAR-24-041 to address this recommendation.

Training Grants for Tribes/Tribal Organizations

1. Provide support for students selected by Tribes/Tribal Organizations to obtain degrees in biomedically-related fields.
   • Undergraduate training grants (T34) –tuition remission and stipends for studies towards a BA/BS degree (including at 2-year colleges); research experiences; mentoring.
   • Graduate training grants (T32) –tuition remission and stipends for studies towards a PhD; research experiences; mentoring; career development.
2. Goal would be to develop future researchers who focus on AI/AN health.
3. Each grant would need to support students from multiple Tribes.
4. NIGMS released the NOFO Tribal Undergraduate to Graduate Research Training and Leadership Experiences (TURTLE) Program (UE5/T34): PAR-24-236 to address this recommendation.

Findings

1. Is the SCORE program meeting its objectives?
   • Increases in research competitiveness, number of underrepresented investigators, number of qualifying laboratories, but few investigators transition to non-SCORE funding.
2. Are the current program objectives the right objectives?
   • While some current objectives are appropriate, others require substantive change.
3. What challenges or difficulties has the SCORE program faced?
   • Institutional support for PI and institutional readiness and commitment to growth are uneven.
   • SCORE is too concentrated in a few college/ university systems.

Recommendations

1. Modify program objectives to catalyze institutional support for SCORE funded investigators
   • Require plan for institutional development of research capacity
   • Require plan for SCORE PI development
2. Modify program objective to emphasized increase number of students engaged in research
   • Clarify student involvement in SCORE supported research in the NOFO
     
   • Prepare prospective evaluation of student involvement in SCORE
3. Revise SCORE PI expected outcomes
   • Expand qualifying funding beyond R01
   • Define competitiveness to align with SCORE goals
4. Revise or consolidate funding mechanism
   • Eliminate the SC1/Maintain the SC2/Strengthen the SC3
5. Develop prospective evaluation plan that aligns data collection with new objectives

Findings from this evaluation were presented during the January 2020 NIGMS Council Meeting

During NIGMS Advisory Council Meeting in May 2020, the Research Capacity Building Division Director, outlined steps to address the recommendations.

Recommendation 1 Response: Focus on a new objective of developing institutional research capacity

• Require an institutional plan for developing research capacity and research culture in the application
• Require a letter of support from an institutional leader specifying release time and resources provided etc.

Recommendation 2 Response: Support student participation in quality research

• Require plan for student participation in applications
• Require a report on student participation and outcomes in renewal applications and annual reports

Recommendation 3 Response: Revise expected outcomes for grantees

• Expand measures for success to include non-R01 grants such as R15, R21, NSF grants etc.
• Rename program to shift the focus from enhancing research competitiveness to developing research excellence in PIs and grantee institutions
• Potential SCORE program rename: Develop Academic Research Excellence (DARE) Award

Recommendation 4 Response: Redesigning funding mechanisms for DARE

• Eliminate SC1
• Replace SC2 with DARE- New Investigator ($125K/year, 4 years, not renewable)
• Replace SC3 with DARE ($100k/year, 4 years, renewable) makes more attractive to established investigators
• Develop DARE Research Center: to help build capacity at eligible institutions to broaden participation in DARE

Recommendation 5 Response: Develop data collection capacity for prospective evolution

• Link DARE student participation and outcome reports with NIH and /or NIGMS data collection systems
• Collect and track grantee institutions' research capacity building data such as their research funding

During NIGMS Advisory Council meeting in May 2021, Ming Lei, the Research Capacity Building Division Director, introduced the Support for Research Excellence (SuRE) program which replaces the SCORE program:

The SuRE program supports three notices of funding opportunities:

• Funding for investigators with no other NIH research funding (R16) See PAR-21-169
• Funding for investigators with no prior research funding (R16) See PAR-21-173
• Funding to build infrastructure and provide training (U24) See PAR-21-227

New features of SuRE program include:

• Replacing program focus from “developing R01 grantees” in SCORE to “Supporting research excellence” in SuRE.
• Require participation of students, especially those from underrepresented groups, in funded research
• Require applicants to provide their institution’s strategic plan for developing research capacity and culture
• Use publicly available student data to set institutional eligibility that is easy-to-follow
• A resource center to serve SuRE-eligible institutions and applicants. The resource center will:
• Build/strengthen Offices of Sponsored Programs (OSPs) through a seed grant program;
• Provide training by Center staff and regional outreach coordinators;
• Track institutions’ progress in research capacity building and its own service efficacy.

Findings

The Working Group on Sepsis used a Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis in the areas of preclinical model systems [PDF], clinical heterogeneity and endotyping [PDF], and the translation of fundamental research into new diagnostic tools and therapies [PDF] (a full list is available in the report) and come up with the following recommendations:

Recommendations

1. NIGMS should significantly expand its support of clinical research related to sepsis
   • Support the collection, biobanking, and distribution of biospecimens and data sets to clinical and basic science investigators.
   • Support the development of clinical informatic and biomarker tools to better identify and assess the complexity and heterogeneity of the disease including the host response in order to endotype patients and study participants.
   • Emphasize the importance of materials from ill patients in mechanistic research programs, including clinical data sets and clinical materials. Modify program objective to emphasized increase number of students engaged in research.
2. NIGMS should broaden its collaborations with other institutes to support clinical trials in sepsis
   • Support the development of tools that directly enhance clinical research and clinical care of septic patients.
   • Support ‘proof of principle’ trials wherein a fundamental disease mechanism is evaluated in a small-scale human trial structure. Endpoints may include surrogate biomarker response and not necessarily clinical outcomes. Priority of such work should be given to mechanisms deemed to be of high translational promise.
3. NIGMS should independently sponsor definitive clinical trials only in extraordinary circumstances
   • In most instances, NIGMS should not independently support definitive clinical trials (e.g., Phase II clinical studies or later) or support trial-planning grants for which follow on funding has not been definitively established elsewhere.
   • Instances where sponsorship could be warranted include studies for which there is broad agreement of otherwise unmet societal need and instances providing a unique opportunity to gather data that will likely significantly improve understanding of the underlying mechanism.
4. NIGMS should work with the Center for Scientific Review to ensure the availability of reviewer expertise
   • Expertise is needed in a broad array of methodologies, including mechanistic, discovery, traditional clinical, and clinical informatic approaches.
5. NIGMS leadership should engage the Office of the Director to conduct a higher-level review of how the NIH can best coordinate efforts across institutes and more fully define NIGMS’s role in that effort in order to translate tax dollars into cures.

Regarding preclinical sepsis research, NIGMS should:

• Support the standardization of animal models and the development of models that more closely mimic (1) non-immunological aspects of sepsis and (2) important comorbidities in human sepsis.
• Encourage use of human clinical material as a necessary means of confirming observations in nonhuman models.
• Support the use of discovery science, computational, as well as cell-culture and organoid-type methods in preclinical sepsis research.
• Evaluate current and new models in part by their ability to provide readouts relevant to the specific translational development of new diagnostic methods and therapies.

Regarding clinical sepsis research, NIGMS should:

• Support clinical research approaches that improve human disease identification, staging, and endotyping.
• Work with current NIGMS-sponsored researchers to identify and catalog extant clinical data sets and biospecimens for use in new research.
• Work with other stakeholder institutes (e.g., NHLBI, NIAID, NICHD) to sponsor a workshop to identify best practices and research priorities related to the use of electronic health records and other technologies to identify and stage septic patients for the purposes of supporting care and clinical trials.

Regarding efforts to translate discoveries into new diagnostics and therapies, NIGMS should:

• Proactively partner with other stakeholder institutes, early-stage pharma and biotech companies, and early-stage investors to identify unmet needs relevant to the translation of NIGMS funded discoveries into the nation’s pharmaceutical and biotech development pipeline.
• Support the use of preclinical research endpoints that specifically align with drug discovery, distinct from mechanism discovery, and that address feasibility needs of early stage biotech companies and investors.
• Support research that improves the rapid identification and characterization of septic patients to improve the efficiency of early trials of new sepsis treatments.

Findings from this evaluation were presented during the May 2019 NAGMS Council Meeting.

During September 2019 NAGMS Council meeting, the Pharmacology, Physiology, and Biological Chemistry (PPBC) Division Director outlined steps to address the Working Group’s recommendations.

Recommendation Response:

• NIGMS has released a Notice of Information to inform the scientific community of several areas within sepsis research that are of special interest to the Institute. See NOT-GM-19-054.
• NIGMS released a Request for Information (RFI) to gather input on strategies for acquiring and providing access to biological materials from humans for use in mechanistic studies of sepsis. See NOT-GM-19-057.
• NIGMS will use the NIH’s SBIR/STTR program to accelerate the development of rapid diagnostics and improved therapeutics for sepsis detection and treatment. See NOT-20-028.
• Along with several ICs, NIGMS released two Notices of Special Interest to promote sepsis research in emergency settings and fundamental and applied inflammation resolution. See NOT-NS-20-005 and NOT-ES-20-046.
• During the September 2020 NAGMS Council meeting, the PPBC Division Director presented a concept clearance for Human Biospecimen Collections for Sepsis Research. The goals of the approved concepts included:
  • Determining the value of existing biospecimen sets for sepsis research
  • Piloting new approaches for collection and analysis of biospecimens
  • Establishing best practices for sharing biospecimens, along with associated clinical and research data
  • Setting standards for future work in the sepsis field

1. NIGMS continue its support of the postdoctoral NRSA CPRT T32 programs and re-assess the emphasis of the programs in regard to changes in clinical research and practice, as well as changes in financial obstacles faced by clinician scientists that have occurred since the inception of the programs (e.g., income differential compared to clinical pursuits).
2. NIGMS should define clear goals for its clinically focused research training programs (CPRT T32, MSTP, K08/K23) and develop metrics than can be routinely collected by each CPRT program and collectively by NIGMS for all CPRT programs. Such data could provide a basis for measuring success and continuing enhancement for each program and for CPRT programs overall. The committee felt that, if required, support for such data collection would be beneficial for all institutional and individual training grant mechanisms.
3. NIGMS should reexamine the focus research areas of Medicine and Clinical Research of these programs to ensure they are best serving the clinical research workforce needs such as: exploring Medical Genetics, identifying NHGRI areas of commonality with the possibility of co-funding or the transfer of medical genetics training to NHGRI. Broaden Trauma, Burn, and Injury to include emergency and critical care medicine. Examine Clinical Pharmacology to see what it encompasses today (for example, pharmacogenetics, systems biology, mathematical models, big data), current workforce needs, and how this fits within the NIH. Consider incorporating pain medicine more clearly within the Anesthesiology focus area.
4. Examine the feasibility of expanding loan repayment eligibility for clinical investigators in the 4 NIGMS focus areas, once updated.

Findings from this evaluation were presented during the May 2018 NIGMS Council Meeting

During NIGMS Advisory Council meeting in September 2018, the Training and Workforce Diversity Division Director, outlined steps that to address the recommendations

Recommendation 1 Response: NIGMS will continue to support and monitor the programs by conducting portfolio analyses, attending relevant clinical research meetings, interacting with relevant professional societies. NIGMS is aware of the financial obstacles faced by clinician scientists and will continue to monitor income differential of clinical research careers compared to clinical pursuits and explore solutions to retain clinician-scientists.

Recommendation 2 Response : NIGMS has posted goals of the programs on the NIGMS postdoctoral T32 website. To enhance data collection, NIGMS formed collaborations with:

• The Census Bureau to analyze student data longitudinally (collaboration through 2019).
• The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

Recommendation 3 Response: NIGMS conducted analyses and outreach in each of the areas to ensure that the training was in alignment with current trends in clinical research. For example:

• Medical Genetics – NIGMS and NHGRI communicate regarding funding of clinical postdoctoral trainees to ensure there is not overlap.
• Trauma, Burn, and Injury – NIGMS changed the name to “Injury and Critical Illness” to reflect the complexity of the portfolio.
• Clinical Pharmacology – NIGMS is in communication with the research community and professional societies to ensure that the training is in alignment with the research workforce needs.
• Anesthesiology – the NIGMS website was updated to reflect the breadth of the portfolio, including pain research.

Recommendation 4 Response: The loan repayment eligibility criteria are controlled by Congress and outside of NIGMS’ authority. NIGMS reached out to the Director of the Loan Repayment Office to pass on the concerns of the committee.

Conclusion:

The absence of high-quality trainee data made it essentially impossible to critically evaluate the Bridges to Doctorate program.

Recommendations:

1. To benchmark the NIGMS B2D program against the NIGMS Post Baccalaureate Research Education Program (PREP) and NSF Louis Stokes Alliance for Minority Participation (LSAMP) Bridge to the Doctorate programs to identify the most successful and cost-effective mechanism(s) to increase underrepresented Ph.D. graduates, with the goal of reducing, remodeling, or phasing out the NIGMS B2D program.
2. To increase resource flow toward (centralized) data collection mechanisms that improve the longitudinal tracking of trainees and trainee outcomes.

Additional Recommendations:

1. To increase resource flow toward (centralized) data collection mechanisms that improve the longitudinal tracking of trainees and trainee outcomes.
2. Conceive strategies to enhance research activity through partnership.

The evaluation was presented at the May 2018 NAGMS Advisory Council by a member of the external evaluating committee.

A presentation of the NIGMS response to the B2D Evaluation was presented to the September 2018 NAGMS Advisory Council by the NIGMS Training, Workforce Development and Diversity Division Director. The main recommendation responses are as follows:

Recommendation 1 Response: The NIGMS Training, Workforce Development and Diversity Division Director, the Undergraduate and Postdoctoral Branch Chief, and Program Staff met with National Science Foundation (NSF) officials overseeing the Louis Stokes Alliances for Minority Participation (LSAMP) Program and discussed the potential for the benchmark comparison, but uniformly arrived at the consensus that the NIGMS and NSF programs are distinct and therefore not comparable. The recent program evaluations based on progress reports show that PREP program have similar outcomes for transitioning into and completing Ph.D. programs.

Recommendation 2 Response: NIGMS has since formed collaborations with the Census Bureau to analyze student data longitudinally and with the NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

Additional Recommendation Response: Based on the outcome evaluation and analysis of successful long-standing programs, the notice of funding opportunity has been subsequently revised and reissued (PAR-21-198). Relevant outcome evaluation features include:

• Stating the proximal and long-term goals of the program. The proximal goal states that the objective is to develop a diverse pool of scientists earning a Ph.D., who have the skills to successfully transition into careers in the biomedical research workforce.
• Emphasizing strong partnerships to enhance bridging, reduce the time to degree, and build research capacity at the master’s degree granting institution (e.g., with effective skill-building activities, course credit articulation agreements, and shared research resources).
• Focusing on training and mentoring interventions before and after the bridge to ensure a smooth transition and increase persistence.
• Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
• Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.

The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The Bridges to the Doctorate Research Training program supports individuals as they make the transition from master’s degree granting institutions to research-intensive to Ph.D. degree granting institutions.

1. NCSBs greatly contributed to the origins of systems biology and continue to contribute uniquely to its development. (Systems biology has grown in parallel with the centers initiative, it extends beyond the centers themselves, and it is increasing interdisciplinary.)
2. Some reductions in overall funding to the NCSB program may be achieved by narrowing the focus and limiting the number of awards during each funding cycle.
3. Systems biology remains potentially transformative field.
4. NCSB program excels at integrating the diverse elements of research, training, and outreach, but should evolve in a structure appropriate to current opportunities and challenges and the pool of potential outstanding applications.
5. Center-like funding is needed to develop the team of interdisciplinary team-orientated scientists to design, implement, and analyze screens, using computation, statistical analyses, bioinformatics, for discovery based and tool-oriented projects.
6. NCSBs are not as cost-efficient as P01s and R01s in the production or publication of some kinds of data, and therefore may not be the best mechanism to support every kind of systems biology research.
7. The prospect for broadening opportunities for applications of systems biology emphasizes the catalytic role NCSBs will have in influencing the rate at which the discipline penetrates the practice of science and medicine.
8. NCSBs represent a 15-year sustained effort in training of junior investigators skilled in systems biology, and the panel recommends that this training mission be sustained and possibly enhanced by encouraging additional innovative combinations of training and outreach in the context of necessary workforce development.

The findings of this analysis were presented during the January 2016 National Advisory General Medical Sciences (NAGMS) Council Meeting).

• The NCSB program was allowed to sunset, see NOT-GM-14-120.
• Committee recommendations for NIGMS to encourage team science were incorporated into the design of the Collaborative Program Grant for Multidisciplinary Teams (RM1).
• Funding announcement, PAR-17-340, for Multidisciplinary Teams (RM1) encourages applicants who propose to conduct research to address complex and challenging biomedical problems, do so through deeply integrated, multidisciplinary research teams.
• The NCSB analysis created an opportunity to further develop standard procedure for NIGMS evaluations.

1. Recipients of an F31 Fellowship (F31 Fellows) were more likely to complete their PhD program than applicants who did not receive the fellowship (F31 Applicants).
2. F31 Fellowships were awarded to a broader range of institutions than T32 Traineeships.
3. Among individuals who completed their PhD programs, F31 Fellows had program lengths similar to those of F31 Applicants and slightly longer (~5 months) than those of T32 Trainees.
4. Among individuals who completed their PhD programs, F31 Fellows were more likely to be an F32 postdoc than F31 Applicants or T32 Trainees in the three most recent cohorts.
5. Among individuals who completed their PhD programs, F31 Fellows were as likely to be an F32 or T32 postdoc as F31 Applicants or T32 Trainees.
6. Among individuals who completed their PhD programs, F31 Fellows were as likely to apply for/receive major research grants as F31 Applicants or T32 Trainees.
7. Among individuals who completed their PhD programs, F31 Fellows were as likely as F31 Applicants or T32 Trainees to be employed in research positions in 2014.
8. F31 Fellows earned more money in 2014 than F31 Applicants or T32 Trainees.

• Findings from this evaluation were presented during the January 2019 NIGMS Council Meeting
• NIGMS will maintain support of the F31 Diversity Fellowship program, as it fills the need to support PhD students from underrepresented groups through degree completion.
• NIGMS will continue to explore mechanisms that provide financial stability to PhD graduate students.

1. If additional analysis of MIDAS outreach is desired in the future, NIGMS should mandate that awardees give additional care to the outreach activities sections of their annual reports.
2. Many investigators stated that they had proposed outreach programs, especially short courses and conferences, which were not implemented due to limited funds, and that cuts in funding would affect outreach activities first.
3. To build on MIDAS’ record of policy engagement and improve its impact on public policy decisions, policy makers and researchers suggested that MIDAS support further engagement including, creative initiatives that allow researchers to work more closely within government offices.
4. Data access policies and data sharing practices presented significant barriers for some MIDAS investigators who wanted to provide policy relevant modeling information to federal officials during disease outbreaks.
5. It is important to develop clear standards and shared expectations to foster successful collaborations among an array of modelers and policy makers. Consensus data sharing agreements, standards, and policies are necessary to expedite time-sensitive collaborations and enhance the utility of modeling outputs for policy decisions.
6. The complexity and diversity of MIDAS infrastructure presents challenges in meeting the programmatic goals of sharing and dissemination of MIDAS supported models and tools.
7. NIGMS may need to develop sharing metrics that are appropriate for the specific type of resource.
8. NIGMS may need to reconsider its goals for sharing and dissemination of MIDAS resources given the current issues and future shifts in information technology, data security and public policy. MIDAS investigators suggested the following to enhance relationships with policy makers:
   • Deployment of modeling research for epidemic support should be an “end to-end data to decision support” paradigm.
   • Modelers should become an active part of public health teams to provide data-driven input regarding policy decisions at all times.
   • MIDAS could have a more formal internal system to identify and match policy makers with the appropriate researchers given the nature of expertise and type of model desired.

• Following this report, NIGMS eliminated all U grants except for a single U24 grant that funds a Coordination Center, which acts as a focal point for collaboration and training, testing and dissemination of MIDAS research projects. See RFA-18-003.
• This funding opportunity provides guidance on the infrastructure and resources for evaluating and disseminating software, collecting and managing data relevant to MIDAS systems.
• MIDAS applicants must now provide support for dissemination and outreach and providing logistical support for MIDAS activities such as scientific and Educational coordination and outreach.

1. Strengthen the language in future NOFOs, particularly pilot programs, to further emphasize the importance of metrics and evaluation.
2. Researchers who develop resources as part of a grant should be encouraged to find ways to track users of the resource and should encourage the citation of the resource by its users/user community in publications, track the number of publications/citations in which the resource is referenced, and find additional methods of measuring productivity and impact.
3. Any concerns noted by reviewers regarding lack of adequate evaluation plans or insufficient metrics should be addressed with PIs upon award to ensure that progress reports contain the required data and that the deficiencies have been addressed and documented.
4. Program officers (POs) should review progress reports to ensure that the required information is included or that there is an explanation why it is not included and should address any concerns with PIs.
5. Consider developing a reporting template or standard reporting format for capturing utilization rates in progress reports and monitor compliance through regular review and follow-up by POs.
6. Encourage PIs to conduct and share results of internal evaluations.

• To continue support of vital resources that are no longer eligible for support under their original initiatives, a separate notice of funding opportunity was created for NIGMS-sponsored "legacy" resources. See PAR-19-208.
• NIGMS program staff continues to work with Legacy Resource Principal Investigators to include metrics for annual Research Performance Progress Report (RPPR) submissions.

1. P01s are awarded 4x as much in Total Costs compared to R01s and Multi-PI R01s, per project, this disappears.
2. Multi-PI R01s outperform R01s and P01s in terms of the cost effectiveness of publication production.
3. No statistically significant difference exists in the cost effectiveness of citations for these mechanisms.
4. No statistically significant difference is present between mechanisms in terms of Relative Citation Ratio.
5. Although both mechanisms publish in similar fields , Multi-PI R01s have more articles in the Top 1% and top 10% of cited papers than P01s.
6. Little difference in the size of the collaborative network between P01 and Multiple R01 cohorts.

The findings of this analysis were presented during the January 2016 NAGMS Council meeting and published in an April 2016 Feedback Loop post.

• This analysis was a primer to identify how NIGMS incentivizes team science.
• The findings of this analysis were presented during the January 2016 NAGMS Council meeting and published in an April 2016 Feedback Loop post
• Post-assessment, NOT-GM-17-016 encouraged those interested in collaborative research that requires multidisciplinary teams to apply to PAR-17-340 "Collaborative Program Grant for Multidisciplinary Teams" (RM1).
• This analysis was an opportunity to clarify and communicate OPAE's role in NIGMS evaluation operations leading to the development of OPAE Standard Operating Procedures.
• This was the first analysis to contribute to the development of cost-effective measures for evaluations; Collaborative opportunity with the NIH Library to create a standardized Extramural Investigator network analysis from Profile Person Identification (PPID)s and PubMed IDs.

1. The BioMath program had a higher percentage of women PIs (however the difference was not statistically significant at the p <= 0.05 level).
2. BioMath PIs were more racially/ethnically diverse (non-White, non-Asian).
3. BioMath PIs were much more likely to come from the Math and Engineering departments.
4. BioMath PIs were far less likely to have prior NIH awards. This is a continued emphasis of the program: since the last evaluation of the program in 2016, 25 out of the 28 NIGMS awards in 2017-2019 were to NIs (13 of whom were also ESIs).
5. BioMath projects were numerically more productive than comparison group projects in terms of publications and citations per project (however these differences were not statistically significant)
6. BioMath PIs were considerably less likely to attempt to renew their R01s. If renewal was attempted, success rates for getting awarded were numerically lower (however, the difference was not statistically significant).
7. The BioMath program has been fulfilling its goal of recruiting PIs from quantitative fields while helping to increase the diversity of the PIs.
8. The BioMath program has been fulfilling its goal of recruiting PIs who would be less likely to receive NIH funding otherwise. Additionally, it should be noted that the BioMath program has continued to place an emphasis on recruiting new investigators since the last reissue of the program in 2016. Of the 28 NIGMS funded projects from 2017-2019, 25 were awarded to NIs (of which 13 were also ESIs).
9. Even though the BioMath projects have been performing as well as the comparison group, BioMath PIs have not attempted to renew their awards.
10. The BioMath program has been fulfilling its goal of receiving and funding good quality projects by productive PIs.

• During the May 2020 National Advisory General Medical Sciences Council meeting, the NIGMS Institute Director presented a concept clearance to reissue the BioMath Program.
• The Memorandum of Understanding between NIGMS and the National Science Foundation (NSF) is expected to continue through FY2023.
• NSF has reissued the BioMath Program solicitation. See NSF-20-575.
• The reissued solicitation of the BioMath program will formalize the two tracks of proposals: Track 1 for projects of high-risk, high-reward exploratory, or those from new teams of collaborators and Track 2 for projects of large scope from well-established teams.
• Program staff reached out to the PIs of the recently ended and current BioMath projects to emphasize that their projects are eligible for renewal and encouraged them to apply for T2 awards. Efforts to reach out to the BioMath PIs on a regular basis to encourage them to renew their projects will continue in the future.

1. IRACDA scholars transition into academic careers at a high rate (73%) and hold academic positions across a broad range of institution types (2-year, 4-year, research intensive, minority-serving) throughout the country.
2. IRACDA scholars contribute the diversity of the workforce with 63% female, 17% Hispanic and 19% African-American participants.
3. IRACDA alumni have success rates in securing research funding similar to a matched comparator set of F32 awardee recipients.

Findings of the report were presented during the May 2016 NIGMS Council, and published in a June 2016 Feedback Loop Post.

• Based on the outcome evaluation and analysis of successful long-standing programs, the notice of funding opportunity has been subsequently revised and reissued (PAR-18-366). Relevant outcome evaluation features include:
  • Stating the proximal and long-term goals of the program. The proximal goal states that the Overarching Objective of the IRACDA program is to develop a diverse pool of well-trained biomedical scientists, who have the technical (e.g., appropriate methods, technologies, and quantitative/computational approaches), operational (e.g., independent knowledge acquisition, rigorous experimental design, and interpretation of data) and professional (e.g. management, leadership, communication, and teamwork) skills necessary to conduct rigorous and reproducible research, and to transition successfully into independent academic careers in the biomedical research workforce.
  • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
  • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
• The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The IRACDA program supports individuals at the early stage of the postdoctoral career.
• There were some challenges during the analysis, including verifying workforce information. To address this, NIGMS has since formed collaborations with:
  • The Census Bureau to analyze student data longitudinally.
  • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

1. Approximately 70% of recent MARC alumni have enrolled in graduate programs or earned subsequent graduate degrees in science or health, with almost one-third in recent years earning a Ph.D. or M.D.-Ph.D.
2. The overall rate for Ph.D. attainment observed for recent MARC alumni is almost twice that seen for undergraduates supported by NIGMS supplements to enhance diversity, and four times higher than that modeled for biology baccalaureates in general.
3. MARC alumni who matriculate in a doctorate program, two-thirds complete the Ph.D., a rate that is higher than the national average in the sciences.

The findings of this analysis were presented during the May 2016 NIGMS Council meeting videocast and published in a May 2016, NIGMS Feedback Loop post.

• Based on the outcome evaluation and analysis of successful long-standing programs, the notice of funding opportunity has been subsequently revised and reissued (PAR-19-219). Relevant outcome evaluation features include:
  • Stating the proximal and long-term goals of the program. The proximal goal states that objective of the MARC program is to develop a diverse pool of undergraduates who complete their baccalaureate degree, and transition into and complete biomedical, research-focused higher degree programs (e.g., Ph.D. or M.D./Ph.D.).
  • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
  • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
• The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The MARC program will support research-oriented individuals at the undergraduate level for 2-3 years.
• There were some challenges during the analysis, including verifying workforce information. To address this, NIGMS has since formed collaborations with:
  • The Census Bureau to analyze student data longitudinally.
  • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

1. Ph.D. Matriculation Rate: Several data sources suggest the current PREP matriculation rate into Ph.D. programs is about 65%.
2. Ph.D. Degree Attainment: For a PREP cohort analyzed, at least 38% earned a Ph.D. (191 earned a Ph.D. out of 501 total PREP participants), at least 16% earned a M.S., 12% earned a M.D., and 7% were still in graduate training at the time of analysis; for 27% no further degree beyond the bachelors could be confirmed.
3. The majority of PREP Ph.D. degrees were earned at research-intensive universities and medical schools.
4. Renewal reports suggest that about 11% of the Ph.D. enrollment at PREP institutions is composed of individuals from underrepresented racial and ethnic groups (range 6-16%; 2011-2012). PREP scholars who have now completed their doctoral and postdoctoral training are predominantly entering the research workforce (43%) and science-related non-research (36%) categories.

The results were published in a September 2015 and discussed in a Feedback Loop post.

• Based on the outcome evaluation and analysis of successful long-standing programs, the notice of funding opportunity has been subsequently revised and reissued (PAR-20-066). Relevant outcome evaluation features include:
  • Stating the proximal and long-term goals of the program. The proximal goal states that the Overarching Objective of PREP remains to develop a diverse pool of well-trained postbaccalaureate participants who will transition into and complete rigorous biomedical, research-focused doctoral degree programs (e.g., Ph.D. or M.D./Ph.D.) in biomedical fields relevant to the NIGMS mission .
  • Delineating the metrics to be gathered by NIGMS. These measures include aggregate number and demographic characteristics of participants, and subsequent educational/career progress.
  • Requiring attachments entitled “Outcomes Data Collection and Storage Plan” and “Dissemination Plan”. These attachments have been added to encourage effective tracking of trainee education and career outcomes and dissemination of any findings or materials developed under the auspices of the program.
• The diversity enhancing programs have been restructured. One positive outcome of this restructuring allows for enhanced capacity for evaluation. For example, all diversity enhancing programs now focus on a specific career stage to facilitate better tracking of trainees through the research training pathway. The PREP program supports individuals at the postbaccalaureate level.
• There were some challenges during the analysis, including verifying workforce information. To address this, NIGMS has since formed collaborations with:
  • The Census Bureau to analyze student data longitudinally.
  • The NIH Office of Extramural Research and eRA Commons to improve the collection of participant outcome data by requiring on-line data entry and establishing a centralized database with participant information.

1. At least 73% of DSP graduate students completed the Ph.D. This completion rate compares quite favorably with national figures of 50-58% Ph.D. completion in all STEM fields by students from underrepresented groups and approximately a 70% Ph.D. overall completion rate in the life sciences.
2. At least 16% of DSP undergraduates earned a Ph.D. and at least 31% earned an M.D. For many undergraduates, the DSP offered a summer research experience at another institution; however, the short duration of a typical undergraduate DSP award may limit its impact on career choice.
3. DSP postdocs were twice as likely as DSP graduate students to have earned their Ph.D. from an institution with substantial enrollment of students from underrepresented groups. This could indicate that the DSP offers postdocs in particular an entry—or re-entry—into research-intensive environments.
4. Careers in research were the most prevalent outcome, with about 65% of all DSP-supported graduate students and postdocs achieving research careers (including academia, industry and government research). DSP participants were employed in career sectors similar to those reported for the U.S.-trained Ph.D. workforce as a whole.