In this episode of Expert Insights for the Research Training Community, Dr. Desirée Salazar, NIGMS’ program director in the Division of Training, Workforce Development, and Diversity, and Dr. Maqsood Wani, chief of the Cell Biology Integrative Review group from the NIH Center for Scientific Review, describe the NIH grant application and review process and explain summary statements and review scores. They also discuss how to get advice from NIH program staff.
The original recording of this episode took place as a webinar on July 23, 2020, with NIGMS host Dr. Dorit Zuk. A Q&A session with webinar attendees followed Dr. Salazar and Dr. Wani’s talk.
Recorded on July 23, 2020
View Transcript Download Recording [MP3]
Announcer:
Welcome to Expert Insights for the Research Training Community—A podcast from the National Institute of General Medical Sciences. Adapted from our webinar series, this is where the biomedical research community can connect with fellow scientists to gain valuable insights.
Dr. Dorit Zuk:
Welcome, everyone, and thanks for joining us. I hope you’re all staying well and safe. First, I’d like to thank everyone who’s doing all they can do to help with this pandemic, particularly those working on the front lines at your local hospital, at the research lab next door working on finding a vaccine, or at the grocery store down the street.
This is one of our second webinar series for students, postdocs, and faculty. It will be an hour long. There will be a 10- to 15-minute presentation by our speakers, followed by a moderated question-and-answer session.
It’s my great pleasure to introduce our speakers today. So first, we’ll have Dr. Desirée Salazar from NIGMS. Desirée is a program director in the Division of Training, Workforce Development, and Diversity. And she administers several programs in TWD, including Innovative Programs to Enhance Research Training, Building Infrastructure Leading to Diversity, and Institutional Research and Academic Career Development Awards, otherwise known as IRACDA.
She also manages research grants in the area of stem cell biology and regeneration in the Division of Genetics and Molecular, Cellular, and Developmental Biology. And before joining NIGMS, Desirée was most recently a scientific program manager at the American Society for Cell Biology. And Desirée will tell us about how to talk to program officers and the application process at NIH.
Dr. Maqsood Wani is from the Center for Scientific Review. He is chief of the Cell Biology Integrative Review Group that reviews research applications on the study of fundamental cell biological processes. Maqsood received his Ph.D. at The Ohio State University and did his postdoctoral work at the University of Cincinnati. And before joining CSR, Maqsood was an assistant professor in the departments of pediatrics and molecular genetics at the University of Cincinnati. And he will tell us all about the review process.
So without further ado, I am going to turn it over to Desirée, and if you can share your screen, that would be great.
Dr. Desirée Salazar:
OK, thank you very much. So today I’m going to talk to you about preparing to write grant applications and how to connect with your program official.
To get started, my position is a program official, also known as program director or program officer or PO for short. And so it’s my role to help provide scientific guidance to people who are applying for grants both before they apply, and I am going to strongly encourage you to contact your program official prior to applying, and then kind of after the review process and throughout the extent of the award. And so we just help provide general guidance.
So early on, before you are submitting an application, a program official can help to determine whether your plan and research is a good fit for a particular funding opportunity announcement—and I’ll talk more about this in a minute—and, critically, whether your research is within the mission of NIGMS or another institute. And then further, whether I would be the right person to manage your grant or whether there is somebody else at NIGMS.
Once you apply and you go through the review process, which Maqsood will talk about in more detail, after it’s gone through the review process and your summary statement is available, then you can again communicate with your program official and we’ll help walk you through the summary statement and let you know what we think about your likelihood of funding and give you an idea of what the timeline is going to be for decisions.
And then once you have a grant awarded, the program officials will help monitor your progress and look at everything that’s going on each year—how are things going—and make sure you’re complying with all the various regulations that go along with having a grant. And we work very closely with grants management staff, who are the staff that help actually push the funds out the door and are also monitoring a lot of the different policies once you have a grant. I’m going to start by just walking through what I think are some steps for preparing a grant application.
The first step is to identify a Funding Opportunity Announcement, or a FOA, and I’m going to walk you through this process in just a minute. So once you have a FOA that you are targeting, I suggest you read through it very thoroughly and then at that point prepare a draft of an abstract, a summary, or specific aims. And then contact an appropriate scientific contact or program director to share your draft—and I’ll talk more about how to find out who that person is. And working with the program staff, you can help verify whether your research fits with a particular Funding Opportunity Announcement, whether it’s within the mission of that institute, and who is the likely program official that might be managing the grant once you submit it.
And so after you’ve done this, and only after you’ve done this, should you then prepare your application. I strongly encourage you to reach out to program staff in advance. We are here to help. And then prepare your application. And I strongly advise that you do so with enough time prior to the deadline that you’re able to share your application with peers and mentors to help make sure that the application is clearly written. And I would encourage you to seek out people to help review your grant who are not just those directly in your field or in your lab, people with a little bit broader expertise, so they will have an understanding whether the application is clearly written and whether the peer reviewers will likely understand your work.
Next I’m going to talk about Funding Opportunity Announcements. This is the legal document that outlines all the requirements for the application. It will allow you to determine whether you’re eligible and whether this is a good fit. These documents are quite long and contain a lot of information, but I encourage you to read through it very thoroughly, and any questions that you have, program staff are here to help address those questions.
So walking through a Funding Opportunity Announcement. Here is an example of one, and this is a Funding Opportunity Announcement for the MIRA program, Maximizing Investigators’ Research Award for Early Stage Investigators. So critically, you’ll see the participating organization, which will include NIH, and then below that will include all the institutes and centers that are participating in this funding opportunity. And very importantly, what you’ll see for this opportunity: NIGMS is the sole participating institute.
It’s very critical that you make sure that if you’re wanting to apply for this funding opportunity, that your research is within the mission of NIGMS, and program staff can help you figure that out. And I would encourage you to do that before preparing a full application. Because we’re the only participating institute, if it’s not a good fit and we don’t find that out until after you’ve prepared an application, there is nowhere else to send that application. It will have to be withdrawn. And very sadly, this does happen every year, so please don’t let that happen to you. Reach out to your program staff.
I just want to highlight a few other key details within the Funding Opportunity Announcement. One is the related notices. These are various things that might have changed since the funding opportunity was first published. So you should always go to the website for the most up-to-date version and check out each of these links and make sure that there aren’t any changes in the funding opportunity that will affect your application.
And the next is up here is the funding opportunity purpose. And this will explain what the goal of the funding opportunity is, and this should help you understand whether this is a good fit for you. And then another super-critical component are the key dates. So there’s a lot of different dates to pay attention to, and in particular are the application due dates. And for this Funding Opportunity Announcement, you should note that applications are due just one time per year, each October for the next three years.
So it’s very critical to pay attention to that due date. Some opportunities have as many as three due dates per year, and because this one has just one, you’ll want to make sure that your application is ready and able for your institution to submit this in a timely manner prior to this deadline—hopefully well in advance of the deadline. Do not wait until 4:30 p.m. that day. And then there’s also this section of table of contents, and this is hyperlinked, and you can look and click through to each section of the Funding Opportunity Announcement.
The last section I’d like to focus your attention on is the agency contacts. There are contacts for submitting the application and, very critically, each Funding Opportunity Announcement lists a scientific research contact. And so this is the individual if you have questions about this or need help identifying who might be the specific program official that might manage your grant, you can contact, in this case it would be Dr. Koduri, and reach out to her. And she will tell you more about the funding opportunity and put you in touch with the best program officer to help you. So this is one place where you can always find a contact to help get started.
And then lastly, I just want to focus everyone’s attention on this great resource called the NIH RePORTER. And this is a publicly available database of NIH-funded grants. So this is a great resource no matter what career stage you’re at. If you’re a student looking to go to graduate school, you can look at the funding at a given institution and see if people at the university that you want to attend have NIH funding. This can be a good way to identify people that you want to rotate with to see whether they have NIH grants, or whether an institution has T32 institutional training grants or other things. Or finding what kind of funding of people who do similar research. So this is a very rich tool.
And in terms of helping identify a program officer, there is a great feature called Matchmaker. And the Matchmaker tool is listed here, and here you can enter in your abstract, aims, or summary of your proposal and click for similar program officials, and matchmaker will identify several different individuals across NIH who might be the program officer that manages grants in your area. So this is a great tool that I urge you to look at.
I look forward to answering questions a little bit later in the webinar.
Thanks very much, Desirée. We will answer all the questions at the end—or as many as we can—so we’ll switch to Maqsood now to tell us about the review process.
Dr. Maqsood Wani:
OK. Thank you, Dorit, and I once again welcome everybody to this webinar. And I’ll just give you a very brief, 5-minute overview of the review process.
Once you prepare the application and you submit, I would like to tell you what happens during the course, from the time of submission until receiving the outcome of the review in the form of the score and your critiques. So like Desirée was saying, when you start preparing your application, you must pay close attention to the FOA that you are submitting it for.
Most importantly for the review purposes, you must pay attention to the submission deadline for that application. Every FOA has a submission receipt date. For most of the FOAs there are standard submission dates, which are three times a year, and for new submissions is a separate submission deadline, for the renewals and resubmissions there is a different one. And there are certain FOAs, like she was saying, that have one submission deadline, or it may be also a deadline that may be different from the standard submission deadline, so you must pay attention.
The most important thing that you should keep in mind is do not wait until the last minute. You should keep an internal deadline, maybe a couple of days before the actual submission deadline that NIH has set in the FOA, which helps you once you submit the application and it gets assembled, it gets received here, and after that you find there were some corrections, something did not get uploaded properly, and some wrong attachments got included in the application, you still have a couple of days to fix that and basically resubmit it. But if you submit at the last moment and find later on that there were corrections to be made, those corrections will not be acceptable, and you will be forced to withdraw the application. So I suggest always keep a couple days before the actual deadline to submit.
Your application is submitted through sponsored programs, so you should work with sponsored programs of your institution in such a manner as that your application gets submitted on time. Once the application is submitted through Grants.gov, it goes through NIH eRA Commons validation. The eRA Commons basically pulls the application from the Grants.gov for the NIH, and after the validations, it is received at the Center for Scientific Review in our Division of Receipt and Referral (DRR). Division of Receipt and Referral basically serves as the central point for all of the NIH applications that are submitted to NIH for the funding purpose.
So Division of Receipt and Referral is getting the application through eRA Commons, checks the application is submitted on time, like I said, and it also checks for the compliance. All the applications that are not in compliance, as per the FOA, those applications get withdrawn and get rejected, so you should, again, pay attention to the FOA before you submit an application. The DRR at that point, once the application is in compliance, it is on time, it is accepted by NIH for review purposes, it assigns it an application number, and it also makes two important functions there.
One is the assignment of the institute for the funding purposes, and the second is the assignment for the review purposes. They are two different assignments. The assignment for the institute is based on the FOA, which institutes are participating in that particular FOA for funding reasons, and the review assignments are different, which is which review group is going to review this application in the initial review process.
So if the FOA has multiple institutes participating in the FOA, the Division of Receipt and Referral is going to assign the institute based on the mission of the NIH and the application. They will see what your application is about scientifically and which institute it fits in, in their particular scientific mission and, accordingly, assigns it as a primary assignment for funding reasons.
And it can also assign secondary assignments if there are overlapping interests between the institutes. So you will have a primary assignment; you may have a secondary assignment for the funding purposes. And the second important assignment, it looks at which review group should review this application. So it gets assigned to an integrated review group, called IRG.
For example, I am the chief of the Cell Biology IRG. If an application fits in the cell biology area, it will get assigned to my IRG. So after the application is assigned to the integrated review group, the chief of the IRG basically looks at the application and sees, within the IRG, which of the study sections it fits. Because each IRG has clusters of review groups called study sections, and based on the size of the application, it is seen where this best fits which study section for this review. So that is the second assignment that gets final assignment for the review group where the application will be reviewed.
For these purposes you are given an opportunity to make a request. When you submit an application, you can make a request in the automated referral form, which is part of the application, where you can suggest what study section you think fits your application, and that’s where it should go. We take that into account. We take the request into account. We see if the application can fit in that study section you have requested. We assign it accordingly.
But we are mostly driven by the review guidelines of the particular study section where we think applications will be best reviewed. But you should, and I urge you, look at the study sections. We have more than 200 study sections and special emphasis panels in the Center for Scientific Review that publish review guidelines, and they publish meeting rosters on our website. You should go, check those study section guidelines, you should check who are the members of those study sections in the last three rounds, the reviewers who have participated in, and from that you can get an idea what study section actually fits your application. So make that request accordingly.
And also in that cover letter you can also request if there is anybody you do not want your application to be reviewed. So you can pull that person in conflict with a proper reasoning, and we take that into consideration that when we make the review assignments we take those conflicts into account. So you can make those suggestions in your cover letter. You can also make a suggestion in the cover letter in the automated referral form why to consider which study section you do not want it to be reviewed. So we’ll take that into account also, if there is proper reasoning.
Once the study section assignment is finalized and the study section is administered by a scientific review officer, basically after that point, the science review officer is the one who administers the review of the application of that study section. That person becomes your point of contact during the review process until the review is complete. And this information you will find in your eRA Commons. When the application gets assigned, it automatically sends an email, and you can check at that point your eRA Commons account and see the study section assignment while there will be also the name of the SRO and his or her contact information in there.
At that point, that person becomes your point of contact if you have any questions, and you can just email and you can communicate with that person regarding the review of your application if you have questions. So once the applications are in the study section, the scientific review officer’s job at that point is to perform the administrative review of the applications, look at the size and see what kind of expertise is needed, look at the conflicts and make sure who are the people who cannot review that application, they are in conflict with that application. And also they look at the compliance, and they look at whether it is a duplicate application, whether there are issues with the application that basically warrant withdrawal. So you have to just also make sure that you comply with all the things so that it doesn’t get withdrawn.
The science review officer’s job is to recruit the proper expertise, assign the application to the proper reviewers. They assign each application three reviewers, and these assignments are provided to the reviewers six weeks before the actual meeting date, and during that time the reviewers review the application. They provide the scores and the critiques.
One week before the meeting and on the day of the meeting the application gets discussed. The top 50 percent of the applications are discussed at the meeting, based on the preliminary scores of the reviewers. And the bottom half that do not score higher, better, those 50 percent of the applications do not get discussed.
So at the meeting then, each discussed application receives a final score from the entire panel, and the average score of each panel is calculated for each application to calculate their overall impact score, which is then used in calculating the percentile. At the study section meetings, we discuss the application clusters based on the activity code, new investigators and established investigators. We just basically discuss them separately in clusters.
The not-discussed applications do not get discussed, but they receive a summary statement that does not include any résumé of discussion, but it includes only the criterion score, not the overall impact score. Whereas the discussed applications receive an overall final impact score and résumé of the discussion, what was discussed in the meeting, including the individual critiques of the reviewers.
And the scores are released within three business days after the meeting, and you have access to those scores in your eRA Commons account. And the summary statements after the meeting are released within 30 days after the meeting is concluded. So you should receive your summary statement within 30 days after. If you do not, you should contact the SRO or the program officer who is involved with the application. After the summary statement is released, you have the summary statement, at that time your point of contact becomes your program officer, whose information and contact phone number and email is included on the face page of the summary statement.
So at that time you cannot just contact the SRO anymore; you have to contact your program officer, who becomes your primary contact at that point. After that, the second level of the review process is when the advisory council of a particular institute meets, and they give the advice to the institute for funding advice. In that, the Center for Scientific Review or the initial peer reviewers are not involved in the process, in the second level review.
This whole thing, from the state of submission to when the entire review is complete, takes about four to four and a half months. So you can see what this involves, so how important for you it is to put a good application so that it gets a good outcome without any waste of time. So I think I gave a little overview, and I will be happy to answer any questions. I’m sure there will be many, and I’m here to answer them.
Let’s start with contacting your program officer, which, of course, is contacting the program officer at NIH, not at your school. So if, for example, it’s an R35, a MIRA, and there’s no specific aims field, what should you send—just the abstract?
Yeah, either the abstract, or if you want to send a more complete summary that’s a little bit stronger or the outline of the application, you can input that… send that either to the program officer or start with NIH RePORTER to identify potential program officers.
So could a program officer also suggest which study section would be most appropriate?
Yeah. Oftentimes applicants will email me and ask what kind of study sections the applications tend to go to. So they can ask, and I’ll let them know that my applications in this area tend to go to these couple of study sections. For MIRA, those ones tend to be in a specific one for my applications as well, but I don’t know how well requests work for MIRA.
Maqsood, anything else to add about how to figure out what study section would be the most appropriate?
Yes. I think, like I said, the applicant should contact the program officers, who can make suggestions. The applicant should look at our website, look at the cross-section of the study sections. They can input certain words in there and it can give them suggestions. They can use Matchmaker, which gives them the best fit, the worst fit. They can use the RePORTER and see what kind of application, based on similar size, which study sections that have been reviewed and have been successfully reviewed. So all these things they can use to suggest a study section.
If they don’t suggest any study sections, they don’t know where it should go, we are here to look at those applications and will make the best decision for every application the way it should get reviewed. And sometimes the applicant makes a request, and it may not be a proper request. We make then the decision what is the proper study section it should go.
And for me, like Desirée was saying, there are no standing study sections right now. We just are in the process of making them. But right now they are reviewed in special emphasis panels. Again, we form them in a way so that every panel will have the proper expertise to review those applications.
OK. And I would just add that there’s also a tool on the website called ART where you put in…It’s like Matchmaker but for study sections, where you put in your text and it will also tell you which study sections. So staying on study sections, somebody says, “I found descriptions of the different study sections and one that fits my project. Should I reach out to anyone in that study section prior to creating my application, or is it specifically the program officer I should contact?” Desirée?
You should contact the program officer. You should not contact anyone from the review panel, because that can put that person in conflict with your application. So I would not encourage you to contact individuals from the review panel.
And if I’ve found the right person to contact and I’ve sent an email to the program officer, how long should I wait until I get a reply and what should I do if I don’t get a reply?
So I’d say I think most of us try to respond within a few days, but if you don’t get a response within a week, we do get a lot of emails and sometimes some fall through the cracks. I would definitely feel free and encourage you to reach out again after a week, if you haven’t heard back. And again, it’s really critical to do these contacts in a timely manner. I sometimes get emails two days before the deadline, and even if I’m able to reply immediately and I have some advice for changes you might need to make, there’s no time to do that, so please contact well in advance.
OK. Question for Maqsood. How early can you submit an application before the due date? A week? More?
Any time. You don’t have any specific deadline how early you should submit. You can submit any time. But we see then by what time the application was submitted for it to go to a particular council. So we go by submission to the council. So if the application is submitted even a week before the actual deadline, it will still go to the same council. If it is submitted four weeks before the submission deadline, it will still go the same council. So there is no earlier date.
OK. And can you ask for exclusion of reviewers for a resubmission or only for new submissions?
You can ask anything you want. We have to see what is appropriate. Even if you ask for an exclusion for a new submission, if it is not justified—see, it is a peer review and sometimes a person may say, “I don’t want this person to review my application because he is in the same field; he’s a competitor.” But that’s not an excuse for a person to not review, because it is a peer review. It will be reviewed by peers unless there is a known scientific disagreement and/or you have collaboration with the person, you have future collaborations going to be. Those kinds of things we take into account, that there is a conflict.
But for the resubmissions, if sometimes we get a request by the PIs as, “This reviewer two or reviewer three was not fair, so can I exclude?” So they can say that, and the SRO makes the determination to see whether it is proper or not to put anybody in conflict.
OK, thank you. So I have a question specifically for predoctoral grants, F31s, and this person is interested in knowing, “What’s the section of the application that holds the most weight in scoring an application? Is it the research, the institution, the mentor, letters of recommendation, etc.?” For either of you. Desirée, maybe you have some insights.
I don’t manage individual fellowships, so I would imagine that the research is very key, but so is the mentoring plan is something that we really look at for any of the training programs or fellowships for trainees.
OK. And then just remind everybody how to get to RePORTER. We had a question about that. I believe the website is report.nih.gov, and they can get to it from there.
Just google NIH RePORTER. It’s the first hit.
Yep, Google is your friend. And then a question for Maqsood. You answered it partially, but what reasons might an applicant want to exclude a study section for their application? Can you exclude an entire study section?
Yes, people do exclude study sections. But again, if there is not proper justification, we may not honor it. If we still think that that’s the right place to get it reviewed, then it will go there. So most of the times the PIs exclude study sections not because there is no expertise, but because they see that previously their application has gone to that study section, which is appropriate, but has not fared well. Now they want to say, “I don’t want to be reviewed by there. I want to be reviewed by some other study section.” Because they have already previously seen that in that study section they have not fared well.
So in those cases, if we see there are some overlapping study sections where there may be expertise in a different study section, we do grant that request, because we do see where the application can fare differently in a different study section. But if the study section they want to exclude and they request a new study section and the new study section does not have expertise to review it, we will not honor that request. It will still go back to the study section where it was reviewed before and where there is proper expertise.
OK. Great. So there’s a couple of questions about international students, and so I’m going to combine them and ask, how do you figure out which FOAs international students are eligible for?
So in the section of the Funding Opportunity Announcement it will indicate who is eligible. So for some different fellowships it may say that it’s restricted to US citizens and permanent residents, and if that restriction is not there, then it’s open. So that’s within the Funding Opportunity Announcement. And if you’re having trouble finding that, you can always reach out to the scientific contact and contact program staff, and we can help you.
Right. If in doubt, ask. So Maqsood, could you speak a little bit about—but not much because it will take us the rest of the hour—about percentiling and the fact that, for example, K awards aren’t on percentile, so how do you understand the scores?
So percentiling is done for the study section, because there is a score calibration from round to round and there is the permanent members who sit on those study sections. So in those cases, the application gets percentiled, where we take the current round and the previous two rounds and see the scores on all three rounds and basically calculate the rank, and based on the rank, we calculate the percentile. So it is very important. And the study sections which do not give percentile are special emphasis panels of the K awards sometimes and some FOAs.
For example, MIRA FOAs we do not percentile, and we just basically use the raw scores. And the reviewers in these panels are recruited on an ad hoc basis every time, so there is no permanent membership. But the reviewers are given proper instructions about the scoring. We do give reviewer orientation before each meeting to the reviewers and explain to them how the grants are reviewed, what are the different criteria for reviewing the applications, and how the applications are to be scored and what kind of applications for what. And we give them proper guidance and orientation and they know the scoring behavior.
OK. Good. So I have a question about an application that was “not discussed,” and I think we can talk a little bit about what does “not discussed” actually mean. The question is, “If the grant hasn’t been discussed, how can we resubmit it if we suspect that we want to change the study section, since we don’t have reviews?” It’s not true that you don’t have reviews.
No, the reviews are there.
Yeah, can you explain what “not discussed” actually looks like?
So “not discussed” application. Before the meeting, all the applications receive minimum of three complete critiques and each critique has a subset of scores—scores for each criterion. And each application is given a preliminary score by the reviewers so that before the meeting each application has a set of three preliminary scores. And these preliminary scores are basically averaged and then they’re ranked, the application in each phase is ranked.
Let’s say there are 100 applications. They are ranked based on the preliminary scores, and we see what are the top 50 percent of the applications. Top 50 percent application doesn’t mean that it should be exactly half. Sometimes it can be 60 percent, because some applications have the same scores. We may end up discussing some other portion of the applications. The top half and the bottom half doesn’t mean it is 50/50. It can be 60/40. It can be 70/30. So then the applications in the top tier, top half, they are the ones that get discussed at a study section. And at that time the preliminary scores are thrown out and the applications are discussed, then the panel as a whole provides the final scores after a discussion of each application. Those are the final scores. They are taken into account for overall impact score and percentiling.
Now, the applications which are in the bottom half, which do not get discussed, they don’t receive their score, because it was not discussed. However, they receive the summary statement, which includes the critiques of all three reviewers, as well as the criterion scores that were given before the meeting. So the applicants get an idea which of the parts of the application were weak. So was it approach? Was it significance? Was it innovation?
Because they have the criterion scores. So they can gauge from that. And also the “not discussed” application summary statement does not discuss application, does not have a résumé of the discussion, because it was not discussed. It has everything else in there, so the applicant should know how to improve their application based on the summary statement.
And Desirée, people whose applications weren’t discussed, they can still contact the program officer, right?
Yes, I would strongly encourage you to discuss your summary statement, no matter your score or “not discussed.” Sometimes we can help give a more neutral reading of the summary statement. I know it’s very difficult to get an application not discussed. You’ve put a lot of work and a lot of effort into submitting it, so that can be very upsetting.
But your program officer can help read through and highlight what they thought were the most significant weaknesses that you should address for resubmission. Sometimes I see people like to focus on one small comment that they take offense to, when that might not have been something that was really contributing to the scores. So I think we can help provide that neutral view and give you some guidance on how to move forward.
Right, thanks. So this was a question about K awards, but it’s actually a more general question. So do the reviewers for a resubmitted application see the original reviews before the study section? Maqsood.
Yes, every resubmitted application the reviewers get the previous summary statement included with the assignment, because many times if the reviewer who reviewed it previously was not at the next meeting, it’s a different reviewer this time who had the same expertise. They get to see the previous summary statement and then see what were the comments that were made by the previous reviewers. So they see the entire summary statement and then they see, based on the previous summary statement, if the application and resubmission was improved and there was proper response to the reviewers’ critiques, reviewers’ comments from the previous review. So yes, the panel gets the summary statement from the previous review for a resubmitted application.
OK. Thanks. A basic question: Could you explain how to interpret scores? Low numbers are good, right?
Yes, low numbers are good. The score range is basically one to nine. An application gets a score of one, two, up to nine, without decimals, when the application is scored. And then all the scores are averaged from the entire panel, and it is then multiplied by 10, so eventually you see a score basically from 10 to 90, or you see an ND, which is not discussed. So if you get a 10, that means a score of one. That’s the best. If you get a 90, that’s the worst. The lower the number, the better it is.
OK. Great. Desirée, I have a couple of questions about diversity and diversity supplements. The first one is could you just explain a little bit about the diversity supplements and what they are at NIGMS?
Sure, yeah. I am the person who manages the diversity supplements at NIGMS. So these are once you have a funded, eligible, typically R grant, but there’s a long list of research grants that are eligible, so you should look at the Funding Opportunity Announcement. So if you have NIGMS funding, we will provide opportunity for you to submit a request for a supplement to support from high school students, undergrad students, postbac students who are looking to firm up their credentials and experience so that they will be able to successfully apply to and be admitted to Ph.D. or M.D.-Ph.D. programs, Ph.D. students, and postdocs at NIGMS.
For any individuals in any of these career levels from underrepresented groups are eligible for diversity supplements. So if anyone has an eligible grant at NIGMS, I would strongly encourage reaching out to me. There’s information on our website, but I can provide some more details. And really what we’re looking for in these applications is a strong individualized training plan to the individual. So you have to identify a particular individual that you’re applying on behalf of and just write a strong plan for how they’re going to get trained and how they’re going to move forward in their career.
Sometimes I get questions from students who are interested in this, and I help point them to NIH RePORTER for how they can find a mentor with an eligible grant and information they might need. So if anyone’s interested, please do email me further and I can give more details.
OK. So sort of a similar topic, “As a member of an underrepresented group, does it make sense for me to submit an application specifically for underrepresented groups or to a broader FOA?”
So it depends what the options are. At NIGMS, our F31 grant, we only support F31s to promote diversity, so there’s not an option. That’s the only F31 that we support. And the other area where there’s an opportunity is the K99/R00. So there’s the parent K99, and then we have our brand-new MOSAIC K99 to promote diversity. And this program offers some additional opportunities to get some cohort-based training. So I think it would be up to the individual to decide whether they want to participate in that additional training or not, which FOA you should choose.
OK. And just to clarify on the diversity supplements. Other institutes and centers at NIH also fund them, right?
Yes. I believe all the NIH institutes and centers participate in the diversity supplements. You would apply to the institute where the PI has their parent grant. And the procedures and preferences and priorities may be a little bit different at each institute, so I’d strongly encourage you to be sure to contact the specific institute with any questions.
OK. Couple questions for Maqsood. So for ESI MIRA, are the applications assigned to different study sections, or are they reviewed by a special study section?
For the ESI MIRA, just like other MIRAs, the applications are currently reviewed in special emphasis panels. So we just basically put the applications based on size together in a particular panel. But these panels are made ad hoc from each round, every time, and the reviewers are recruited new every time. So currently we review them in special emphasis panels.
But you review all the ESI MIRAs in…
ESI MIRAs are only once-a-year deadline, so in the round when the ESI MIRAs come, they’re the only ESI MIRAs. All the ESI MIRAs are reviewed together. There is no other MIRA in those panels. There are only ESI MIRAs.
OK. So slightly different topic. “I’ve heard that on occasion a revision gets not discussed or receives a lower score than the original. Can you comment on how to deal with this situation and why this may happen?”
Yeah, it is not really unusual when a resubmitted application gets a worse score than the previous one, because every round the applications are compared to each other. So in a particular round there may be better applications, which receive better scores, and then the application that received a better score in the previous round now is receiving a worse score in the current round. And also, like I said, there are new reviewers each time, and if an application is seen by a new reviewer or even sometimes a previous reviewer can find a new weakness in the application and basically decide to give them not as good a score as previously. So it is not unusual to get a worse score second time if it received previously a better score. It’s not unusual.
And is the presence of other funding looked favorably upon during review? Could it help the score?
It basically shows the track record, and also if somebody had previously some grants, obviously that person should be productive also. And actually if somebody had a grant without any productivity, it may reflect badly. So if you have grants and on top of that you’re already being very productive, you’re contributing to the science, yes, it will help.
OK. Back to Desirée and the diversity supplements. How long is the time frame for a diversity supplement? Is it a year, three years? How long do they usually last?
So that’s variable across the institutes and centers. At NIGMS, typically our awards for the graduate student and postdoc level are 24 months. And postbacs are typically about a year, give or take, and then undergrads can be anywhere from three months to two years and occasionally a little bit more than two years, but that’s pretty rare that somebody applies on behalf of an undergrad who has more than two years left. But, again, this is going to vary between different institutes and centers.
Right. So couple of questions about MIRAs and ESI MIRAs. First of all, like other grants, do MIRAs get scored for each criterion, and how long does it take for the summary statement to come out? Maqsood?
MIRA applications do not get criterion scores, although there are review criteria for those, just like other grant mechanisms. They have similar standard review criteria which is based on the significance, approach, and research environment. And then there are FOAs with MIRA-specific criterion added to each of the criteria. But MIRA applications do not get criterion scores. Like R01s, they do not get criterion scores. They get only the overall impact score.
And the release of the summary statement is the same as any other mechanism. Within 30 days after the meeting, they receive the summary statement. And some people can receive it in the first week, because the summary statements are released in a certain order. But everybody must get it within 30 days, and some people may get it in less than 30 days.
And just a reminder that you really should wait until you get the summary statement before contacting your program officer, because if you contact them when you have the score but not the summary statement yet, they really can’t help you. How many ESI MIRAs are funded on average?
You may have that answer.
I don’t have the answer to that. We do publish data and we’ve just recently published on our Feedback Loop Blog the variety of scores for both ESI and established investigator MIRAs showing which scores get funded, but I don’t know what the number is per year.
Right, and maybe I’ll elaborate just a little bit. It, of course, depends on how many applications we get and how much funds we have, but every year we publish a Feedback Loop Blog post which tells you the success rate for all these mechanisms, including the ESI MIRAs, so you can get a sense of how many we funded. It’s a pretty high success rate, has been in the previous years.
Just in general I recommend that you sign up for our Feedback Loop Blog, because we put a lot on it, a lot of information, information that’s good for you to know. And just Google NIGMS Feedback Loop, and then you can look there and you’ll see one from not that long ago that will tell you the overall impact scores for MIRA. We also have a number of Twitter accounts—@NIGMS is the general NIGMS one, which provides lots of different types of information. But also @NIGMSGenes, which is from program officers and grants management staff to anybody interested in funding at NIGMS. And @NIGMSTraining, which focuses on our training and capacity-building programs. That’s just a little reminder.
I can put up the slide really quick with all the Twitter information.
Exactly. Maybe at the end we’ll put that up. So a lot of the information, the data on the success of our awards, are in the Feedback Loop posts, and when we post there, we also tweet about it. So there’s a question about whether the summary statement states whether the grant will be funded and maybe, Desirée, you can just go through very quickly what happens after the summary statement is out.
Yes. So once the summary statement is out, I encourage you to contact the assigned program officer and discuss with them what the likelihood of funding is. We might have some initial indication, but at that time it’s just an initial indication—does this have a possibility of funding, likely funding, or not likely to be funded is pretty much all we’ll say at that stage.
But then it still has to go through the NIGMS Council-level review, and so once that happens, typically Council does look at all the applications and will look at the summary statements for all but will focus on some specific issues. If we have particular NIGMS-specific funding opportunities or also looking at our appeals and if there’s any people who have a lot of initial funding, they’ll kind of focus on that. And typically, what happens is the applications just get voted as a whole to allow us to consider them for funding.
So it’s after Council then that each division will meet and begin making funding decisions, usually within a month after Council. And then after that it may take some time, depending on what time of year it is, before we make decisions. But if you contact your program director, we can let you know when we think we might have an answer, and typically I’ll say, “If it went to May Council, then maybe by the end of June I should have some definitive information, and I’ll be back in touch. And if you haven’t heard from me, please follow up.” It’s just going to depend on each Council round.
OK. I’m looking at the time, and it looks like we have about three minutes left. So I’d like to give each of you just a minute or two for concluding remarks, and say anything you think that people need to hear. So let’s start with you, Maqsood.
Like I said, I just emphasized in the beginning a couple things that I will emphasize again: to make sure to submit your application on time. Do not wait until the last minute. Give yourself a couple of days before the actual submission deadline so that if there are any corrections to be made, you still have a couple of days to make them. And if the application got uploaded incorrectly or some information did not get included, it still can get corrected. After the deadline is passed, we do not accept any late submission applications, so you lose a lot of time.
Secondly, look at the new guidelines of the study sections. Get familiar with the study topics that are discussed and reviewed in those study sections. Get familiar with the rosters and things like that. And then get in touch with the scientific review officer if you have any questions. And when you see the roster, the roster of the meeting in the round where you have an application is published 30 days before the actual meeting, so that time it becomes public you have the access at that time.
You should look at that roster to see if there are people who have expertise for your application. If you see any issue, you should contact the scientific review officer at that time. Or if you see a person on the roster at that time you may have a serious conflict with, scientific disagreement, or a member of the family or collaborator, you should let the scientific officer know at that time. Do not wait until the application review, then bring the issue up so you have that chance. So I think these things…
I want to give Desirée the last minute. Thanks, Maqsood.
I just encourage anyone interested in applying for funding to reach out to program staff. Reach out early, before you’re creating your application. We are here to provide guidance and help and assist you with this process. And in addition to before completing an application, if you have questions about any kind of supplements that you might be eligible for, we’re happy to discuss that with you and provide you guidance so that you can put in the best application possible that has the best chance for success. So just please contact us. We are here to help.
OK. Thank you both, Desirée and Maqsood, for a very informative seminar, and thank you all for signing in. And as a reminder, a recording of this webinar will be available on the NIGMS website in a few days. So thank you all and bye-bye.
