Recommendations For Change At The NIH'S Center For Scientific Review

PHASE 1 REPORT
PANEL ON SCIENTIFIC BOUNDARIES FOR REVIEW
January 2000

EXECUTIVE SUMMARY

This examination is being carried out in two phases, with extensive involvement of the extramural research community in each phase. With submission of this report to the CSR Advisory Committee, we conclude Phase 1 of our Panel's work. Our Panel has proposed a set of Integrated Review Groups (clusters of scientifically related study sections, referred to as IRGs). These IRGs are designed to facilitate the review of contemporary scientific opportunities and should contribute to the translation of progress in basic research into clinical and behavioral practices. In addition, the revised structure should help CSR anticipate emerging fields of research and accommodate the rapid pace of scientific change. In designing the proposed set of IRGs, we were guided by three principles: (i) There should be at least one appropriate venue for the review of all science that is relevant to contemporary health-related research. (ii) The research topics encompassed by each IRG should be sufficiently cohesive to allow that IRG's external advisory group of scientists to judge the content of the IRG's entire portfolio. (iii) The organization should be flexible enough to adjust to the rapid changes in scientific opportunities expected in the years ahead.

The Panel recommends that CSR's peer review panels be organized in 24 IRGs, compared with 20 IRGs in the current structure. Some IRGs focus on fundamental research that has no immediate or specific application to human health. Others provide for review of fundamental, translational, and clinical research within the context of the biological problem being addressed. Within the proposed structure, IRGs and component study sections that were recently created for the review of neurosciences, AIDS and related research, and behavioral and social sciences will be left intact, pending evaluation of the effectiveness of their organization after they have been functioning for several years. However, new study sections may be added to some of these IRGs in Phase 2.

The Panel assigned high priority to the goal of reviewing applications that apply to a given disease/organ system in the context of the biological question being addressed because we believe that such a system results in the greatest net benefit. However, the Panel is aware that, as is the case no matter which principle guides primary distribution, some research areas will be distributed across many IRGs and study sections. Because applications within a given research area may suffer when they are a small minority of those reviewed by a study section, the Panel expects those individuals involved in Phase 2 to design a review system that ensures that applications related to crosscutting areas are clustered. Above all, peer review must be fair and must be perceived to be fair.

In addition to proposing a set of IRGs, we have also outlined some cultural norms that we believe should govern the CSR review process. The Panel believes that active researchers working in the area of research being proposed are the best judges of the scientific merit of a research application. Advocacy or gate keeping for a field, discipline, or style of research is not the function of a peer reviewer. A peer reviewer's only role is to judge the research proposed, providing the Institutes with honest informed advice about its merits, without undue emphasis on minor technical details. The Panel also notes that if NIH is to accomplish its full mission, applications that propose exploratory research and methods development, in addition to those that propose hypothesis-driven research, must be judged on their potential to impact the ability of the NIH to achieve its mission. Furthermore, the Panel cautions that that an obsession with preliminary data discriminates against bold new ideas.

In Phase 2, which will begin in 2000 and continue through the next two years, expert groups of extramural scientists and NIH staff will create the scientifically related study sections that will populate each IRG on the basis of principles outlined in this report. Existing study sections that are currently functioning according to these principles may remain intact and be placed within one of the IRGs in the new organization. Recommendations will be implemented slowly over the following years.

This Phase 1 report was finalized after careful consideration of the thoughtful comments submitted by more than 800 individuals and scientific societies. In the report, we present our view of the goals of an optimal review process for selection of the highest quality research grant proposals; a proposed new IRG organization; some cultural norms according to which study sections and IRGs should operate; and the procedures and principles planned for Phase 2. We recognize that we are dealing with complicated issues that can not be addressed solely in the abstract. Thus, during Phase 2, when applications are sorted into potential study sections, we may well discover areas in which adjustment to the Phase 1 organization will need to be made. The Panel's progress in phase 2 may be followed at http://www.csr.nih.gov.

INTRODUCTION

The NIH peer review system is designed to ensure that the resources provided by the public for the support of health-related research are allocated as the result of a fair and rigorous competition among scientists. The long-range purpose of this research is to develop knowledge that will add, both directly and indirectly, to the improvement of human health. At the same time, research supported by the NIH plays a critical role in training the next generation of health-related researchers.

NIH's peer review system is recognized as the cornerstone of the NIH extramural program, because it is the principal mechanism by which the Institutes and Centers identify high quality research that is worthy of funding. Established over 50 years ago, the system has been outstandingly successful, and, in fact, may be the most important single reason for the remarkable success of our federally funded biomedical research enterprise. During these 50 years, NIH peer-reviewed research has fueled the acquisition of basic knowledge about every aspect of biomedicine. Application of this knowledge to human health issues now advances at an accelerating rate, profoundly affecting our ability to prevent, diagnose and treat human disease. These stunning successes have resulted in an outpouring of data and have given rise to promising new technologies, causing the way we do science to change and expanding the opportunities for the NIH to fully achieve its mission to improve human health.

The Center for Scientific Review (CSR) manages the peer review process for the majority of the grant applications submitted to the NIH². Since its establishment, the CSR peer review system has evolved continuously. Currently, the CSR Advisory Committee is playing a key role in advising the Director on all aspects of CSR function, and it has initiated a number of activities to improve the peer review process at CSR (see Appendix II). In addition, the Peer Review Oversight Group (PROG) was established in 1995 to provide advice regarding the NIH-wide peer review process (http://grants.nih.gov/grants/peer/peer.htm#prog). However, the rapid progress in biomedicine and its accelerating rate of change increasingly challenge the CSR review system to keep pace.

Through extensive outreach to the extramural research community, as well as through her own assessment, CSR's Director, Dr. Ellie Ehrenfeld, has identified a number of issues regarding study section organization and composition. While these are subjective impressions that are hard to document, they are worthy of concern and sufficiently common to ask whether there are better ways to organize the review process³. For example, researchers perceive that there are no appropriate study sections for some newly emerging fields; that applications describing some of the most productive, highest impact work may be assigned to too few study sections, causing too much of the "best science" to compete with itself; that the scope of some study sections is restricted to research with relatively low impact, resulting in undeserved "entitlements;" and that the breadth of knowledge needed to assess the importance and potential impact of research proposals may sometimes be sacrificed when narrowly focused review committees are formed. Also, certain segments of the research community, including some clinical researchers, behavioral scientists, bioengineers, and developers of technology and instrumentation, believe that they are inadequately served by the existing system.

Many researchers fear that conservatism in the system and an undue requirement for preliminary data discourage innovation. They also note the need to define best practices and to institute procedures to enable them to be applied consistently by peer reviewers, study section chairs, and the Scientific Review Administrators (SRAs), who are the NIH staff officials managing the activities of individual study sections.

Recently, to establish better opportunities for teamwork, as well as a flexible distribution of applications, and a sharing of reviewer expertise believed to be most appropriate for the review of today's science, the integrated review group (a cluster of scientifically-related study sections, referred to as IRG and previously called an initial review group) has replaced the individual study section as the functional unit of review.⁴  For example, although seven study sections review applications related to various aspects of molecular, cellular, and developmental neuroscience, they are organized into one IRG, much as multiple courses on related subjects are organized into a single academic department's curriculum. External advisory groups composed of extramural scientists are being established for each IRG as ad hoc working groups of the CSR Advisory Committee to provide expert input for the continual improvement of its review processes⁵. However, until now there has been no overall assessment as to whether the present IRGs and their component study sections are properly configured to respond to current and future research opportunities, so as to best promote the long-range health goals of society.

For this reason, our Panel on Scientific Boundaries for Review (see Appendix III for the roster of our Panel) was established in April 1998 as a working group of the CSR Advisory Committee to undertake a comprehensive examination of the organization and function of the CSR review process. Our Panel is conducting its evaluation in two parts, with extensive involvement of the extramural research community. The Panel has now concluded Phase 1, in which we have proposed a set of IRGs that are designed to optimize the CSR review process, as well as identified a set of cultural norms that we believe should govern the operation of the CSR review process. Here, we have benefited from a number of ongoing initiatives and recent changes that have established a foundation upon which we can build more comprehensive reforms (described in Appendix II).

In Phase 2, which will begin in 2000 and continue through the next two years, expert groups of extramural scientists and NIH staff will create the scientifically based study sections that will populate each IRG. Recommendations will be implemented by CSR carefully over the following years.

Our work has benefited from the comments of more than 800 individuals and societies who reviewed our Phase 1 draft report⁶. Based on their input and on consultations with several concerned communities, we have expanded and refined our proposed organization of IRGs, and have attempted to achieve greater clarity regarding our intentions in this final report to the CSR Advisory Committee. We begin with our view of the goals of an optimal review process for identification of the highest quality research grant proposals (Section I). The report then continues with a proposed new IRG organization (Section II); some cultural norms according to which study sections and IRGs should operate (Section III); and an outline of the procedures and principles planned for Phase 2 (Section IV). To benefit the reader, these sections are deliberately concise, with details and expanded explanations provided in footnotes and accompanying appendices that: provide a fuller description of the IRGs (Appendix I); describe ongoing related activities to improve the NIH review process (Appendix II); name the members of the panel (Appendix III); and summarize the current IRG structure (Appendix IV).

I.      GOALS OF THE RESEARCH GRANT APPLICATION REVIEW PROCESS

In addition to making efficient use of time and resources, CSR must continue to ensure that the following criteria for an optimal review process are met:

• Set high standards: The review process should continue to set a high standard for scientific  excellence, facilitating the selection of the best scientists and ideas on any given topic for NIH support.
  
  
• Contribute to the advance of health-related science: The peer review process should optimize the benefits that can be gained from the progress of science and its contributions to health.
  
  
• Encourage innovation and risk taking: Peer reviewers must strike a difficult balance. They must weed out projects that have no chance of success while keeping in mind that when knowledge is generated and combined in new and unexpected ways, discoveries are made that can advance health-related research and dramatically improve human health.
  
  
• Exercise fairness: For scientists and the public to maintain confidence that the NIH review system is fair, there must be clarity about the means by which reviewers and staff are chosen, trained, and held accountable. Judgments made in distributing limited resources are difficult, both for those who make them and for those being judged. There is unanimous support for the principles of peer review, despite the inherent complexity in executing these principles.
  
  
• Be monitored continuously: To ensure that it is responsive to the rapid changes that energize and renew health-related research, the system should be assessed at regular intervals. Such examination should be conducted not only at the level of the IRG, via external advisory groups as planned, but globally by the CSR Advisory Committee based on input from applicants, reviewers, and NIH staff as described in Appendix II.
  
  
• Be clearly explained: The process must be clearly explained so as to be understood not only by scientists but also by the general public, which needs to comprehend both the substance of NIH-supported science and the selection processes that are used to fund it.
  

II.     PROPOSED STRUCTURE OF INTEGRATED REVIEW GROUPS

A.     Guiding Principles

In designing the proposed set of IRGs described below, we have been guided by the following general principles:

1. There should be at least one appropriate venue for the review of all science that is relevant to contemporary health-related research.
   
   
2. The research topics encompassed by each IRG should be sufficiently cohesive to allow an external advisory group of scientists for that IRG to judge the content of its entire portfolio.
   
   
3. The organization should be flexible enough to adjust to the rapid changes in scientific opportunities expected in the years ahead.
   

B.      Proposed IRGs

In designing IRGs, we have attempted to cluster, whenever possible, all types of research whose major emphasis is focused on a given organ system or disease in an IRG devoted to that system or disease. For example, the IRG for cardiovascular sciences would include basic studies of heart and blood vessel development and physiology, studies of pathophysiology of the heart and vasculature, and clinical studies pertaining to specific cardiovascular diseases and their treatment. However, within the IRG, individual study sections may be either designed to permit clustering of similar types of scientific approaches (e.g., molecular projects or patient-oriented studies) or to integrate such studies as is deemed appropriate for the field (See Section IV). For example, in the present system, a grant application proposing an investigation of the detailed mechanisms that cause specific genes to respond to hormonal ligands would likely be reviewed in the Cell Development and Function IRG. In the proposed new system, the application would likely be reviewed in a study section in the Endocrinology, Metabolism, and Reproductive Sciences IRG, albeit one with a molecular focus.

The rationale for this decision is as follows.

• The overarching mission of NIH to improve human health will be best served by reviewing clinically relevant science in the context of the basic knowledge on which it is founded, and by reviewing basic science in the context of the human condition that it is designed to improve.
  
  
• The new opportunities created by research require that the new knowledge from basic studies be applied to disease problems in a timely and appropriate way, so as to translate progress in basic science into progress in improving human health. Reciprocally, a strong communication link is needed between those engaged in human disease research and basic scientists, in order to help direct new efforts aimed at understanding disease processes through fundamental research. Thus, the organization we propose provides for review of fundamental, translational, and clinical research in the context of the biological problem being addressed. In addition, it accommodates the growth of molecular medicine by acknowledging organizationally that it is through application of approaches in molecular and cell biology and in genetics that many key advances will be made.
  
  
• Within an IRG, the exchange of ideas and perspectives among reviewers who work on the same biological system or group of diseases, but who have very different skills and points of view, may help to accommodate review and promote identification of more ambitious and interdisciplinary proposals and to encourage the submission of broader and more innovative research applications to the NIH.
  
  
• The review of basic and applied science in the same IRG can provide an overview of the quality of the work across all of its study sections. This may enable NIH Institute staff to monitor the relative scientific merits of fundamental and clinical research continually to ensure that it responds to changes in the relative opportunities in a given field, as well as to the relative quality of the investigators available to conduct the work.
  
  
•  Finally, the new structure will allow the distribution of applications that use a particularly powerful methodology to different IRGs or study sections, thus enhancing the probability of applying these new methods to solve many disease and system-oriented problems.
  
  

We have also created IRGs for review of basic scientific discovery and the development of methods that do not apply to any specific system or disease. We have done so because a wealth of experience and documentation has made it clear that clinical advances often rest on the results of basic research. To ensure a vigorous foundation for future progress, NIH must continue to support a sizable proportion of research that has no immediate or specific application to human health. Much of this research must be reviewed in a fundamental context, without regard to a specific organ, biological system, or disease.

The Panel recognizes that no system can achieve perfect clustering for all areas of science. No matter which principle guides the primary distribution, conflicts will arise. Our Panel assigned high priority to the goal of reviewing applications that apply to a given disease/organ system in the context of the biological question being addressed. We believe that such a system results in the greatest net benefit⁷. We are aware that some research areas will be distributed across many IRGs and study sections, with no single focus for review and that applications within a given research area may suffer when they represent a small minority of those reviewed by a study section. Therefore, the Panel expects those individuals involved in Phase 2 to ensure that applications related to crosscutting areas will be clustered so that they constitute at least 30 percent of applications in a study section. The membership of each study section should also include at least 30 percent who are experts for each area to be reviewed. Having the IRG serve as the functional unit of review should be helpful in this regard.

Our Panel recommends the following 24 IRGs that are intended as a framework to guide creation of study sections in Phase 2. There is much in the current IRG design that is consistent with our proposed structure, and these aspects have been retained. Similarly, the Panel expects that many current study sections that are functioning successfully according to the principles outlined in section IV may be retained and placed within one of the IRGs in the new organization. Our Panel has not considered study sections in Phase 1, and we expect those involved in Phase 2 to adjust as necessary the superstructure we propose in this concept paper. Furthermore, as new areas emerge in the future, the IRG system must remain flexible so as to accommodate them. A fuller description, including representative topics to be covered within each IRG and the relationship between the current and proposed IRGs, is provided in Appendix I.

1. BIOLOGICAL CHEMISTRY AND MACROMOLECULAR BIOPHYSICS IRG - This IRG will consider research applications focused on the detailed structures, chemistry, and physics of macromolecules and interacting small molecules.
   
   
2. MOLECULAR APPROACHES TO GENE FUNCTION IRG - This IRG will consider research applications focused on the molecular mechanisms and regulation of the global processes of gene expression that are fundamental to all living cells.
   
   
3. CELL FUNCTION AND INTERACTIONS IRG - This IRG will consider research applications at the level of the functions, interactions, and regulation of cells and cellular organelles, focusing on fundamental cell biological processes.
   
   
4. FUNDAMENTAL GENETICS AND POPULATION BIOLOGY IRG - This IRG will consider research applications focused on general issues in genetics, genomics and population dynamics.
   
   
5. BIOLOGY OF DEVELOPMENT AND AGING IRG - This IRG will consider research applications focused on understanding the biology of the development of cells, tissues, and organisms -- from the single cell stage through maturity, senescence, and death.
   
   
6. FUNDAMENTAL BIOENGINEERING AND TECHNOLOGY DEVELOPMENT IRG - This IRG will consider research applications to develop fundamental, broadly applicable new methodologies and instrumentation.
   
   
7. HEALTH OF THE POPULATION IRG - This IRG will consider research applications focused on broad social, environmental, ethical, cultural, and other contextual influences on health behavior.
   
   
8. RISK, PREVENTION, AND HEALTH BEHAVIOR IRG - This IRG will consider research applications focused on risk, prevention, and health in the individual; studies of risk and protective processes, including genetic risk factors; and studies of interventions aimed at reducing these risks.
   
   
9. BEHAVIORAL AND BIOBEHAVIORAL PROCESSES IRG - This IRG will consider research applications focused on cognitive, perceptual, intra- and interpersonal processes, studies of normal and disordered movement, and studies of developmental, psychopathological and substance use disorders.
   
   
10.  IMMUNOLOGY IRG - This IRG will consider research applications ranging from basic through clinical studies focused on immunology and on the diseases that are principally immunological in their origins or manifestations.
    
    
11. INFECTIOUS DISEASES AND MICROBIOLOGY IRG - This IRG will consider research applications ranging from basic through clinical studies focused on infectious diseases and microbes.
    
    
12. AIDS AND AIDS-RELATED RESEARCH IRG - This IRG will consider research applications ranging from basic through clinical studies focused on all aspects of AIDS and related research.
    
    
13. ONCOLOGICAL SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on cancer.
    
    
14. HEMATOLOGY IRG - This IRG will consider research applications ranging from basic through clinical studies focused on blood cells and their diseases, and studies of the coagulation system and its pathology.
    
    
15. CARDIOVASCULAR SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the heart and vasculature and specific cardiovascular diseases and their treatment.
    
    
16. ENDOCRINOLOGY, METABOLISM, AND REPRODUCTIVE SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the endocrine and reproductive systems and associated diseases, as well as basic intermediary metabolism and metabolic disorders.
    
    
17. BONE, MUSCLE, CONNECTIVE TISSUE, AND SKIN IRG - This IRG will consider research applications ranging from basic through clinical studies focused on connective tissues, which include bone and dental tissues, skeletal muscle, skin, and the extracellular matrix, and their associated diseases.
    
    
18. DIGESTIVE SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the entire gastrointestinal tract and related organs.
    
    
19. PULMONARY SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the lung and respiratory system and pulmonary diseases and their treatment.
    
    
20. RENAL AND UROLOGICAL SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the renal and genitourinary systems.
    
    
21. SURGERY, APPLIED IMAGING, AND APPLIED BIOENGINEERING IRG - This IRG will consider research applications focused on surgical and related disciplines and on applied imaging and applied bioengineering.
    
    
22. MOLECULAR, CELLULAR, AND DEVELOPMENTAL NEUROSCIENCE IRG - This IRG will consider research applications focused on the basic mechanisms that determine the structure and function of neurons, glia, and other excitable cells, and aspects of development in both the central and peripheral nervous system.
    
    
23. INTEGRATIVE, FUNCTIONAL, AND COGNITIVE NEUROSCIENCE IRG - This IRG will consider research applications focused on how the nervous system is organized and functions at an integrative, systems level.
    
    
24. BRAIN DISORDERS AND CLINICAL NEUROSCIENCE IRG - This IRG will consider research applications focused on disease and injury to the nervous system, including issues of neural substrate, functional consequences, and rehabilitation.
    
    

III.      CULTURAL CHANGES

In addition to optimally organizing IRGs and study sections, the Panel believes that the key to maximizing the health of peer review in CSR lies in instilling in the applicants, reviewers and administrators a shared perception of their roles and responsibilities: a broad agreement -- in principle and in practice -- concerning not only what reviewers should be doing, but also how they should be doing it. Below, our Panel recommends a set of cultural norms according to which we believe study sections and IRGs should operate. Adopting them need not await the proposed reorganization. Rather, the system could benefit from implementing them now.

The Panel notes that this process is already well underway, in conjunction with the CSR Advisory Committee. Thus documents that define and codify the responsibilities and functions of study section chairs, SRAs, and study section members, and procedures for the periodic review of the performance of study sections are being developed as described in Appendix II. As they are completed these documents will be made publicly available at http://www.csr.nih.gov. They can also serve to provide research grant applicants with useful insight into how study sections are expected to operate.

1. Who is a peer? - Peer review is based on the idea that scientists who are active researchers working in the area of research being proposed are the best judges of the scientific merit of a research proposal (including the quality and originality of the ideas, the level of craftsmanship, the likelihood of success, and the eventual impact on medicine, science, health, and society). The Panel believes that an appropriate peer reviewer is an active researcher, who can understand and judge both the research goals and the research means being proposed by the applicants. Peer reviewers need not be scientific "competitors" of the applicant, or even be studying the same disease or organ system. But they should be experienced researchers who are reasonably diverse in seniority, outlook, geographical location, gender, and ethnicity and who have achieved recognition for their own research accomplishments. Such recognition may include successful competition for research grants, but will vary depending on whether they work in academia, industry or government-sponsored intramural laboratories, or are, for example, biologists, engineers, or physical scientists.
   
   
2. What is the role of a peer reviewer? - The Panel believes that a peer reviewer's only role is to judge the research proposed, providing the Institutes with honest and informed advice about the merits of the proposals under review. Advocacy or gate keeping for a field, discipline, or style of research is not the function of a peer reviewer. Thus the Panel urges that every effort be made to recruit the wisest and most knowledgeable peer reviewers, and to instruct them that their only function is to judge the research proposals on their merits. Any residual confusion about the role of reviewers in keeping fields alive, or in helping researchers succeed, should be eliminated. These functions may be important to the overall scientific enterprise, but they are inappropriate to the review process and should be pursued separately.
   
   
3. What are the roles and responsibilities of the Scientific Review Administrators (SRAs), Chairs, and Members of study sections? - The chair and the SRA are partners with complementary roles and responsibilities in the peer review system; both are essential if the system is to work well. The SRA (as the Designated Federal Official in charge) is responsible for managing the proceedings, ensuring that they are fair and thorough, and maintaining the integrity of the process. The composition of the study section, the assignments of grants for review, and the final summary statements are by statute the responsibility of the SRA. However, each of these activities should be conducted in close partnership with the Chair of the study section, who has major responsibility for ensuring that the study section members determine scientific merit accurately and fairly and for the substance of the judgments of the grant applications. Chairs must therefore be very carefully chosen for their leadership abilities and breadth of knowledge.
   
   
4. What is the appropriate relationship between study sections and disciplines? - There are true disciplines (biochemistry, genetics, physiology, statistics, pathology) that have come to underlie and be deeply entwined in all kinds of research. But the Panel believes that fostering the health and propagation of disciplines is not the function of NIH, but rather of universities and other extramural research institutions. In its proposals for reorganization, the Panel recognizes the need for well-informed biochemists, geneticists, physiologists, statisticians, and pathologists on many diverse peer review groups. Although there will be valid reasons for technology development in particular disciplines, the need for "disciplinary" study sections as such may well have passed. The review system must emphasize that the NIH supports research, not disciplines or fields. Making review groups suit the nature of an applicant's research, irrespective of his/her discipline or field, should be a continuing goal of CSR and the NIH peer review system.
   
   
5. What types of research have the potential to have an impact on the ability of the NIH to achieve its mission? - If NIH is to accomplish its full mission, all research styles must be judged on their potential to advance our understanding of biological systems, to improve the prevention and treatment of disease, and to enhance the health and well being of people. Certainly, the development and application of new methods and technologies have been critical to the advance of biomedical science. However, peer review of NIH research grants has traditionally emphasized the testing of hypotheses. Although "hypothesis-driven" research could be interpreted broadly to include all the styles of research - including generating the knowledge needed to solve important problems and developing novel techniques or instrumentation to enable knowledge generation -- the practice of many NIH study sections has been to interpret it narrowly as a formal exercise in the proposal and proof of a well circumscribed idea. Under these conditions, exploratory research using and developing new technology suffers, and opportunities for generating new knowledge and ideas are thereby lost. Such an exclusionary insistence on hypothesis-driven research can impede the ability of NIH to accomplish its broader charge. Both hypothesis-driven and exploratory research should be judged on their merits and promise.
   
   
6. What should a grant application propose? The American public expects the NIH to seek out ideas and knowledge that can, directly or indirectly, improve human health. Thus, the NIH must support a broad portfolio of grants ranging from projects that offer significant evolutionary advances through high risk/high reward efforts with the potential to trigger revolutionary changes. However, the Panel is concerned that in practice the present system tends to discourage risk taking and to undervalue new ideas. We urge that reviewers endorse the importance of ideas that are original and have yet to be tried. Peer reviewers should eschew the common current tendency to find fault or to identify minor errors. Instead, they should strive to assess the potential impact of the proposed research and to encourage good ideas and novel concepts, even if they appear to be risky. Countering the conservatism of the peer review system is a critical issue that should become a primary long-term focus for the CSR.
   
   
7. What perspective should be used in review, and how should the results of the review be communicated? All members of study sections should judge grant applications on their fundamental merits (significance, approach, innovation, investigator, and environment), paying close attention to potential impact and the quality of the investigator without undue emphasis on minor technical details. Although it may be appropriate for peer reviewers to provide some helpful general advice on ways to improve the application, they should not be in the business of designing the next experiments or grant applications for applicants. If a proposal is poor because, for example, it is likely to have little or no impact, it should receive a poor score. The summary statement should convey the rationale for the score to the applicant, Institute staff and advisory councils. It should not be a detailed recounting of errors nor a compendium of redesigned experiments for a future amended application. Adherence to this recommendation would simplify the task of reviewers, who must nevertheless provide enough justification to satisfy the other study section members with respect to the fairness and scientific merit of the review and the recommendations.
   
   
8. What is the role of preliminary data? Although preliminary data can assure reviewers that the applicant has the means and the understanding needed to carry out the proposed studies, the Panel cautions that an obsession with preliminary data discriminates against bold new ideas, against young scientists, and against risk taking. For new ideas, little or no preliminary data may be required.
   
   
9. What procedures can be introduced to improve the operation of study sections? In addition to the above norms, the Panel urges that procedures be adopted that will encourage and make feasible the detailed study and review of each proposal by more members than provided in the "first and second reviewer" format used at present. We also encourage the CSR to explore procedures whereby, to the maximum extent possible, applicants can participate in the initial referral of their applications to what they consider to be the most appropriate IRGs and study sections to evaluate the work proposed. Finally, with the trend toward extensive collaborations within and across disciplines and institutions, the Panel recommends that CSR explore the more flexible ways of operation that are likely needed to facilitate the multi-investigator projects that increasingly characterize today's science.
   
   

IV.      PROCEDURES AND PRINCIPLES TO BE FOLLOWED IN PHASE 2

In Phase 2, the study sections within the IRGs will be designed, beginning in 2000 and likely continuing throughout the next two years. The Panel recommends that the seven recently created IRGs for review of neuroscience, behavioral and social sciences, and AIDS and related research, and their current study sections, be exempt from this design procedure at this time and that the effectiveness of their present organization be evaluated after they have been functioning for several years. Such an evaluation process is currently underway for the recently reorganized neuroscience IRGs.

Dozens of experts in the specific scientific and medical fields under review⁸ will be enlisted to work with NIH staff and at least one member of the parent Boundaries Panel to design study sections for the remaining 17 proposed IRGs. Several groups will be convened in 2000 and the rest in 2001. These groups will conduct mock sorts of hundreds of applications into postulated study sections. Opportunity for even greater input from the extramural community will be provided by posting the proposed study sections on the web for broad public comment and subsequent refinement. A phased implementation may begin in 2001.

To ensure that the review process evolves to accommodate the ongoing emergence of new scientific opportunities and practices, our Panel recommends that formalized evaluations be carried out at regular intervals to assess the structure and function of the CSR peer review process. Thus, the study section design produced in Phase 2 should be reviewed on approximately a five-year basis by the ad hoc external advisory groups for each IRG that are being established to report to the CSR Advisory Committee. In addition to this intra-IRG evaluation, a global assessment of the study sections, based on input from applicants, reviewers, and NIH staff, should be made periodically by the CSR Advisory Committee as described in Appendix II. Furthermore, our Panel recommends that the CSR employ, for all IRGs, the formal evaluation efforts that are currently being developed for the IRGs and study sections created as a result of the recent reorganization of neuroscience and behavioral and social science review committees.

Our Panel believes that study sections should be created according to the following principles, even while recognizing that some among them represent competing and conflicting goals and that judgment will be required to reconcile these conflicts.

1. Not too narrow: The range of science that is considered by each study section should be sufficiently broad as to allow a comparative evaluation of the merit of grant proposals in several related fields. Study sections should not be restricted either to a single field or to a single experimental approach.
   
   
2. Not too broad: On the other hand, the range of science considered by each study section should be sufficiently coherent to allow its members to reach independent evaluation of the merit of each proposal, following the Panel discussion of each one.
   
   
3. Overlap is desirable: To allow flexibility in review, the range of scientific expertise of study sections within an IRG (and often between IRGs) should overlap, such that more than one study section could appropriately review any individual grant application.
   
   
4. Connected to specific diseases or organs: Whenever appropriate, basic research should be reviewed in the context of the biological question to which it relates; i.e., in an IRG focused on a particular organ, biological system and/or disease. However, within the IRG, individual study sections may be designed either to cover a single type of scientific approach (e.g., molecular studies versus patient-oriented projects), or to integrate such studies as deemed appropriate for the field.
   
   
5. Connected to basic science: Because applied research must be rooted in a firm understanding of underlying physical, biological, and behavioral mechanisms, it is important for grant applications on applied subjects to receive a careful review of the basic science on which they are founded. Furthermore, while clinical research applications should be reviewed in clinical research panels to the extent possible, their review should be informed by the perspective of basic scientists to ensure that the highest quality of contemporary science is applied to each research project.
   
   
6. Density of expertise: When it is necessary or desirable to review very different types of research, for example clinical and basic, or hypothesis- and problem-driven, in the same study section, at least 30 percent of the applications reviewed should be of each type. In addition, reviewers with expertise in each broad category of research should comprise no less than 30 percent of the total membership of the study section. To provide appropriate review for grants focused on patient-oriented and translational studies, each of the disease/system -based IRGs should have study sections that include clinical and basic scientists with the necessary expertise to review patient-oriented and translational studies. On occasion, applications proposing translational research will include a range of investigators from disparate disciplines. When review of such applications requires technical expertise not available within the study section, consultants should be called upon to provide reviews of the specific methodological or technical approaches in question. If the volume of grants relating to patient-oriented and translational research is not adequate to justify a dedicated study section within a given IRG, a trans-IRG study section should be constituted to provide appropriate review for these applications.
   
   
7. Balance of breadth and depth of study section members: The study section should be populated to balance the sometimes conflicting goals of furnishing sufficient expertise for rigorous review and providing sufficient breadth and understanding of the critical issues, so that the true merit and significance of each research proposal can be well assessed.
   
   
8. Flexibility of reviewers: To help achieve sufficient breadth and depth of expertise on each study section, and to create flexibility while maintaining consistency of review procedures and practices within (and even across) IRGs, some study section members should be recruited into an IRG-wide pool of "mobile experts," to serve as more broadly-based general advisers to related study sections. Review of specific methodological issues requiring particular expertise that may not be available to most study section members should be requested from appropriate experts as "mail reviewers" in advance of the study section meeting.
   
   
9. Serving multiple Institutes: CSR should organize study sections clustered within irgs by scientific area, not by programmatic mission. In most cases this will mean that study sections will evaluate applications assigned to multiple institutes. While it is also the case that some study sections may deal with applications assigned to one institute, this should result from scientific coincidence and not from a deliberate pairing.
   

CONCLUSION

In this Phase 1 report, the Panel has attempted to provide our best advice on the revised organization and cultural norms that should govern the review of NIH grant applications by the Center for Scientific Review. Many details and adjustments remain to be made in Phase 2, probably including some fine tuning of the proposed IRG organization, and we recognize that judgment will be required to balance obvious tensions between worthy goals.

This has been a very challenging task. We are grateful to the many scientists and scientific societies that have provided thoughtful comments. This final Phase 1 report has certainly been improved thereby, and an even greater opportunity for public input will be provided during the development of Phase 2.

While recognizing that perfection is unattainable, our goal is to optimize the CSR review system to provide a review process that encourages risk-taking and innovation and is flexible and responsive enough to keep up with the many new opportunities developed by the striking advances in health-related science. We hope that the final result will be a dynamic system that appreciates new ideas and all research styles - and that facilitates acceleration of the pace of progress in NIH-supported research through an improved, merit-based competitive review of all applications⁹.

APPENDIX I
PROPOSED IRG STRUCTURE

Twenty-four IRGs are listed below, beginning with those concerned with the most fundamental biological processes or technologies that underlie approaches to the properties and treatment of all organ systems, disease processes, and general health issues. Limited examples of topics to be covered within each IRG are provided. These topic lists do not represent study sections. As noted previously, study sections will be defined in Phase 2. Note also that the Panel made no attempt to be exhaustive. Thus, many additional topics appropriate for each IRG have not been mentioned, and the absence of any one topic does not in any way imply its intentional omission. For example, many cross-cutting issues, such as patient-oriented research, complementary and alternative medicine, autonomic nervous system function related to all organs, rehabilitation, life-course changes, effects of environment, pharmacology, toxicology, molecular and cell biology and development of organs, have not been listed within the many IRGs. The Panel expects the scientists responsible for Phase 2 to design a review system that ensures that applications related to crosscutting areas are clustered so that they constitute 30 percent of applications in the study section and that at least 30 percent of expertise for any given area reviewed is achieved. The Panel also did not consider the balance of grant applications across IRGs in Phase 1 and assigned one IRG for each broad domain of science, expecting that adjustments to accommodate numbers of grant applications will be made in Phase 2¹⁰.

In the proposed organization, the seven IRGs created to integrate the review of neuroscience research, behavioral and social sciences research, and AIDS and related research from the former Alcohol, Drug Abuse, and Mental Health Administration Institutes [Molecular, Cellular, and Developmental Neuroscience; Integrative, Functional, and Cognitive Neuroscience; Brain Disorders and Clinical Neuroscience; Social Sciences, Nursing, Epidemiology and Methods (titled Health of the Population in the new structure); Risk, Prevention, and Health Behavior; Behavioral and Biobehavioral Processes; and AIDS and related research] have been retained essentially intact in proposed IRGs (http://www.csr.nih.gov/review/reorgact.asp). Moreover, our Panel intends for the study sections that were created in these recently reformulated IRGs to be left intact at this time, and that the effectiveness of their organization be evaluated after they have been functioning for several years.

Two current IRGs (Biochemical Sciences and Nutritional and Metabolic Sciences) are not continued, and their research applications are widely dispersed. Another six of the 20 current IRGs (Biophysical and Chemical Sciences; Genetic Sciences; Immunological Sciences; Infectious Diseases and Microbiology; Oncological Sciences; and Endocrinology and Reproductive Sciences), while altered in their scope, clearly relate to proposed IRGs. In general, the current Cell Development and Function IRG would be divided into the Molecular Approaches to Gene Function; Cell Function and Interactions; and Biology of Development and Aging IRGs; the Musculoskeletal and Dental Sciences IRG into the Bone, Muscle, Connective Tissue, and Skin and the Renal and Urological Sciences IRGs; the current Surgery, Radiology, and Bioengineering IRG into the Fundamental Bioengineering and Technology Development and the Surgery, Applied Imaging, and Applied Bioengineering IRGs; the current Pathophysiological Sciences IRG into the Digestive Sciences and the Pulmonary Sciences IRGs; and the current Cardiovascular Sciences IRG into the Cardiovascular Sciences and the Hematology IRGs. The current IRGs are listed in Appendix IV and described at http://www.csr.nih.gov/review/irgdesc.htm.

1. BIOLOGICAL CHEMISTRY AND MACROMOLECULAR BIOPHYSICS IRG - This IRG will consider research applications focused on the detailed structures, chemistry, and physics of macromolecules and interacting small molecules. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: chemical dynamics and mechanisms; chemical synthesis; enzymology and catalysis; protein chemistry, structure and folding; nucleic acid chemistry, structure and folding; oligosaccharide and polysaccharide biochemistry; and lipid chemistry.
   
   
2. MOLECULAR APPROACHES TO GENE FUNCTION IRG - This IRG will consider research applications focused on the molecular mechanisms and regulation of the global processes of gene expression that are fundamental to all living cells. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: genome replication; regulatory mechanisms of gene expression; chromosome structure and dynamics; DNA recombination and repair; DNA transcription and translation; RNA processing, stability and degradation; and protein processing, stability and degradation.
   
   
3. MOLECULAR APPROACHES TO CELL FUNCTION AND INTERACTIONS IRG - This IRG will consider research applications at the level of the functions, interactions, and regulation of cells and cellular organelles, focusing on fundamental cell biological processes. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: membrane transport, structure and dynamics; membrane and protein trafficking; organelles and compartments; cytoskeleton, cell division, and cell movements; bioenergetics; intra- and intercellular signaling; cell cycle control; apoptosis; cell junctions and extracellular matrix.
   
   
4. FUNDAMENTAL GENETICS AND POPULATION BIOLOGY IRG - This IRG will consider research applications focused on general issues in genetics, genomics and related aspects of population dynamics. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: prokaryotic and general eukaryotic genetics and genomics; mammalian and human genetics and genomics; quantitative genetics and genetics of complex traits; population genetics; evolutionary genetics; genetic/molecular epidemiology; and genome structure and function.
   
   
5. BIOLOGY OF DEVELOPMENT AND AGING IRG - This IRG will consider research applications focused on understanding the biology of the development of cells, tissues, and organisms from the single cell stage through maturity, senescence, and death. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: stem cells; embryonic development and cell differentiation; organization of cells in tissues; cell and tissue maintenance, repair, and aging; and age-dependent physiology and pharmacology (childhood through geriatrics).
   
   
6. FUNDAMENTAL BIOENGINEERING AND TECHNOLOGY DEVELOPMENT IRG - This IRG will consider research applications to develop fundamental, broadly applicable new methodologies and instrumentation. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: engineering underpinnings of biomedicine; instrumentation development; computational methods and informatics; biostatistical methods; mathematical modeling; imaging techniques; genome technology; and general methods for gene therapy. It is expected that these basic technologies, when mature, will be applied to address a range of biological problems that would be reviewed in the Surgery, Applied Imaging, and Applied Bioengineering IRG and in other disease/systems IRGs as appropriate.
   
   
7. HEALTH OF THE POPULATION IRG - This IRG, currently named Social Science, Nursing, Epidemiology and Methods (SNEM), will consider research applications seeking to understand and elaborate the broader socio-environmental contexts in which health and health-related behavior are embedded and to examine the interaction of these socioenvironmental factors with the health and health-related behaviors and populations. Studies of biomedical ethics will also be considered in this IRG. This IRG reflects the rapid growth of a variety of disciplines related to the general theme of health promotion and disease prevention, including behavioral, social, and population-based approaches. Healthy behavior refers to studies related to behaviors important for the health of the population, including nutrition, physical activity, adherence to medical treatment regimes, smoking, substance use and violence. Biomedical ethics includes studies related to the ethics of the human genome project and end-of-life issues. Biostatistical issues include those related to population studies. While other study sections may be added to this IRG, the existing study sections, created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/bss.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
   
   
8. RISK, PREVENTION, AND HEALTH BEHAVIOR IRG - This IRG will consider research applications focused on a wide range of biological, psychological, cultural and social conditions and traits that affect the manifestation, prevention, treatment or management of physical and mental diseases and disorders. While other study sections may be added to this IRG, the existing study sections, created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/bss.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
   
   
9. BEHAVIORAL AND BIOBEHAVIORAL PROCESSES IRG - This IRG will consider applications on biobehavioral and behavioral processes across the lifespan. Research on non-human animals as well as humans is included and both normal and disordered processes are addressed. While other study sections may be added to this IRG, the existing study sections created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/bss.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
   
   
10. IMMUNOLOGY IRG - This IRG will consider research applications ranging from basic through clinical studies focused on immunology and diseases that are principally immunological in their origin or manifestations. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: immunochemistry and immunogenetics; cellular, molecular and developmental immunology; immunodeficiency diseases; mucosal immunology; inflammation; innate immunology; immunological aspects of asthma and hypersensitivity diseases; vaccines; rheumatology; tumor immunology; chronic fatigue and related disorders; tolerance; autoimmunity and autoimmune rheumatological diseases; and transplanation immunology. Where immunological responses to pathogens or specific disease pathogens or diseases are secondary, proposals should be assigned to IRGs more appropriate for the specific condition.
    
    
11. INFECTIOUS DISEASES AND MICROBIOLOGY IRG - This IRG will consider research applications ranging from basic through clinical studies focused on infectious diseases and microbes. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: bacteriology and bacterial pathogenesis; virology and viral pathogenesis; mycology and fungal pathogenesis; parasitology and parasitic diseases; and vaccines. Studies of oral, surgical or other "subspecialty" infections should be reviewed in this IRG. Infections as triggers for chronic diseases, including cancers, overlap with other IRGs and might be reviewed elsewhere; examples include hepatitis viruses causing liver failure and cancer, as well as papilloma viruses causing cancer.
    
    
12. AIDS AND RELATED RESEARCH IRG - This IRG will consider research applications ranging from basic through clinical studies focused on all aspects of AIDS and related research. The existing study sections in this IRG, created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/irgdesc.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
    
    
13. ONCOLOGICAL SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on cancer. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: cancer genetics and cell biology, including metastasis; pathology; environmental and dietary carcinogenesis; drug development and evaluation; therapeutic oncology; and leukemias and lymphomas. Pathology is assumed to include immunopathology, diagnostic methods, and cytogenetics. Therapeutic oncology includes chemotherapy, hormonal therapy, combined modality therapies, radiotherapy, biological response modifiers, and chemoprevention. Tumor immunology and immunotherapy are represented within the immunology IRG. It is intended that applied clinical immunotherapeutics, including tumor vaccines, would be reviewed here.
    
    
14. HEMATOLOGY IRG - This IRG will consider research applications ranging from basic through clinical studies focused on blood cells and their diseases, and studies of the coagulation system and its pathology. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: stem cell biology and hematopoiesis (including cytokines); red blood cell biology and structure, including sickle cell anemia; leukocyte biology; bone marrow transplantation and transfusion; coagulation biochemistry and disorders; platelet biology and disorders; vascular biology; thrombosis; platelet function; and iron metabolism as pertinent to hematology. Thrombosis proposals primarily addressing platelet biology and pathology would be included here, but those with emphasis on other aspects of thrombosis would be referred to the Cardiovascular Sciences IRG.
    
    
15. CARDIOVASCULAR SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on THE heart and vasculature and cardiovascular diseases and their treatment. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: electrophysiology/arrhythmia; cardiovascular pharmacology; atherosclerosis/nutrition including lipid metabolism; vascular biology/thrombosis; cytokines/nitric oxide; angiogenesis; cardiac physiology; bioengineering, including devices; heart failure and transplantation; hypertension; cerebrovasculature; and peripheral vascular disease. Proposals focusing on lipid metabolism could be reviewed either here or within the Endocrinology, Metabolism, and Reproductive Sciences IRG. Proposals in the area of thrombosis are included here, but those with a primary focus on platelet function and biology would be more suited for the Hematology IRG. Similarly, proposals on bioengineering related specifically to devices for cardiovascular disease (stents, pacemakers, etc.) would be assigned here, but those involving more general aspects of bioengineering would be reviewed elsewhere. Likewise, transplantation as applied to the heart would be assigned here, but studies of fundamental transplant biology, even as relevant to the heart, could be reviewed elsewhere.
    
    
16. ENDOCRINOLOGY, METABOLISM, AND REPRODUCTIVE SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on endocrine and reproductive systems and associated diseases, as well as basic intermediary metabolism and metabolic disorders. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: fetal, maternal, and neonatal physiology; hormone physiology and biochemistry; endocrine function and disease; diabetes and diabetic complications; neuroendocrinology; reproduction, fertility and contraception; metabolism and metabolic diseases; nutrient metabolism, obesity and clinical nutrition; toxicology; and menopause. Diabetes affects many target organs. It is envisioned that studies of diabetes, pathophysiology, and control will be reviewed within this IRG. Those involving effects on other organs will be reviewed in other organ-specific IRGs if their focus is primarily on the function of the respective organs rather than on more general aspects of diabetes pathogenesis. Neuroendocrinology within this IRG will include those areas most closely allied with endocrinology, such as studies of the endocrine functions of the hypothalamic-pituitary axis. Other neuroendocrine studies relating to central nervous system (or brain) function or disease will be reviewed within appropriate neuroscience IRGs. Studies of metabolism included here span biochemistry of intermediary metabolism, cellular metabolism and physiology, and specific disorders involving these pathways.
    
    
17. BONE, MUSCLE, CONNECTIVE TISSUE, AND SKIN IRG - This IRG will consider research applications ranging from basic through clinical studies focused on connective tissues, which include bone and dental tissues, skeletal muscle, joints, skin, and the extracellular matrix, and their associated diseases. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: bone biology and disease; joint biology and disease; extracellular matrix components and biology, including cartilage/ligament/tendon biology and disease; craniofacial development and disorders; dental sciences; skeletal muscle biology and disease; skin biology and disease; sports medicine; exercise physiology; chiropractic medicine; rheumatological disorders; and biomineralization. Areas of potential overlap with other IRGs include infections, tumors, and surgical procedures related to dentistry, immunological features of dermatological disease, and aspects of orthopedics. The primary focus of the application should determine the choice between overlapping IRGs. This IRG also encompasses musculosketal function, including aging, biomechanics and biomaterials, and tissue engineering as they pertain specifically to the musculoskeletal system.
    
    
18. DIGESTIVE SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the entire gastrointestinal tract and related organs. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: gastroenterological pharmacology and physiology; esophageal, gastric, intestinal, and pancreatic diseases, including disorders of the gall bladder and biliary system; liver diseases, including liver transplantation; clinical nutrition/malnutrition (not including prevention); drug metabolism, pharmacology, and toxicology; and digestion, absorption, and nutrient transport. Liver diseases include the effects of alcohol and viral infections on the liver, and their treatment by transplantation and other modalities. Intestinal diseases include the entire spectrum from motility disorders to chronic inflammatory diseases, including the relevant immunology and pathogenesis studies. Nutrition proposals aimed at disease prevention would be referred to the Risk, Prevention and Health Behavior IRG. Those directed to this IRG relate to more traditional aspects of clinical nutrition (e.g. malabsorption, metabolic effects of malnutrition).
    
    
19. PULMONARY SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the lung and respiratory system and pulmonary diseases and their treatment. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: lung development; pulmonary physiology; chronic airway diseases, including asthma, cystic fibrosis, obstructive airways diseases, and interstitial lung diseases, including pulmonary fibrosis; environmental and occupational respiratory diseases (i.e. diseases of the lung); pulmonary toxicology; lung transplantation; adult respiratory distress syndrome; pulmonary vascular diseases; and sudden infant death syndrome. Applications dealing with asthma might be reviewed within the immunology IRG if the focus is on fundamental immunology. Proposals on cystic fibrosis might be reviewed within the Digestive Sciences IRG if the focus is on gastrointestinal manifestations or treatment of cystic fibrosis.
    
    
20. RENAL AND UROLOGICAL SCIENCES IRG - This IRG will consider research applications ranging from basic through clinical studies focused on the renal and genitourinary systems. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: renal transport mechanisms; renal blood flow; hypertension; inflammatory processes affecting the renal and genitourinary system; prostate and related disorders, including obstruction and inflammation; and clinical studies in nephrological and urological disorders, such as end-stage renal disease, incontinence, infection and urinary stones.
    
    
21. SURGERY, APPLIED IMAGING, AND APPLIED BIOENGINEERING IRG - This IRG will consider research applications focused on surgical and related disciplines, and on applied imaging and applied bioengineering. Some, but not all, of the topics that would be reviewed by study sections within this IRG that will be created in Phase 2 by members of the extramural community and NIH staff are: anesthesia; biomaterials; bioengineering and radiologic imaging (application of e.g., MRI, PET scan); burns and trauma; diagnostic bioimaging; surgery; radiation oncology; and drug delivery, bioavailability, and pharmacokinetics. Surgery includes most of surgical research except some transplantation research; the latter would be reviewed in the context of the chronic diseases for which transplantation is a major therapeutic modality. Surgical infections will, in the main, be reviewed in the Infectious Diseases and Microbiology IRG.
    
    
22. MOLECULAR, CELLULAR, AND DEVELOPMENTAL NEUROSCIENCE IRG - This IRG will consider research applications focused on the basic mechanisms that determine the structure and function of neurons, glia, and other excitable cells, and aspects of development in both the central and peripheral nervous system. While other study sections may be added to this IRG, the existing study sections created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/irgdesc.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
    
    
23. INTEGRATIVE, FUNCTIONAL, AND COGNITIVE NEUROSCIENCE IRG - This IRG will consider research applications focused on how the nervous system is organized and functions at an integrative, systems level. While other study sections may be added to this IRG, the existing study sections created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/irgdesc.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
    
    
24. BRAIN DISORDERS AND CLINICAL NEUROSCIENCE IRG - This IRG will consider research applications focused on disease and injury to the nervous system, including issues of neural substrate, functional consequences, and rehabilitation. While other study sections may be added to this IRG, the existing study sections created as part of the ADAMHA integration activity (see Appendix II) and listed at http://www.csr.nih.gov/review/irgdesc.htm, will remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years.
    
    
    
    

APPENDIX II
SOME RECENT CHANGES IN NIH REVIEW PROCESSES

1. Establishment of the Integrated Review Group (IRG) as the Functional Unit of Review

For decades, the individual study section was the functional unit of review in CSR (formerly DRG, the Division of Research Grants). However, CSR has recently adopted the IRG (a cluster of scientifically related study sections) as the functional unit of review. The IRG is an administrative unit including a number of study sections encompassing a broad scientific domain (analogous to an academic department). Leadership and management for the IRG are provided by a chief ("the department chair"), who coordinates the activities of several SRAs ("the faculty members"). The SRAs are in turn responsible for managing the activities of the component study sections within the IRG, each of which focuses on a specific area within the IRG's broad scientific purview. The IRG does not operate as an entity to review grant applications.

Establishment of the IRG as the functional unit of review has presented opportunities to optimize the review process. CSR has been working to capitalize on these opportunities, such as increased chances for teamwork, flexible distribution of applications, and sharing of reviewer expertise. Initial referral of applications is now made to the IRG, with subsequent assignment to individual study sections by IRG staff. In addition, some IRGs are arranging concurrent meetings of several or all of their study sections, so that reviewers with specific expertise can participate in multiple study sections as needed.

2. Periodic Review of Study Sections

The CSR Advisory Committee is developing recommendations for periodic review of study sections. Accordingly, study section members and applicants would be surveyed at frequent intervals to track study section performance, and the organizing principles and operating procedures for each study section would be assessed at approximately five-year intervals by external advisory groups as described below. Suggestions for changes and improvements as appropriate will be returned to the SRAs, Chairs, and members through the IRG Chiefs, and the results of the surveys will be monitored by the CSR Advisory Committee on an ongoing basis.

3. Establishment of IRG External Advisory Groups

For several years, CSR has been establishing external advisory groups for IRGs as ad hoc working groups of the CSR Advisory Committee to assess, approximately once every five years, the organizing principles and operating procedures for the study sections within each IRG. These external advisory groups, composed of extramural scientists, are asked to evaluate the appropriateness of research topics and scope of applications reviewed; the evolution of topics and scope of research; how well newly emerging research areas are being incorporated; the pool of reviewers and, in addition, the performance of SRAs, Chairs and members.

To date, such groups have been or are in the process of being instituted for eight of the 20 current IRGs: Cell Development and Function; Oncological Sciences; Biophysical and Chemical Sciences; Musculoskeletal and Dental; Cardiovascular Sciences; Brain Disorders and Clinical Neuroscience; Molecular, Cellular, and Developmental Neuroscience; and Integrative, Functional, and Cognitive Neurosciences.

The recommendations of these and additional external advisory groups will be provided to the expert groups responsible for Phase 2 of the Scientific Boundaries Panel activity, whose goal is to design study sections de novo within new IRGs that are recommended in this report.

4. ADAMHA Integration

To complete the merger of the Institutes of the Alcohol, Drug Abuse, and Mental Health Administration into the NIH that began in 1992, review activities of the National Institutes on Alcoholism and Alcohol Abuse, Drug Abuse, and Mental Health have now been integrated into the CSR. Members of the scientific community, CSR personnel, and Institute staff developed 21 new neuroscience study sections that are grouped in three IRGs, 16 new behavioral and social sciences study sections that are grouped in three IRGs, and eight new study sections within the AIDS and Related Research IRG. (See http://www.csr.nih.gov/review/reorgact.asp.) The reorganization accomplished by this team has been incorporated into the larger organizational framework proposed by our Panel as described in Section II. Our Panel intends for the study sections that were created in these recently reformulated IRGs to be left intact, pending evaluation of the effectiveness of their organization after they have been functioning for several years. Such an evaluation process is currently underway for the recently reorganized neuroscience IRGs. Our Panel recommends that the process these teams followed for forming these study sections be emulated in Phase 2 of our work, and furthermore that the process that is being developed to evaluate the effectiveness of the reorganization be extended to all subsequently reorganized IRGs.

5. Increasing the Quality of Reviewers and Broadening Their Participation

The CSR is exploring flexible ways to overcome the present obstacles to reviewer recruitment. The Center is testing the effectiveness of having several members share a single appointment and will evaluate the use of senior scientists to augment technical expertise with broad perspective. It will also assess ways to overcome the reluctance of researchers to serve on study sections because they perceive that their funding may be jeopardized by the requirement that their grant be reviewed in another study section or by a special emphasis panel. CSR is also working to expand the reviewer pool and improve the nomination process. The SRAs have been asked to broaden their nets when identifying new study section members, and they now provide more extensive information to the CSR Director about the source and rationale for specific reviewer nominations.

6. Research Communities That Feel Disadvantaged

In response to extensive outreach activities, three groups were highlighted for attention: (1) clinical researchers, (2) behavioral and social sciences researchers, and (3) bioengineers and developers of technology and instrumentation.

To address long-standing concerns regarding the review of clinical research, in 1997 CSR engaged an expert to act as liaison to the clinical research community (http://www.csr.nih.gov/events/research.htm). Previous analyses indicate that clinical researchers are disadvantaged when their applications are reviewed in study sections that had been assigned only a small number or proportion of clinical proposals. (A small proportion has been arbitrarily defined as less than 30% of the grant portfolio of the study section.) To solve this problem for about 50% of the clinical applications, the CSR has clustered clinical oncological sciences applications in one special emphasis panel and clinical cardiovascular proposals in another.

A second expert was retained to serve as liaison to the behavioral and social sciences research community, which has a long-standing concern about how fully and successfully the subject matter of this research has been integrated into the NIH mission. Serving as a consultant for the ADAMHA integration activities related to behavioral and social sciences research, his recommendations were incorporated in designing relevant study sections.

A special Working Group on Review of Bioengineering and Technology and Instrumentation Development Research was created to identify the obstacles to fair, high-quality, rigorous review and develop a set of principles to guide CSR in establishing a review infrastructure that will fairly evaluate interdisciplinary research. This committee's report (http://www.csr.nih.gov/bioopp1/select.htm) notes the need to develop a flexible organizational structure, to revise operating principles and practices, to promote system agility to adapt to the continually changing scientific landscape, and to declare the importance of multiple types of research. Our Panel has incorporated the relevant recommendations of this working group into our broader activity to ensure that the changes proposed by both groups can be implemented in a coordinated fashion.

7. Review of Fellowship Applications

To determine optimal ways to identify the most promising candidates for training support, an outside expert was engaged to study the variable venues and practices currently used to review fellowship applications within CSR. The resulting recommendation is to establish dedicated fellowship study sections to review the majority of these applications. Some pilot studies will be undertaken during the next year.

8. Training for SRAs

To ensure that CSR staff consistently apply uniform principles in managing the review process, a group of senior SRAs initiated a training program for their colleagues. New SRAs are presented with an extensive overview of policies and procedures, and all SRAs learn of changes to the system and are trained in specific aspects of peer review through a program of continuing education.

9. Development of Guidelines for Chairs and Members Of Study Sections

The CSR Advisory Committee has developed a set of guidelines that outlines "best practices" for Study Section Chairs. The document outlines the need for Chairs to (i) become familiar with all the applications to be considered; (ii) consult with the SRA in assigning applications to reviewers; (iii) focus discussions on key issues; (iv) maintain fairness of the review process; and (v) promote consistent criteria-based scoring. Chairs are also encouraged to assist SRAs in identifying and assessing candidate reviewers and, where appropriate, to make initial contacts with candidates; to help train members; to help develop a successor; and to invite performance assessments of all participants, including that of the Chair, in the process. A parallel document to outline "best practices' for study section members is also being drafted.

10. Review Criteria

Reviewers are now instructed to address each of five review criteria (significance, approach, innovation, investigator, and environment) and assign a single global priority score rating for each scored application (http://www.nih.gov/grants/guide/1997/97.06.27/notice-review-criter9.html). The score should reflect the reviewer's judgment of the likelihood that the proposed research will have a substantial impact on advancing our understanding of biological systems, improving the control of disease, and/or enhancing health. This NIH requirement to evaluate five criteria established in 1997, is intended to overcome an inappropriate focus on technical feasibility and provides an opportunity to rate all types of research equitably in terms of their potential impact. Innovation has been specifically included among the criteria to combat the present excess conservatism in the review system. The focus on the investigator is meant to take into account the skill of the scientist, based on his or her past accomplishments and/or appropriateness of training.

[¹] Health-related research encompasses the entire portfolio of research that is currently supported by NIH, including clinical research, studies in the behavioral and social sciences, and those conducted by members of the allied disciplines (chemistry, physics, mathematics, engineering, and computer science). These fields underlie our developing understanding of basic biology and medicine, and also contribute independently to the mission of the NIH.

[²]Approximately 40,000 applications are subjected to the peer review process each year. Approximately one-fourth of these applications are evaluated in scientific review groups (SRGs) managed by the Institute to which the application is assigned for potential funding. In general, these are applications for which optimal review requires greater programmatic context (for example, Center (P50) and institutional training (T32) grant applications). Three-fourths of all applications (primarily individual investigator-initiated (R01s), fellowship (F32s) and Small Business Innovation Research Program (R43, R44) applications) are evaluated in the Center for Scientific Review (CSR), which is one of NIH’s 25 Institutes and Centers (ICs) and whose mission is to provide this service for all NIH extramural programs.

[³]While these impressions were not based on objective data, it is incumbent upon CSR to respond to uniformly voiced concerns of the community, and to ensure that proposed changes are endorsed by the community before implementation.

[⁴] The generic, functional term for any group engaged in scientific and technical peer review is Scientific Review Group (SRG). SRGs are commonly called study sections in CSR. They may either be individually chartered or part of an Integrated Review Group (IRG), which is a cluster of Scientific Review Groups chartered as a single entity with a related integrated scientific focus. A Special Emphasis Panel (SEP) is a scientific review group whose membership is fluid, with members designated to serve for individual meetings rather than for fixed terms of service.SEPs are constituted by each IRG to evaluate applications that do not fit well in one of the component study sections.In this report, the term study section, while not always technically correct, is invoked liberally to match its use by members of the extramural scientific community. In addition, IRGs, while officially specified as Initial Review Groups and still designated as such when operated by the Institutes, are herein referred to as Integrated Review Groups in CSR to better describe their function.

[⁵] For a complete explanation of how the IRG functions and the distinction between IRGs and study sections, please see Appendix II. See Appendix II also for an expanded explanation of how the external advisory groups function and the distinction between their efforts and those of the expert groups that will design study sections in Phase 2.

[⁶] The opportunity to comment on the report was advertised widely, and approximately 125 societies were asked to notify their members of the opportunity to view the report on the CSR web site and to submit comments via the electronic form provided at the site. The full set of comments, along with summaries prepared by CSR staff at the request of the Chair, were provided to the Panel members in advance of their meeting on November 8-9, 1999. During this meeting, at which all but one member was in attendance, the Panel discussed an extensive list of agreed upon issues. Subsequently, the Panel conducted a series of conference calls with experts in specific areas for which consultation was required to gain the further input needed to refine the recommendations in this report.

[⁷] The creation or elimination of an IRG does not affect the funding allocated to any given area of science. IRGs do not set award levels. Rather, Congress appropriates funds to the individual Institutes and Centers. Each of these Institutes addresses a specific health-related mission focused either on diseases (e.g., the National Cancer Institute, the National Institute for Arthritis, Musculoskeletal and Skin Diseases); organ systems (e.g., the National Heart, Lung, and Blood Institute, the National Eye Institute); populations (e.g., the National Institute on Aging, the National Institute of Child Health and Human Development); or cross-cutting issues and infrastructure (e.g., the National Institute of General Medical Sciences, the National Center for Research Resources).

[⁸] The Panel will consult relevant societies and NIH staff to identify appropriate members of the expert groups that will be formed to design study sections. Many who wrote thoughtful comments related to the Phase 1 report also will be called upon.

[⁹] We would be remiss if we failed to point out that our recommendations to operate and continually refine a more complex and flexible system will place a considerably greater burden on the CSR staff.  Therefore, as needed, the CSR must be provided with the additional resources that will allow its staff to effectively manage and improve a system of peer review that lies at the heart of the enormous success of the U.S. scientific enterprise.

[¹⁰] The area of neuroscience is an exception.The fact that the three IRGs created as a result of a reorganization in 1998 are listed in no way implies assignment of greater value to this area of science. Other areas may also require more than one IRG.