![[Up Arrow]](../images/uparr.gif){kind=small}
Return to Top
II. Report of the Acting Director
Dr. Donald Luecke
III. View from the Office of Extramural Research
Dr. Wendy Baldwin
IV. NIH and DRG Budgets
Mr. Fred Wong
V. NIH Reinvention Update
Dr. Anthony Demsey
VI. The Initial Review Group Concept
Dr. Donald Luecke
VII. Overview of the Pilot Study of Initial Review Groups
Dr. Donna Dean
Dr. Raymond Bahor
VIII. Fellowship Reviews as a Model of Initial Review Group
Flexibility
Dr. Jerry Roberts
Dr. Everett Sinnett
IX. Joint Workshops and Study Section Meetings
Dr. Sooja Kim
Dr. Fayez Gishan
Dr. Paul Strudler
X. Anticipating and Responding to Change in Science
Dr. Michael Knecht
Dr. Gerald Greenhouse
XI. Outreach to the Scientific Community
Dr. Gerald Greenhouse
Dr. Anita Weinblatt
XII. Developing Closer Links with NIH Institutes and Centers
Dr. Anita Weinblatt
Dr. Paul Strudler
Dr. Stephen Rose
XIII. Report of the Subcommittee on Rating Grant Applications
Dr. Walter Stolz
Dr. Hugh Stamper
Assigned Discussants
Dr. Nesselroade
Dr. Roger Bakeman
XIV. Cooperative Review Effort between the National Institute on Alcohol
Abuse and Alcoholism (NIAAA) and DRG
Dr. Jeanne Ketley
Dr. Faye Calhoun
Dr. Kenneth Warren
XVI. 1996: 50 Years of NIH Peer Review
Dr. Donald Luecke
Dr. Suzanne Fisher
Dr. John Sherman
XVII. The Next Step for DRG and the DRG Advisory
Committee
Dr. Donald Luecke
XVIII. Closing Remarks
Dr. Thomas Braciale
DRG Advisory Committee Roster (November 1995)
Dr. Thomas Braciale, Chairperson of the DRG Advisory Committee, called to order the 13th meeting of the Committee, thanking the ad hoc advisors for joining the Committee for the meeting. The first item of business was to approve the minutes of the May meeting. With the exception of a correction in the number of applications received by DRG in the last year, the minutes of the May meeting were unanimously approved.
II. Report of the Acting Director
Dr. Donald Luecke
Dr. Luecke explained the procedures for the impending Government shutdown. He then noted that the meeting was a historic event, the first opportunity to meet in the new Rockledge Building.
Next, Dr. Luecke welcomed eight ad hoc advisors: (1) Dr. Keith Yamamoto, Chairperson of the Department of Pharmacology at the University of California at San Francisco; (2) Dr. John Nesselroade, Acting Chairperson for the Department of Psychology at the University of Virginia; (3) Dr. Bruce Stillman, Director of the Cold Spring Harbor Laboratory; (4) Dr. David Center, Professor and Chief of Pulmonary and Critical Care Medicine at the Boston University School of Medicine; (5) Dr. Howard Schachman, ombudsman at NIH working in the Office of the Director; (6) Dr. Richard Hynes, Professor of Biology and Director of the Center for Cancer Research at MIT; (7) Dr. Eveline Schneeberger, Professor in the Department of Pathology at the Mass General Hospital in Boston; and (8) Dr. Fayez Ghishan, Professor and Chairperson of the Department of Pediatrics at the University of Arizona Medical Center in Tucson.
With respect to the search for a new Director for DRG, the application period closed November 1st. The selection will probably be some time in the spring. The Co-chairs for the search committee are Dr. Wendy Baldwin, Deputy Director for Extramural Research, Office of the Director, and Dr. Claude Lenfant, Director of the National Heart, Lung, and Blood Institute.
Dr. Luecke then summarized the developments related to the Cassman Committee report, which had been presented at the May meeting of the DRG Advisory Committee. The Peer Review Oversight Group (PROG) has a charter signed by Dr. Varmus. PROG will have 16 members, 10 representatives from the extramural scientific community, 6 from within NIH. One of those 6 will be the Director of DRG. Dr. Baldwin will serve as the Chairperson for PROG.
Drs. Luecke and Baldwin agreed on the staff who would be leaving DRG to join the Office of Extramural Research. DRG will be losing 89 people, 79 from the Information Systems Branch and the remaining 10 from various other offices. All of these people will remain in the new Rockledge building, either in vacant space on the sixth floor or in former DRG quarters.
III. View from the Office of Extramural Research
Dr. Wendy Baldwin
Dr. Baldwin explained that NIH is working under a continuing resolution, which is $200 million below the NIH FY 94 level of funding; so it has been a very difficult time. If there is a new continuing resolution, it would likely be at the FY 95 level, which would be a help. NIH continuing grantees have received a letter from Dr. Baldwin advising them of the budget situation and explaining that, unless there were some dramatic change, NIH would not be able to support the normal four percent escalation in future years. For new grantees, the plan is to take reductions at the outset, so that investigators will have a sense of stability and know funding levels over the lifetime of the grant. However, every Institute is going to have to look at its own available funds, and the specifics will vary from Institute to Institute.
Dr. Baldwin then commented on a recent report by a committee looking at the relationship of the NIH with the Howard Hughes Medical Institute (HHMI). First, the policy of excluding HHMI investigators from the room when other HHMI investigators are being reviewed, done initially at the request of HHMI, has been eliminated. As long as there is not a conflict of interest, they can now remain in the meeting room. This will effectively give NIH a larger pool of reviewers. The second feature is the need for NIH reviewers to focus continually on scientific issues. Other issues that reviewers may have knowledge or opinions about should be placed in an administrative note, and not be part of the scientific critique. Finally, NIH should return to the grant-in-aid approach to partially support research that may also be partly supported by other sources.
In response to a question about whether the features were unique to the Howard Hughes Institute, Dr. Baldwin noted that such features were not specific to, but highlighted in, NIH's experiences with Howard Hughes. These are general issues and would have general solutions.
What about the issue of overlap between Howard Hughes and NIH support? For some investigators it is quite easy to see the distinction between the Hughes-supported and NIH-supported research, but in other circumstances it blurs dramatically. Dr. Baldwin noted that NIH, in the past, always had situations where an overall activity is not a severable NIH activity. Basically, this is a co-funding strategy, and Dr. Baldwin noted that it must be handled carefully or NIH will create a new set of problems.
With respect to the PROG Committee, Dr. Baldwin noted that the majority of the membership will come from the scientific community. PROG should be in place by about March 1996.
The decision on the new DRG Director could be made as early as January, although it will likely be in the spring. When the individual could assume the position is uncertain.
IV. NIH and DRG Budgets
Mr. Fred Wong
Mr. Fred Wong, DRG Budget Officer, noted that the current budget disagreement and potential shutdown impact on the annual appropriations. The FY 1995 estimate was $11.3 billion. The FY 1996 President's Budget was $11.8 billion, a four percent increase over the FY 95 estimate. After Congress reviewed the President's Budget, the House report was $11.9 billion, a 5.7 percent increase over FY 95. The Senate's proposal, $11.6 billion, represented a 2.7 percent increase over the FY 95 estimate. The DRG budget has been about $40 million.
V. NIH Reinvention Update: Present and Future Activities
Dr. Anthony Demsey
Dr. Demsey, Acting Deputy Director, noted that reinvention information is available to the public on the Worldwide Web. Unfortunately it has been revised infrequently. A number of activities underway at NIH can be classified under the broad umbrella of reinvention. The transfer of DRG staff from the Information Systems Branch to the Office of Extramural Research is related to the overall electronic research administration effort at NIH. The project has received a much greater degree of prominence because of the move to centralize activities within the Office of the Director of NIH.
Less global, but of equal interest, are endeavors such as the "Just in Time" submission of application data. The concept is fairly straightforward. Some of the information that applicants submit with their applications is not necessary for the review of an application but only for the award of a grant. So why not ask for this information at the time that NIH plans to fund the approximately 25 percent of the reviewed applications?
Another concept is called the modular grant, asking for a block of money, a set amount of dollars. Mechanisms that have restrictive budgets and so lend themselves to this idea are Small Grants (RO3s), FIRST awards (R29s), Phase 1 Small Business grants (STTRs and SBIRs, R41, R43), and some career development awards (K series).
A number of other activities are generally related to study section operations. For example, in the rating of grant applications, the current single score could become multiple scores based on various criteria. A second interrelated activity is the National Institute of General Medical Sciences experiment on rounding off priority scores and percentiles. Instead of a 187, the score would be 190. Then the calculation of percentiles is further rounded off. Essentially the Institute is trying to create a large number of tie scores in an attempt to reflect the fact that there are not necessarily significant differences between a 187 and a 192 priority score or a 23rd and a 21.7 percentile.
Another committee is working on the consolidation of granting mechanisms. The proliferation of mechanisms is confusing to the scientific community. Another issue is who can vote at study section meetings. The NIH Reviewers Reserve currently allows ad hoc reviewers to vote. A fresh look is being given to whether that practice should be to continued or whether all individuals, ad hoc or not, could vote. Also, as Dr. Baldwin explained, Howard Hughes investigators no longer will be considered automatically in conflict during the review of another Hughes investigator simply on the basis of their employment with the Howard Hughes Medical Institute.
A final issue is limiting the number of amendments. Possibly that would provide less ownership by the study section, as well as more flexibility in scoring the application. An additional concern is for a more rapid turnaround of amended applications.
Discussion
In response to a question about whether study sections should suspend their mentoring role in the grant application process (for example, amended applications being continually fine tuned by reviewers), Dr. Demsey noted this philosophical debate has been going on for some time. That reflects the concern about study sections continuing to provide input round after round, which then turns the process more into a tutorial than it should be.
One concern was raised that eliminating or reducing mentoring might be acceptable to a seasoned, experienced individual, but it may be a hardship with young investigators. However, it might improve some inappropriate practices of universities and departments where grant applications are tied to salary increments.
One discussant agreed that NIH cannot write rules for such a diverse population, but noted that something has to be done to change the concept that summary statements are supposed to be tutorials for everybody. Summary Statements need to be concise, to deal with the important issues, instead of including all the strong and weak points on every grant application. Such excruciating details on summary statements of excellent applications are a burden on the study sections. Certain amendments ought to be handled at the Institute level without a resubmission, because the application is clearly fundable; others should have been turned down and the applicant free from the illusion that amending the application may lead to an award.
One person noted that with senior members on study sections, there is a mentoring role for young investigators. Often a study section can, in the first round, help good investigators focus their research. The investigator then comes back with a superb application, and most importantly, goes on to do research that is universally rated as outstanding.
VI. The Initial Review Group Concept
Dr. Donald Luecke
Dr. Luecke reminded the Committee that reorganization of study sections started as a result of an executive order in 1993 by President Clinton to reduce the number of public advisory groups. In response, DRG went from 85 chartered groups or study sections, with 105 different groups, to 18 initial review groups (IRGs). Each IRG has a charter. The study sections are not chartered, but exist as subcommittees within an IRG. Dr. Luecke then noted two important topics: (1) occasionally DRG has not been as interactive with Institute staff as they should be; and (2) DRG needs to engage the community in a continuous process of looking at the way the Division is constituted, the way in which the advances in science may be reflected in the different kinds of applications, and how to better accommodate those changes in the study sections.
VII. Overview of the Pilot Study of Initial Review
Groups
Dr. Donna Dean
Dr. Raymond Bahor
Dr. Donna Dean, Acting chief of the DRG Referral and Review Branch, explained that the goal of the Pilot Independent IRG was to work towards a more flexible peer review structure and operative environment in DRG. Thus, the Division could be more immediately responsive to changes in emerging new areas of science, while still keeping the primary focus on the intrinsic high quality peer review that has been the Division's strength over the last 50 years.
Since the first of June, 8 of the 18 IRGs have been operating in this rather independent capacity; the other 10 IRGs have been operating in the traditional review section structure within the Division. The Division has looked at restructuring not only structurally but also functionally, examining issues of breadth and focus. How does one periodically reevaluate current practices? (Certainly this also is a valuable function of the Advisory Committee.) How does the Division form new committees to cover areas? How does DRG handle cross-fertilization across fields of science? How can members be used where their expertise is most needed? Throughout, the Division should create an environment of increased communication among the IRG, NIH staff, and the scientific community.
After briefly summarizing the organization and staff of the Pilot Independent IRG structure (Figure 1a and Figure1b), Dr. Dean ended with four quotes, which summarized the unavoidable effects of organizational changes: "I was gratified to be able to answer promptly. I said I don't know." (Mark Twain); "There 's no off season any more." (Nolan Ryan); "If you come to a fork in the road, take it." (Yogi Bera); and "You miss 100 percent of the shots you never take." (Wayne Gretsky).
Dr. Raymond Bahor, Chief of the Pilot Independent IRG, indicated that this project was really an empowerment of the SRAs. They know the available resources and are best able to make the decisions to use those resources most effectively. He regarded his position as making administrative decisions or facilitating the work of the SRAs who are involved in review.
VIII. Fellowship Reviews as a Model of Initial Review
Group Flexibility
Dr. Jerry Roberts
Dr. Everett Sinnett
Dr. Roberts, SRA of the Mammalian Genetics Study Section, spoke about the review of fellowship applications in the Genetic Sciences Initial Review Group. Their Biological Sciences 1 Study Section had reviewed only fellowship applications, covering a broad area of science, from clinical studies to Drosophila and yeast genetics. With the move to disband fellowship study sections and move fellowship applications into RO1 committees, this was a chance to move some RO1s into Biological Sciences 1 and experiment with the review of these applications using a broader committee.
To equalize the workload, fellowship applications were moved from the Biological Sciences 1 Study Section to three RO1 study sections. A problem with the mix of RO1 and fellowship applications arose because fellowships are reviewed together since the review criteria are different. However, some Institute program staff, who needed to be present during the review of their RO1 applications, had to return for the review of fellowship applications. This was a logistical problem, which pointed to the need for understanding and accommodation on both sides.
Discussion
In the study sections where fellowships were included with research grants, were the fellowships counted in the percentile of the total? Dr. Luecke responded that only RO1s and R29s (FIRST applications) are included in the base for percentiling. RO3s (Small Grants), and R15s (Academic Research Enhancement Awards-AREAs) are also not currently included in the percentiling base. A second comment pertained to the potential advantage with this new structure in that the applications with broad-based approaches could more easily find "outside reviewers" from within other parts of the same IRG rather than having to get a mail review from someone outside the process.
Some SRAs noted that when related study sections met at the same time and in nearby locations, the sharing of reviewers worked out pretty well. Several members cautioned that while it is desirable to broaden the scope of review groups, there is also value in having a stable membership component. Both functions must be considered in the initial review process.
Dr. Sinnett, SRA of the Respiratory and Applied Physiology Study Section, who was the next assigned discussant, spoke about their recent experience reviewing fellowship applications. Altogether there were only 14 fellowships for the whole IRG. As to the fairness and the appropriateness of the review, several issues deserve comment. One issue was the lack of experience of their study sections with fellowship. Therefore, information on fellowships was sent to the reviewers in the pre-meeting mailings. In addition, they made sure that reviewers with previous fellowship review experience attended the meeting.
There seemed to be little problem for the reviewers in terms of incorporating additional criteria in the reviews and discussion. In terms of scheduling, they were requested to review all the fellowships at the start of the second day, and with the small numbers, they were able to comply with that request. But Dr. Sinnett could foresee future problems of not only Institute program staff availability but also reviewer availability in reviewing fellowships as a group.
In the discussion, one SRA explained that their IRG decided to cluster their 40 fellowship applications into a special emphasis panel. The special emphasis panel worked very well, and the Institute program representative wrote commendatory letters to the Division.
IX. Joint Workshops and Study Section Meetings
Dr. Sooja Kim
Dr. Fayez Ghishan
Dr. Paul Strudler
Dr. Kim, SRA of the Nutrition Study Section, reported on a Nutritional and Metabolic Sciences IRG Workshop on Molecular Biology, which was held on October 23, 1995. The workshop, which involved the membership of the three IRG study sections, included both scientific sessions and also a discussion about the NIH peer review process.
Dr. Ghishan, a participant in this workshop, then summarized the proceedings of this successful workshop on the new tools of molecular biology. The peer review system discussion covered several areas. With respect to scoring, the consensus was that a system using scores for different criteria would not be ideal, and that the application should continue to be reviewed and scored as a whole.
Concerning R29 (FIRST) and RO1 applications, the attendees believed that no need existed for separate review systems, and these applications should be reviewed together in the same study section. They also felt that new and continuation grant applications should not be reviewed separately by different review groups. Clinical and basic research should also continue to be reviewed together; however, reviewers who are more familiar with clinical studies should be members of the study section or ad hoc reviewers.
Given the continuous changes in science and new and emerging scientific areas, how can a study section deal with these areas most effectively? The attendees recommended periodic workshops to bring these newer areas of science to the members of the study section. The addition of ad hoc members with specific expertise would also be useful.
The workshop attendees were concerned about the triage system. In some universities, if an investigator receives a non-competitive score, then their funding is cut. However, if the investigators receive a score better than the 20th percentile, the universities can provide them with bridge funding through the next cycle. In their opinion, the triage system should be modified to identify non-competitive but very good applications, so that investigators who may just miss the cut off will not have their funding cut.
Can the review process be further streamlined? For example, should a limit be placed on the number of times an application can be resubmitted? The consensus was that there should be a limit. Should an application that received an outstanding score but just missed being funded have to be rereviewed during the next grant review cycle? They believed it should not be rereviewed.
Dr. Paul Strudler, SRA of the Radiation Study Section, who was the next presenter, noted that a central theme in recent workshops and peer review studies is isolationism--the notion that DRG is isolated from the Institutes, and that study sections are isolated from the science. Dr. Strudler discussed study section interactions and the opportunities for SRAs to be current in their scientific areas.
This winter, the Radiation and Chemical Pathology Study Sections* were scheduled to meet jointly at the Keystone Resort for one day in conjunction with the AACR Symposium on Cancer Susceptibility Genes and Molecular Carcinogenesis. The Chemical Pathology Study Section reviews applications where the emphasis is on the various chemical and physical aspects of pathology; the Radiation Study Section is more involved with the physical aspects of pathology. Pragmatically, the two study sections overlap in the areas of free radical chemistry, DNA repair, and UV induced mutagenesis. The workshop will be largely devoted to free radicals and oxidative damage, matters of considerable concern to both study sections. Both study sections have a history of publishing their workshop reports, and this one will undoubtedly also be published. * Note: Due to the subsequent Government furlough, it was necessary to reschedule these meetings.
The DRG management is working very hard under budgetary constraints to meet the desires of the community that the SRAs attend more professional meetings to remain current with the science and to interact with scientists. But there is only so much that can be done, and Dr. Strudler and other SRAs volunteer their time and monies to attend scientific meetings. For example, in October 1995, with DRG paying only the registration fees, Dr. Strudler attended scientific meetings in Houston and in Baltimore of prime interest to his scientific work, and gave presentations on peer review. In this issue of money versus travel, the difficulties of dealing with the desires of the community are felt by SRAs trying to meet these obligations.
Discussion
Was Dr. Strudler implying that the SRAs would like to become more involved in giving both positive and negative feedback to applicants after they receive their reviews? Dr. Strudler responded that he was referring to attending meetings and conferences and making himself available so that people wouldn't feel frightened to pick up the phone and call. He indicated his personal belief that it would be expeditious for an SRA to provide direct feedback to applicants as quickly and as early post-review as possible. However, at present, SRAs are not to have contact with applicants post-review; such contact is to be done by the Institute program administrators. This works well when there is good communication between review and program staff.
X. Anticipating and Responding to Changes
in Science
Dr. Michael Knecht
Dr. Gerald
Greenhouse
Dr. Knecht, SRA of the Biochemical Endocrinology Study Section and Coordinator of the Endocrinology and Reproductive Sciences IRG, explained that a major future consideration is to meet with scientists, the heads of various endocrine-related societies, Institute program staff, and SRAs to review the field of endocrinology to see where the field is headed and how the IRG should evolve. For example, the numbers of applications in thyroid, ovarian, and neuroendocrine research areas have decreased recently, while applications in other areas, including insulin, GI hormones, and endocrine cancers, have risen. Study section membership and guidelines need to meet these changing challenges.
Dr. Gerald Greenhouse, SRA of the Cell Biology-1 Study Section and Coordinator of the Cell Development and Function IRG, noted that the Acting Director of DRG is allowing and indeed encouraging the SRAs and coordinators to make substantial decisions at the basic level of the study section.
The Cell Development and Function IRG, has reviewed cell and developmental biology for some time. Dr. Greenhouse has been Referral Officer for the group since 1982. The component study sections of the IRG have a long history of interacting together with a certain amount of continuity. Because of a reasonable amount of overlap among study sections, every application assigned to the IRG could be assigned to at least two study sections. They have also exchanged members, in the past, although not every round and not too often. When they exchanged members the members generally stayed for the entire meeting. For example, with the addition of fellowship applications, the CTY study section became overloaded, so they moved a subset of applications involving cytoskeleton studies to another study section within the IRG. Another example was at a recent Gordon Conference on biological Structure and Gene Expression, where he spoke with researchers interested in how structure affects gene regulation. They did not feel represented well enough on the study sections. After talking more to the community, the Division recruited some additional members to existing study sections.
Discussion
Comments were made by several Advisory Committee members that applications reviewed by study sections are what science is going to be. In contrast, at major meetings, the lectures often are about where science has been. The SRA should certainly go to such meetings, but should also have broader interactions with members of the scientific community. Institutes also push their agenda in certain ways by issuing Requests for Applications (RFAs), but some of those are not necessarily the best way to do science. Dr. Luecke responded that for many reasons, some of which relate to science, the number of RFAs issued by many Institutes will be reduced. One of the real problems that resulted from the widespread, indiscriminate use of RFAs, many of them searching for investigator-initiated ideas, is that applicants are not successful in their first review in an ad hoc setting, and DRG is then faced with reviewing the amended applications in study sections.
XI. Outreach to the Scientific Community
Dr. Gerald Greenhouse
Dr. Anita Weinblatt
Dr. Greenhouse noted that in conjunction with the American Society of Cell Biology, they have organized sessions for attendees to meet the SRAs in an informal setting. The main objective is for people to understand that SRAs are approachable and can be contacted. SRAs consider such discussions to be an enjoyable part of their job.
Dr. Weinblatt, SRA of the Immunological Sciences Study Section and Coordinator of the Immunological Sciences IRG, then spoke about her interactions with the American Association of Immunologists. Dr. Weinblatt had two face-to-face meetings with the Executive Director on issues of mutual concern, and in addition, she was invited to address the Executive Committee and hear their concerns. Recently, she used the study section as a mini-advisory committee. At the end of the study section meeting, the floor was open to hear any concerns of the study section members. That was a very useful exercise, for it gave them an opportunity to give Dr. Weinblatt some feedback about various issues.
Discussion
In response to a question about whether these groups included colleagues from the Institutes, Dr. Weinblatt stated that they have not yet been included, but felt it would be a good idea.
The next question was that having had a high level of success, does DRG plan to make more explicit and directed contact with the scientific societies? Dr. Weinblatt noted that for a combined meeting between the American Association of Immunologists and the American Association of Asthma and Allergic Diseases, several study sections are discussing a joint workshop.
The request was made for data on the number of individuals who have had continuous NIH funding for a period of years but who have not sat on DRG study section, as well as data on those individuals who have and in what capacity for the last 10 or 15 years. It was further suggested that the database include serving on review panels outside the NIH. Such service is becoming an increasing burden for many investigators, and should be taken into account.
XII. Developing Closer Links with NIH Institutes
and Centers
Dr. Anita Weinblatt
Dr. Paul Strudler
Dr. Stephen Rose
Dr. Weinblatt spoke about recent activities in the Immunological Sciences IRG that started at the beginning of June 1995. The SRAs in their IRG met with the program staff from the National Institute of Allergy and Infectious Diseases that they interact with--the Division of Allergy, Immunology and Transplantation. They invited the program staff to Rockledge II to see DRG's new quarters and also to discuss some issues. They discussed the Pilot Independent IRG experiment and the composition of their IRG, especially the possible restructuring of the former fellowship study sections.
Their goals were to use Institute program staff as a source of advice and also to explore ways to work better together. From that meeting came their plan to do an in-depth analysis of each study section to see if there are ways of restructuring that would help them review the science better. The flexibility of the IRG will, she hoped, allow SRAs to be more flexible in their thinking about program and review interactions.
Dr. Strudler noted that the stakes are high, for this involves careers and people's lives. The level of shrillness and paranoia will perhaps continue increasing despite the increasing efforts of DRG to interface with the community. The SRAs in Dr. Strudler's IRG met separately both with the program administrators from the National Cancer Institute (NCI) and also with the SRAs in the NCI.
The main accomplishment from the meetings was dealing with the mistrust; but they also discussed that especially for program announcements, it would be valuable to involve the Division's SRAs in the proposed subject matter. Also, traditionally, the Institute/Center review branch selection of site visitors occurs without interaction with DRG, and the Division's SRA's offered their assistance. In all, the prime result of these meetings with NCI was to break down the barriers, so that they will use each other as cooperative resources.
Dr. Stephen Rose, Chief, Genetics and Transplantation Branch, National Institute of Allergy and Infectious Diseases, noted their interactions with the SRAs of the DRG immunology study sections have always been phenomenal. The SRAs have been professional and highly competent and have treated program staff with respect, and vice versa. Their Institute has held several focus groups with the scientific community, and a major issue is that feedback from Institute program staff on review must be timely and blunt. Investigators are getting considerable pressure from their organizations; they can not go unfunded as long as they could in the past. The safety nets are basically disappearing.
The focus groups also unfortunately reinforced something that they had heard frequently: people still view NIH as a black box. They send in an application and then something comes back. For example, even senior grantees complain that "DRG" did not fund their grant. One of the topics they have been trying to deal with internally is how Institute program staff attending a study section meeting should react if they believe that something is missed in the review. Granted, their role is not to participate in the review. But if something is missed by the reviewers and the program staff notice it, how can this be brought to the attention of the study section? One possibility, suggested by an SRA was to write a note, and slip it to the SRA. If an application page number is available, include it in the note.
Discussion
Does electronic transmission technology help the process of getting the critiques out to the investigators? Dr. Weinblatt responded that at the present time, this does not speed up the procedures. Streamlining has been more helpful in expediting the preparation of summary statements.
Another question was whether Institute program staff not attending study section meetings was a major problem. Are program staff required or just encouraged to show up? Dr. Rose responded that in his Institute, their job is to be there. Attendance is important for performance of their responsibilities. There is, however, a scheduling problem, with too few staff to cover too many meetings. One problem is that the schedule for a particular study section may not be in the computer system.
The next topic for discussion was clearances needed for appointing members. Currently, the SRA needs to get clearance from the NIH Director, and the nominations are at the level of a specific study section, rather than at the level of the entire IRG. For example, an IRG may have one charter for seven study sections. They can be flexible about the way to use these members; but when they appoint members, they are still held to produce seven separate slates; it would be to their advantage to be able to combine all the nominations into one big package. They could do a much better job of balancing all the administrative constraints while still trying to find the best scientists. Dr. Luecke responded that some Institutes have already begun to implement this practice and the Division may also be able to accomplish this. Progress on this matter will be reported to the Advisory Committee.
The next topic of discussion was whether all the study sections within an IRG should meet at a specific time and location, so that the possibility of movement of members within an IRG could be facilitated. However, it was pointed out that the usefulness depends on the overlap in science of the individual study sections. Switching members is a new process, and the SRAs are working out some issues, including the logistical difficulties and the different voting behaviors within study sections. Another SRA explained that based on overlap and workload, they have recently begun shifting blocks of applications and reviewers rather than one at a time. It also was a way to level workloads in the current fiscal climate and downsizing. If SRAs take a reviewer out of a study section, they must realize the impact on the study section, and it may be necessary to get another ad hoc reviewer. It might be better to find an outstanding ad hoc reviewer to go to the first study section. Given the obvious logistical problems of trying to get all the scientists and the Institute program people together, one discussant questioned the advantage of shifting reviewers as opposed to applications. One discussant noted, however, that the best science in the future will probably be nonhomogeneous, and that individual projects will require a large range of technological and scientific expertise for review.
One SRA, who was also a Referral Officer, noted that when an application does not fit anywhere, they negotiate with the SRA of the study section closest to it, often discussing possible ad hoc reviewers. This may involve senior investigators who are willing to come for one meeting.
One person suggested encouraging SRAs to appoint individuals for shorter terms, for 2 years, to be renewed for a 3rd year and 4th year if their particular areas of expertise still match the numbers of relevant applications received.
One SRA explained that a possible drawback to cross-fertilization and trading members for one meeting is that the individual would not feel a part of the camaraderie and collegiality that develops among members who get to know each other well.
Dr. Luecke responded that with regard to the tradeoffs of short-term versus long-term membership, there is no "right" answer. It depends on the judgment of the SRA and willingness of the reviewer. DRG management is not comfortable with creating rules that must be strictly adhered to; rather the process should enable SRAs to recognize and hopefully better utilize available resources. The final integration is going to have to occur at the SRA level.
Another discussant stressed the importance of flexibility, complimenting DRG for its various experiments in peer review. Although many study section members greatly value the particular culture of their study section, cross-cultural education is also useful at times, and they can learn from seeing how different groups operate.
The next topic was the role of the SRA in interacting with the extramural community. How should that be implemented? What role should DRG play in that process? Should DRG have a model, wherever practical, that could be used across all components? If so, how could the Advisory Committee help implement this?
One discussant noted that when SRAs attend scientific meetings, it really makes a difference. Could this become part of the requirements of an SRA? Dr. Luecke responded that it is not so much a problem of the function of the SRA, but more a problem of resources. Thanks in part to the DRG Advisory Committee, there has been some recent progress in this area; DRG needs to explore other innovative ways to facilitate the travel of SRAs, possibly in a way that would not tie a particular SRA to any group of applicants because of conflict-of-interest regulations.
There was considerable discussion about having societies create
funds for travel for NIH staff to go
to meetings. That problem has to be resolved in terms of even
the appearance of conflict of interest.
One person wondered whether a way to avoid conflicts of interest
might be to have a group or
consortium of societies put money into one large account as a
type of blind trust. Dr. Luecke noted
the DRG has begun to explore various possibilities. One of the
members of the Advisory Committee
suggested that the Committee draft a recommendation about the
need of the SRAs to get out to the
community by attending meetings, preparing workshops, or doing
other outreach activities for the
communities' benefits. If the Division and the Advisory
Committee can work to improve the
information flow, this is a useful goal.
XIII. Report of the Subcommittee on Rating Grant Applications
Assigned DiscussantsDr. Walter Stolz
Dr. Hugh Stamper
Dr. Nesselroade
Dr. Bakeman
Dr. Stolz, Director of the Division of Extramural Activities, National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), and Co-chairperson of the Committee, explained its establishment as a Subcommittee of the NIH Committee for the Improvement of Peer Review. The Subcommittee met regularly for about a year to evaluate the system currently used by NIH to rate grant applications and to address the means to report scores of scientific merit. The Subcommittee considered the areas of psychometrics and decision science, and enlisted the aid of prominent experts in the behavioral sciences. Recommendations presented in the report are in four general areas: (1) the review criteria used to develop an assessment of scientific merit; (2) the rating scale; (3) the comparability of scores across reviewers, review groups, and review cycles; and (4) how scores are reported.
In the first area, the Subcommittee agreed that the current review criteria included the principal components to assess scientific merit, but felt these should be reformulated into three major criteria: significance, approach, and feasibility. The summary statement critiques should address each criterion separately, and provide a score for each. The Subcommittee consultants advised that the reliability of rating would increase through the disaggregated criterion process, i.e, discussing, rating, and scoring each criterion separately.
The second area is the rating scale, which currently goes from a high of 1.0 to a low of 5.0, in 0.1 increments, thus providing 41 points on the scale. The Subcommittee agreed that the number of scale positions should be commensurate with the potential number of reliable discriminations. Based on relevant psychological literature, it was recommended that a seven-step scale be used, with the addition of "0" to represent unacceptability of the characteristic being rated. In addition, the rating scale should be inverted, so that larger values represent more favorable characteristics of the criterion being rated.
In the issue of comparability of scores, two approaches were presented to make scores more comparable across reviewers, groups, and time. Both of these are employed currently at the NIH. One approach is to instruct reviewers to anchor the scale positions in an attempt to use scores the same way and to mean the same thing to different people. NIH uses adjectives to anchor the scales. In another approach, after the ratings have been done, scores are processed statistically in an attempt to achieve uniformity in rating behavior. NIH uses percentiling in an attempt to achieve uniformity in rating behavior. The Subcommittee recommended that the scale should be anchored only at the ends. Also, instead of percentiling by group, which is the current procedure, the Subcommittee recommended that scores be standardized within each reviewer.
In the fourth area, reporting priority scores, it was recommended that scores should be reported with a precision commensurate with the meaningful information contained in the scores. Two significant digits were recommended. Even though reviewers rate using one digit, the resulting values summarizes a group of possibly 10 to 20 scores for each application, and so two digits are useful to report the priority scores.
Finally, a majority of the committee believed that an overall priority score should be computed by averaging the three criterion scores. In response to a question of whether a greater weight is to be placed on significance as opposed to feasibility, Dr. Stolz said that the advantages and disadvantages of weighting scores are currently under discussion.
Dr. Hugh Stamper, Director, Division of Extramural Activities, National Institute of Mental Health, and Co-chairperson of the Subcommittee, then commented on the Subcommittee's recommendations. He noted that standardizing scores by reviewer rather than by study section allows a person to serve and vote more systematically on more than one study section. When members attend another study section meeting, even for one grant application, their own voting distributions are used in the standardized process. He added that each review criterion would be scored individually, and each criterion along with the overall score would be reported on the summary statement.
Dr. Nesselroade, the first discussant, pointed out a potential problem with standardizing priority ratings by reviewers rather than by the group. If there are two reviewers, one of whom is doing a valid job of making difficult discriminations among a set of applications and another reviewer who is not, by standardizing, those two distributions may have the same variance. Thus, the process may not correct for the differences between these two types of reviewers. His next comment pertained to how recommendations would be evaluated. The report mentions the possibility of having some applications reviewed also by a control study section, and then determining how well the two groups agree. This is a typical test to determine the correlation between parallel forms to measure reliability. Other evaluation methods also could be considered, such as comes from the generalizability tradition, which would take into account the idea that each application has a universal score. That might be the score it would be assigned if it could be given to all possible study sections. One then could estimate how much variation in these scores is actually being contributed by having it go to one study section versus another study section.
Dr. Bakeman, the second discussant, suggested developing a spreadsheet with histories of reviewers to determine how each is represented in terms of numbers. There are individual differences, and different people can validly make different discriminations. When a reviewer joins a study section, the study section can provide a recommended scale based on the spreadsheet; but in fact, reviewers can choose their own scale. For each reviewer, one possibility might be to standardize only over their last three review cycles. Many individuals change considerably over time. Basically, Dr. Bakeman thought standardized individual ratings have some strong advantages over the current system. He felt, however, that it was arguable whether there should be three or more review criteria. Furthermore, the division into significance, approach, and feasibility may lengthen review discussions. He suggested asking current reviewers' opinions on this recommendation.
Discussion
The possibility of a fourth criterion -- innovation or creativity -- was suggested. The three criteria might tend to favor more conservative applications from long-term investigators. Dr. Stamper responded that the committee felt that significance and feasibility included the issue of innovative, high-risk research. Having creativity as an explicit criterion would make each reviewer pay attention to this category, which might tend to balance against the well-established investigator. The DRG Advisory Committee agreed to make the addition of a creativity or innovation criterion an explicit recommendation for the Scoring Subcommittee.
One question pertained to competing renewals, where the investigator's productivity over the past years is a major factor in rating an application. How are these three criteria used when reviewing a competing renewal application? Dr. Stamper responded by noting that the criterion of feasibility is intended to include past progress. However, it was noted that feasibility versus past productivity are usually not connected, because feasibility relates to the future and productivity to the past. This review criterion may need to be rewritten.
Another suggestion was that DRG should provide only the three criterion scores and not try to create an overall score. There are various means of weighting of those three criteria among the various Institutes. Also, with the greater number of criteria, it may take a lot longer to discuss each application. Dr. Stolz responded that this system is similar to the way contract proposals are reviewed. Staff who are experienced with contract reviews said that time should not be an issue, thatreviewers get accustomed to this kind of format and soon review expeditiously.
Another concern was that people are resistant to change, and the committee should make recommendations on how these changes will be introduced. Dr. Stamper agreed, but noted that the committee will not be overseeing implementation. However the report will include a strong recommendation for careful testing and evaluation.
Another question concerned the effect of inflated priority scores, or grade creep, that might result from the proposed seven point scale. Dr. Stamper responded that the reviewers would know the effect of their own ratings; it is different when done on an individual basis rather than on a committee basis. Dr. Luecke added that there was some enthusiasm for having another recalibration of the current study sections' voting patterns and having reviewers spread their ratings so that they provide greater discrimination. This requires close cooperation with the Institutes. Dr. Luecke also addressed the issue of standardizing ad hoc reviewers. How often must they attend study section meetings and how many applications must they review in order to qualify for standardization? This concern does not preclude judiciously moving ahead in that direction.
Dr. Stamper noted that the report, when revised, will go to the Committee to Improve Peer Review, and then will be submitted to Dr. Wendy Baldwin for possible implementation.
XIV. Cooperative Review Effort between the National
Institute on Alcohol Abuse and Alcoholism (NIAAA)
and DRG
Dr. Jeanne Ketley
Dr. Faye Calhoun
Dr. Kenneth Warren
Dr. Ketley provided some background on this cooperative project. In October 1992, the research arms of the three Institutes that made up the ADAMHA agency were brought into the NIH. However, their review functions, which had been done independently, would not be brought into the DRG immediately. As part of an effort to merge NIAAA reviewers into DRG, Dr. Faye Calhoun, now the Deputy Director for Collaborative Research at NIAAA, suggested that DRG and NIAAA begin working to integrate part of their biomedical portfolio with the two DRG toxicology study sections.
Dr. Kenneth Warren, Director, Office of Scientific Affairs, NIAAA, provided more detail on the ongoing NIAAA-DRG Pilot Peer Review Integration Program. Dr. Warren explained the creation of NIAAA in 1970, and its mission to develop and conduct comprehensive health education and research activities around alcohol abuse and alcoholism. In July 1992, the ADAMHA Reorganization Act reorganized the Alcohol, Drug Abuse and Mental Health Administration, or ADAMHA, and returned the three research Institutes to the NIH, leaving a separate organization (SAMHSA) for the services, training and block grant functions.
After a brief presentation about the organization and activities of NIAAA, Dr. Warren turned to the issue of peer review integration. The same law that moved NIAAA back into NIH states that after FY 1996, the peer review systems utilized by NIAAA and the other two Institutes within the former ADAMHA would be integrated into the DRG. For the purpose of the discussion, Dr. Warren concentrated on their two biomedical committees, ALCB-1 and ALCB-2. The ALCB-1 committee focuses on research applications related to: alcohol metabolism, including cytochrome P450 enzyme systems; physiologic and toxicologic mechanisms of alcohol on various organ systems, including the immune system and reproductive system; and development as it pertains to alcohol teratogenesis. ALCB-2 has as its major focus: behavioral teratology, alcohol-related neurotoxicology and neuroendocrinology, particularly as it relates to reproductive biology; alcohol-derived neuropathology; and the study of alcohol's actions at the molecular and cellular level with specific receptor systems, ion channels and signal transduction systems.
The task in developing the integration pilot was to review the structure of the 18 DRG IRGs, and find a home for the basic research portfolio of these alcohol-related applications. The Health Promotion and Disease Prevention IRG seemed the most appropriate, specifically its two toxicology study sections, because of similarities in their scientific areas and their focus on adaptive biology and the physiological consequences of the actions of xenobiotics. For the February 1996 review round, DRG and NIAAA staff analyzed similar areas of research to determine how the two portfolios fit together and how the workloads could be restructured to assure quality review. After evaluating the process, DRG and NIAAA will make adjustments as needed for the June 1996 review round.
The processes and procedures used by all the review groups are nearly identical, such that several research areas could be reviewed together. The SRAs exchanged information about the applications reviewed. They also had meetings with appropriate program staff and DRG management. In addition, a subcommittee of their primary constituency organization, the Research Society on Alcoholism, was formed to consider the implications of this peer review integration. NIAAA staff addressed members of the Society in a plenary session at their annual meeting last June. Another closely affected Institute is NIEHS, and so DRG and NIAAA management staff traveled to North Carolina to visit and brief both the program staff and also the Council of NIEHS. Fortunately, both ALCB-2 and Toxicology-2 met at the same time and within five blocks of each other. Therefore, they were able to arrange a social gathering to discuss informally this issue. In February, all four review groups will meet at the same time and in close proximity to each other. Staff are currently designing the methods for evaluating the pilot and gaining feedback on the study.
Dr. Luecke noted that this pilot project came about, in large measure, because of a groundswell from the SRAs in both organizations. The interactions grew where they should have been growing, at the grassroots level. He thanked Dr. Warren and the other extramural staff at NIAAA for their initiative, and expressed the hope that additional efforts will happen with the National Institute on Drug Abuse and the National Institute of Mental Health. Dr. Faye Calhoun added that they are planning to have senior members of both research communities involved in the February meeting. They will attend as ad hoc members to help monitor and evaluate the process.
Dr. Luecke brought up the topic of the NIH Reviewers Reserve. The issue is whether such ad hoc reviewers really have full membership standing, and how they might participate more fully in a meeting. He suggested identifying individuals as part of an IRG Reviewers Reserve. These would be former members of an IRG, who would be willing to come back once or twice a year to serve on a study section within a specific IRG. There would be some sort of designation related to the Reviewers Reserve to identify individuals who are a part of a particular IRG.
Dr. Schachman agreed, suggesting also that these ad hoc individuals might be available for every study section with a title such as "distinguished scientists." Some concern was raised with the use of this term distinguished and the possible interpretation that other reviewers are not. There might be two such scientists from past study sections for every study section meeting, who would have some assignments, and who would vote. They could make enormous contributions. They bring maturity, historical background, a lot of expertise, and breadth to the process. However, there was some concern about the title and its affect on the study section, and a suggestion that this idea be tried informally at first without a title, and then evaluated.
Dr. Luecke noted that under his IRG reserve concept, more junior members could be used depending on their expertise. Dr. Braciale stated that the two suggestions (to build a reviewer reserve within a particular IRG and to establish a mechanism so that appropriate "senior" people can provide input and support at a particular study section meeting) are overlapping and not mutually exclusive.
It was suggested that the topic of special ad hoc advisors to an IRG and to the DRG should be a major agenda item for the next meeting.
In response to a question about how individual applications are assigned to particular IRGs, Dr. Luecke summarized the current system involving written referral guidelines that are changed every two years, input from the Institutes, and assignment by referral officers, who are senior SRAs, most of whom also manage study sections. In a proposed modification, the referral officers would judge which IRG would be best, but then the fine tuning to study sections would occur at the level of the IRG. Most of the new IRG coordinators will be referral officers.
To what extent can the investigator have input to the referral process? Dr. Luecke responded that he did not envision a change from current practices, whereby investigators can ask for up to three Institute and/or three study section assignments. DRG encourages such initiatives, which are taken into account when assignments are made. During discussion, it was noted that as science is increasingly crossing techniques and crossing disciplines, study sections are using more ad hoc reviewers. There may be a significantly different study section composition each meeting than what appears on the standing roster. The current process appears to work well. Perhaps the flexibility that exists now for investigators to write letters requesting assignments should be more widely advertised.
One suggestion--especially for newly independent investigators who are worried about finding an appropriate review group--is that they call an SRA, before submitting an application, to talk about possible study sections. Finally, several discussants preferred the current flexibility and rationale of using a cover letter with an application rather than checking off boxes on a form.
XVI. 1996: 50 Years of NIH Peer Review
Dr. Donald Luecke
Dr. Suzanne Fisher
Dr. John Sherman
Dr. Luecke remarked that 1996 is the year in which NIH will celebrate 50 years of peer review. The emphasis is not on DRG alone, but on 50 years of peer review at NIH.
Dr. Suzanne Fisher, Assistant Chief of the Referral Section, and coordinator of the celebration of 50 years of peer review and 50 years of the Division of Research Grants, invited everyone to the major event, an all day symposium in the Natcher conference facility on June 20th. June 21 will be a predominantly in-house event involving present and past employees.
The symposium will focus on the past, present and future of peer review. The organizers hope that this will provide a very positive statement about peer review. This event will mark the debut of a book called Half a Century of Peer Review: A History of the Division of Research Grants. The book is being written by Dr. Richard Mandel, a historian. The present, will include a report on the current initiatives in DRG. There has been much discussion about how much emphasis should be on science versus how much on public policy issues. The organizers want to include science, because excellent science is the product of peer review. The goal is to have talks from scientists who have served on study sections about their experiences in serving and having their applications reviewed, and also about the resulting scientific achievements. The various speakers will probably include active research investigators, university administrators, NIH staff, members of advocacy groups, and people in the media. The symposium will end with discussions of the future of peer review by Dr. Varmus and the Director of DRG. The day will conclude with a reception at the Natcher Conference Center.
The Division's efforts are being greatly aided by excellent advice, logistical support, and financial support from a group called Friends of DRG. Dr. Fisher then introduced Dr. John Sherman, one of the principal persons working with the Friends of DRG. Dr. Sherman has held numerous positions at NIH, including Deputy Director, Acting Director , and Associate Director for Extramural Programs, and has been Executive Vice President of the American Association of Medical Colleges. Dr. Sherman explained that instead of approaching pharmaceutical companies or other big organizations, they looked to professional societies, voluntary health organizations, and others interested in the concept of peer review. They have approached approximately 75 major voluntary health organizations, professional societies, organizations, institutions and so forth. So far, the drive has revealed a heartwarming response. The FASEB organization has already come forth with a sizeable pledge, and the AAMC has offered their expertise in program planning and management of meetings as well as in handling all the accounting of funds. Their goal is to raise $50,000 to $75,000.
XVII. The Next Step for DRG and the
DRG Advisory Committee
Dr. Donald Luecke
Dr. Luecke spoke about the future organization of review into three groups, each with about six IRGs (Figure 2). Dr. Luecke then briefly described this organization and staff. With respect to the DRG Advisory Committee, Dr. Luecke proposed expanding the membership to 15 advisors to enhance discussions. In addition, he envisioned bringing on four or five people on an ad hoc basis for each meeting. Besides the two meetings a year, he proposed forming working groups to report on such issues as clinical research.
A question was asked about the relationship of the DRG Advisory Committee to the Peer Review Oversight Group (PROG). Dr. Luecke felt the Advisory committee would carve out a role looking at the structure, the function, and the ongoing evaluation of the DRG. Dr. Luecke explained that PROG's role goes beyond DRG.
One suggestion was for PROG to be involved with issues related to requests for applications (RFAs) and program announcements (PAs). Those are Institute specific and are not well coordinated. PROG could have a subcommittee that receives and coordinates information from various scientific groups A good example is a recent National Cancer Institute solicitation for research in neurofibromatosis. This is a very broad disease that includes not only cancer but also learning disabilities, and therefore crosses Institute boundaries. PROG could thus have a broader view of not just the peer review process but also the impact of new directions that the NIH is taking across Institute boundaries. PROG should not be micromanaging the individual study sections or the IRGs.
To enhance communications between the DRG Advisory Committee and PROG, in addition to the DRG Director being on PROG, the Chairman of the DRG Advisory Committee should be on PROG, either as one of the extramural members or as an ex officio member.
Another recommendation was related to the DRG goal of periodic review of study sections within the Division. This could be one of the major responsibilities of the Advisory Committee. Such a task requires careful study, clear data and strong justification. The Advisory Committee could oversee that effort, designing it carefully and having the appropriate advisors. In this process of helping the DRG formulate a quality control review mechanism, it might be valuable to have the Director and the Chair of the Committee meet or be in communication on a more regular basis and use that as a conduit for communications with the membership. With e-mail and World Wide Web, they could do a fine job of keeping the membership informed and also communicating with the scientific community.
Another suggestion was to designate members of a working group of the DRG Advisory Committee to a particular study section or group of study sections. They would spend time sitting in study section meetings and evaluating the process. One proposal was for a five- year review cycle. Any greater frequency might be too chaotic, and there would not be enough time to deal with all the study sections. The study sections would be grouped by IRGs. The main intent would be to have a process where the study sections and the IRGs are evaluated by the same standards.
Dr. Braciale thanked his colleagues for their sacrifices in coming here to help out the DRG. He then thanked Dr. Luecke and the DRG staff for their willingness and efforts to open up and examine the peer review process with the Committee.
There being no further business to discuss, the meeting was adjourned at 10:30 a.m. on November 14, 1995.
We do hereby certify that, to the best of our knowledge, the foregoing minutes and attachments are accurate and complete.
/s/
_____________________________
Samuel H. Joseloff, Ph.D.
Executive Secretary
Division of Research Grants
Advisory Committee
/s/
_______________________________
Donald H. Luecke, M.D.
Acting Director
Division of Research Grants
/s/
______________________________
Thomas Braciale, M.D., Ph.D.
Chairperson,
Division of Research Grants
Advisory Committee
Note. These minutes will be formally considered by the Advisory Committee at its next meeting on May 20 & 21, 1996; any corrections or notations will be incorporated in the minutes at that meeting.
For questions or comments regarding the DRG Advisory Committee Meeting Minutes, please contact Dr. Samuel Joseloff: joselofs@drg.nih.gov.
![[Return to DRG Home Page]](../images/drghome.gif)