Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute

In the Patient Protection and Affordable Care Act of 2010, PCORI was authorized to assist patients, clinicians, purchasers, and policymakers in making informed health decisions through research and evidence synthesis (1). Central to PCORI's strategy is the engagement of patients, caregivers, and other health care stakeholders in key aspects of the research enterprise (2). One critical opportunity for engaging patients and stakeholders is in the research application review process. PCORI posted its first broad funding announcements for comparative effectiveness research on 22 May 2012 (www.pcori.org/funding-opportunities/funding-announcements/closed-opportunities). Awards were for a maximum of $1.5 million in direct costs over 3 years. The first portfolio of projects was awarded on 18 December 2012 (www.pcori.org/pfaawards). Between May and November 2012, PCORI established and conducted a peer-review process that involved scientists; patients; and other stakeholders, such as clinicians, policymakers, and funders. PCORI is the first major U.S. funding agency to systematically require the inclusion of reviewers who are not scientifically trained in reviewing funding applications. Inclusion of such reviewers in this complex process is rare, and little guidance exists in scientific literature (3). Agencies, such as the National Institutes of Health, Agency for Healthcare Research and Quality, and U.S. Department of Defense, have some experience with including nonscientist reviewers, and some benefits have been suggested (47). However, little evidence from these efforts is available to determine whether selection of research projects is altered when patients and stakeholders are included (8). Evidence suggests that peer review using only scientists is biased against novelty (9) and may lead to selection of applications that are similar to the interests of the reviewers (10). It is speculated (but not proven) that participation of nonscientifically trained reviewers or scientists from very different fields may help correct these problems and may also improve the relevance of research to stakeholders who would implement study findings. In the context of health-related research, these end users include patients, caregivers, clinicians, and clinical policymakers. In this review, patients could either represent personal patient or caregiver perspectives or represent patients in their professional capacity (foundation or advocacy employees). They were not required to have or represent the condition discussed in a particular application. PCORI recognizes that there may be differences in these perspectives but sees value in both. Although scientists and stakeholders may also bring a patient perspective from their personal lives, reviewers who self-identified as scientific reviewers were categorized as such for the purposes of this review. This article explores the merit-review process of PCORI for its inaugural round of funding and investigates the contributions of scientist, patient, and stakeholder reviewers. PCORI's Merit-Review Process PCORI set up a 2-phase review process (Figure 1) because it received nearly 1300 letters of intent indicating an intention to apply and a 1-phase review to handle this volume of applications did not seem feasible. (PCORI received only 480 complete applications, but planning was based on the number of letters of intent received.) In phase 1, each application was reviewed by 3 scientific reviewers who submitted reviews online. There was no discussion among reviewers for this phase. PCORI had considered inviting patients and stakeholders to review in phase 1 (either alone or in addition to the scientific reviewers). However, there was not sufficient time for recruitment and training and PCORI did not yet have experience in inviting patients and stakeholders to review applications online and without discussion. Therefore, phase 1 was conducted with scientific reviewers alone. Figure 1. Inaugural peer-review process. PCORI = Patient-Centered Outcomes Research Institute. A total of 363 scientific reviewers were selected through an open call on the PCORI Web site between August and September 2012. An automated search (Reviewer Finder) was also used to identify potential researchers matched for research expertise, and they were contacted by e-mail to gauge their interest. Potential scientific reviewers were evaluated by PCORI staff using criteria, such as training; previous review experience; research experience, including receipt of grants; and professional engagement as measured by membership in key professional societies, publications, and presentations at scientific meetings. The scientists participating in phase 1 were not the same as those in phase 2. Reviewers were required to recuse themselves from reviewing an application in the cases of actual or perceived financial, professional, or personal associations with the applicant or the applicant's institution. These reviewers were trained on PCORI's merit-review process and the 8 merit-review criteria through a series of Webinars (Appendix Table 1). These criteria differ substantially from those of most scientific reviews because, in addition to the scientific rigor of the study, they include patient-centeredness, the engagement of patients and stakeholders in the conduct of the research, and the likelihood that the research could alter patient or clinician practices. Scientific reviewers in phase 1 used these review criteria to score applications and also provided an overall score for their assigned proposals. They were further asked to pay particular attention to the scientific rigor of the proposal when writing their overall assessment. Proposals with an average score among the 3 scientific reviewers that ranked in the top one third of all applications in phase 1 moved forward to phase 2 (n= 152). Appendix Table 1. PCORI's 8 Merit-Review Criteria Used in November 2012 Applications in phase 2 were first scored by 4 lead reviewers: 2 scientists, 1 patient, and 1 other stakeholder. We refer to these as prediscussion scores; the applications were then given a final score by each member of a 21-person panel during a face-to-face meeting (referred to as the postdiscussion score). Applications for patient and stakeholder reviewers were solicited in a 2-month open-application process (411 applications were received). Written statements from applicants were scored by PCORI staff using a 4-point scale to assess the applicant's motivation, relevant experience, and understanding of PCORI's mission. Nearly one half of the selected patient and stakeholder reviewers self-identified as patients, patient advocates, patient family members, or unpaid caregivers (42%) (Appendix Table 2). Patients and stakeholders were provided with several mandatory trainings by PCORI (Webinars and a 1-day face-to-face meeting). Appendix Table 2. Characteristics of Patient and Stakeholder Reviewers Overall, 59 scientists and 52 patients and stakeholders participated in phase 2. Each of the 4 lead reviewers was tasked with providing an overall score for their applications before the in-person meeting based on the phase 1 reviews. Lead reviewers had access to the critiques and scores provided by phase 1 reviewers. Because some patient and stakeholder reviewers had little scientific training, they were invited to base their overall score on 3 of the 8 merit criteria: innovation and potential for improvement (criterion 2), patient-centeredness (criterion 4), and patient and stakeholder engagement on the research team (criterion 7). After the initial phase 2 scoring, these reviewers met in person as part of panels composed of 21 reviewers (the 4 from the initial phase plus additional scientist, stakeholder, and patient reviewers identified as described earlier) and led by a chairperson on 18 November 2012. Applications that scored in the top two thirds based on the average of the 4 lead reviewers scores in phase 2 before the meeting were discussed in the larger 21-person panels (98 applications). At the meeting, verbal input was provided by the lead scientific reviewers, stakeholder reviewer, and patient reviewer. Lead reviewer scores were made available to all reviewers during the discussion. After discussion, each proposal received a final overall score from the 21 scientific, patient, and stakeholder reviewers on the panels, including revised scores from the 4 lead reviewers. All reviewers in both phases were required to complete a conflicts-of-interest disclosure statement on any financial relationships with health care entities and were required to recuse themselves from reviewing applications or participating in discussions or scoring in the case of actual or perceived financial, professional, or personal associations with an applicant or an applicant's institution. A total of 98 conflicts were noted on 69 of 480 applications. Scoring from all phases was done on a scale from 1 (exceptional) to 9 (poor), with numerically higher scores indicating weaker proposals. Final scores used in the analyses were multiplied by 10 for simplification purposes and to avoid using decimals. The 25 applications with the best (and lowest) scores, based on average postdiscussion scores of all participating reviewers from phase 2 (up to 21 scores), were approved for funding by PCORI's Board of Governors. Agreement Among and Within Scientific, Patient, and Stakeholder Reviewer Scores Before and After Discussion We conducted exploratory analyses of the level of agreement between scientist scores and patient and stakeholder scores before and after the in-person panel discussions. For visual comparison, scatterplots of stakeholder-versus-scientist and patient-versus-scientist scores were developed for both prediscussion and postdiscussion scores by application. We present 2 sets of data: those from the relevant 4 lead reviewers only and those resulting from the postdiscussion scores of all 21 revie