Uncertainty and Coverage With Evidence Development: Does Practice Meet Theory?

OBJECTIVES In theory, a successful coverage with evidence development (CED) scheme is one that addresses the most important uncertainties in a given assessment. We investigated the following: (1) which uncertainties were present during the initial assessment of 3 Dutch CED cases, (2) how these uncertainties were integrated in the initial assessments, (3) whether CED research plans included the identified uncertainties, and (4) issues with managing uncertainty in CED research and ways forward from these issues. METHODS Three CED initial assessment dossiers were analyzed and 16 stakeholders were interviewed. Uncertainties were identified in interviews and dossiers and were categorized in different causes: unavailability, indirectness, and imprecision of evidence. Identified uncertainties could be mentioned, described, and explored. Issues and ways forward to address uncertainty in CED schemes were discussed during the interviews. RESULTS Forty-two uncertainties were identified. Thirteen (31%) were caused by unavailability, 17 (40%) by indirectness, and 12 (29%) by imprecision. Thirty-four uncertainties (81%) were only mentioned, 19 (45%) were described, and the impact of 3 (7%) uncertainties on the results was explored in the assessment dossiers. Seventeen uncertainties (40%) were included in the CED research plans. According to stakeholders, research did not address the identified uncertainty, but CED research should be designed to focus on these. CONCLUSIONS In practice, uncertainties were neither systematically nor completely identified in the analyzed CED schemes. A framework would help to systematically identify uncertainty, and this process should involve all stakeholders. Value of information analysis, and the uncertainties that are not included in this analysis should inform CED research design.

[1]  Jonathan Karnon,et al.  Model Parameter Estimation and Uncertainty: a Report of the Ispor-smdm Modeling Good Research Practices Task Force-6 Background to the Task Forcemodel-parameter-estimation-and- Uncertainty-analysis.asp). a Summary of These Articles Was Pre- Sented at a Plenary Session at the Ispor 16th Annual Intern , 2022 .

[2]  K Claxton,et al.  Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development. , 2012, Health technology assessment.

[3]  Marianne Klemp,et al.  What principles should govern the use of managed entry agreements? , 2011, International Journal of Technology Assessment in Health Care.

[4]  Warren E. Walker,et al.  Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support , 2003 .

[5]  R. Edlin,et al.  Access with Evidence Development Schemes , 2012, PharmacoEconomics.

[6]  Andrew Briggs,et al.  Performance-based risk-sharing arrangements-good practices for design, implementation, and evaluation: report of the ISPOR good practices for performance-based risk-sharing arrangements task force. , 2013, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[7]  Ellen Vos,et al.  Wrestling with uncertain risks: EU regulation of GMOs and the uncertainty paradox , 2008 .

[8]  M. Drummond.  When do performance-based risk-sharing arrangements make sense? , 2015, The European Journal of Health Economics.

[9]  Cheryl Tatano Beck,et al.  Resource Manual for Nursing Research: Generating and Assessing Evidence for Nursing Practice , 2011 .

[10]  S. Ramsey,et al.  Development and Evaluation of an Approach to Using Value of Information Analyses for Real-Time Prioritization Decisions Within SWOG, a Large Cancer Clinical Trials Cooperative Group , 2016, Medical decision making : an international journal of the Society for Medical Decision Making.

[11]  Peter Littlejohns,et al.  Evidence-based decision making: when should we wait for more information? , 2008, Health affairs.

[12]  P. Sainsbury,et al.  Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. , 2007, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[13]  S. Palmer,et al.  A Comprehensive Algorithm for Approval of Health Technologies With, Without, or Only in Research: The Key Principles for Informing Coverage Decisions. , 2016, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[14]  C. McCabe,et al.  Funding the Unfundable , 2012, PharmacoEconomics.

[15]  A. Hollis.  Sustainable Financing of Innovative Therapies: A Review of Approaches , 2016, PharmacoEconomics.

[16]  Karl Claxton,et al.  Coverage with evidence development, only in research, risk sharing, or patient access scheme? A framework for coverage decisions. , 2012, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[17]  M. Janssen,et al.  The Impact of Decision Makers' Constraints on the Outcome of Value of Information Analysis. , 2017, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[18]  P. Kanavos,et al.  Dealing with uncertainty and high prices of new medicines: a comparative analysis of the use of managed entry agreements in Belgium, England, the Netherlands and Sweden. , 2015, Social science & medicine.

[19]  Manuela A. Joore,et al.  Healthy Decisions: Towards Uncertainty Tolerance in Healthcare Policy , 2014, PharmacoEconomics.

[20]  P. Sonneveld,et al.  Policymaker, please consider your needs carefully: does outcomes research in relapsed or refractory multiple myeloma reduce policymaker uncertainty regarding value for money of bortezomib? , 2014, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[21]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[22]  Gordon H Guyatt,et al.  GrADe : what is “ quality of evidence ” and why is it important to clinicians ? rATING quALITY of evIDeNCe AND STreNGTH of reCommeNDATIoNS , 2022 .

[23]  Thomas A. Schwandt,et al.  Judging interpretations: But is it rigorous? trustworthiness and authenticity in naturalistic evaluation , 2007 .

[24]  Claire Rothery,et al.  Characterising Uncertainty in the Assessment of Medical Devices and Determining Future Research Needs. , 2017, Health economics.

[25]  P. Scuffham,et al.  The Australian Managed Entry Scheme: Are We Getting it Right? , 2018, PharmacoEconomics.

[26]  Salah Ghabri,et al.  Towards a New Framework for Addressing Structural Uncertainty in Health Technology Assessment Guidelines , 2018, PharmacoEconomics.

[27]  Mark Strong,et al.  The HTA Risk Analysis Chart: Visualising the Need for and Potential Value of Managed Entry Agreements in Health Technology Assessment , 2017, PharmacoEconomics.

[28]  J. Kleijnen,et al.  Nivolumab for Treating Metastatic or Unresectable Urothelial Cancer: An Evidence Review Group Perspective of a NICE Single Technology Appraisal , 2018, PharmacoEconomics.

[29]  Andrew R Willan,et al.  Expected value of information and decision making in HTA. , 2007, Health economics.

[30]  P. Kanavos,et al.  Managing Risk and Uncertainty in Health Technology Introduction: The Role of Managed Entry Agreements , 2017 .

[31]  S. Palmer,et al.  Unifying Research and Reimbursement Decisions: Case Studies Demonstrating the Sequence of Assessment and Judgments Required. , 2015, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.