The reliability and internal consistency of an Internet-capable computer program for measuring utilities

Objective: The purpose of this study was to assess the reliability and internal consistency of measurements of utilities performed with a computer program (iMPACT2) designed for Internet surveys and Internet patient decision-support systems. Methods: We implemented the Internet Multimedia Preference Assessor Construction Tool, version 2 (iMPACT2) program using the combination of a web server, HTML files, and a web-accessible database. The program randomized subjects, screened their responses for missing data and failures of internal consistency, assisted patients with resolving certain inconsistencies, and, upon a subject's completion of the protocol, provided a report of results to the research assistant administering the program. To validate the iMPACT2 program, we recruited 60 healthy community volunteers and elicited preferences in a research-lab setting using a visual analog scale (VAS) and the standard gamble (SG) for subject's current health and three hypothetical states. For purposes of comparison, we also administered a Short Form-12 (SF-12) health-assessment questionnaire. Subjects used the computer software on two occasions separated by 2–4 weeks of time. Results: Visual analog scale and standard gamble ratings for subjects' current health were reliable (intraclass correlation coefficient (ICC) of 0.82 and 0.84 (two outliers excluded − 0.60 without exclusions), respectively) were comparable with the reliability of the Physical and Mental Component scales of the SF-12 (ICCs of 0.84 and 0.75, respectively). Subjects could easily discriminate between hypothetical states (D scores 0.74 for SG and 0.90 for VAS), and 94% had a completely internally consistent ordering of preference ratings for states. Conclusions: iMPACT2 produces measurements of standard gamble utilities that are reliable and have a high degree of internal consistency. Procedures for assessment of utilities developed for desktop computer programs can be translated to software designed for the Internet, facilitating the use of utilities and endpoints in clinical trials and development of web-based decision-support applications for patients. However, further testing, including direct comparisons with traditional interviewer administered utility elicitation protocols, is needed.

[1]  W Sumner,et al.  U-titer: a utility assessment tool. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[2]  Leslie Lenert,et al.  Research Paper: Rapid Approximation of Confidence Intervals for Markov Process Decision Models: Applications in Decision Support Systems , 1997, J. Am. Medical Informatics Assoc..

[3]  H. B. Webb.  The measurement of health. , 1956, A.M.A. archives of industrial health.

[4]  R. Soetikno,et al.  Studying ulcerative colitis over the World Wide Web. , 1997, The American journal of gastroenterology.

[5]  W. vanEimeren.  Measurement of health , 1977 .

[6]  Catherine A. Sugar,et al.  Empirically defined health states for depression from the SF-12. , 1998, Health services research.

[7]  L A Lenert,et al.  Effects on Preferences of Violations of Procedural Invariance , 1999, Medical decision making : an international journal of the Society for Medical Decision Making.

[8]  R. Soetikno,et al.  Quality-of-life research on the Internet: feasibility and potential biases in patients with ulcerative colitis. , 1997, Journal of the American Medical Informatics Association : JAMIA.

[9]  L A Lenert,et al.  Associations between health status and utilities implications for policy. , 1999, Medical care.

[10]  Michael W. Kattan,et al.  Telephone interviews vs. workstation sessions for acquiring quality of life data , 1999, AMIA.

[11]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[12]  L A Lenert,et al.  IMPACT: an object-oriented graphical environment for construction of multimedia patient interviewing software. , 1995, Proceedings. Symposium on Computer Applications in Medical Care.

[13]  C. Sugar,et al.  Comparison of preferences for health outcomes in schizophrenia among stakeholder groups. , 2000, Journal of psychiatric research.

[14]  B. O'brien,et al.  Valuing outcomes in health care: a comparison of willingness to pay and quality-adjusted life-years. , 1998, Journal of clinical epidemiology.

[15]  Leslie Lenert,et al.  SecondOpinion: interactive Web-based access to a decision model , 1997, AMIA.

[16]  Leslie Lenert,et al.  Validity and Interpretation of Preference-Based Measures of Health-Related Quality of Life , 2000, Medical care.

[17]  J. Ware,et al.  A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. , 1996, Medical care.

[18]  D K Owens,et al.  Cost-effectiveness of warfarin and aspirin for prophylaxis of stroke in patients with nonvalvular atrial fibrillation. , 1995, JAMA.

[19]  L. Lenert,et al.  The effect of assessment method and respondent population on utilities elicited for Gaucher Disease , 1997, Quality of Life Research.

[20]  D K Owens,et al.  Distributed Decision Support Using a Web-based Interface , 1999, Medical decision making : an international journal of the Society for Medical Decision Making.

[21]  L. Lenert,et al.  The Effect of Search Procedures on Utility Elicitations , 1998, Medical decision making : an international journal of the Society for Medical Decision Making.

[22]  D. Pryor,et al.  Variation in patient utilities for outcomes of the management of chronic stable angina. Implications for clinical practice guidelines. Ischemic Heart Disease Patient Outcomes Research Team. , 1995, JAMA.

[23]  S. Kaplan,et al.  Foot infections in diabetic patients. Decision and cost-effectiveness analyses. , 1995, JAMA.

[24]  Harold C. Sox,et al.  Variation in Patient Utilities for Outcomes of the Management of Chronic Stable Angina: Implications for Clinical Practice Guidelines , 1995 .

[25]  C. Ashton,et al.  Assessing the performance of utility techniques in the absence of a gold standard. , 1999, Medical care.

[26]  M K Goldstein,et al.  Measurement of the validity of utility elicitations performed by computerized interview. , 1997, Medical care.

[27]  R. Kane,et al.  Methodology for measuring health-state preferences--II: Scaling methods. , 1989, Journal of clinical epidemiology.

[28]  Leslie Lenert.  iMPACT3: online tools for development of web sites for the study of Patients' preferences and utilities , 2000, AMIA.

[29]  D. Gerding.  Foot infections in diabetic patients: the role of anaerobes. , 1995, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[30]  P. Dolan,et al.  Valuing health states: a comparison of methods. , 1996, Journal of health economics.