Reliability and Validity Issues Related to Interactive Tailored Patient Assessments: A Case Study

Recently there has been a proliferation of interactive tailored patient assessment (ITPA) tools. However, evidence of the reliability and validity of these instruments is often missing, which makes their value in research studies questionable. Because several of the common methods to evaluate instrument reliability and validity are not applicable to interactive tailored patient assessments, informatics researchers may benefit from some guidance on which methods of reliability and validity assessment they can appropriately use. This paper describes the main differences between interactive tailored patient assessments and assessment instruments based on psychometric, or classical test, theory; it summarizes the measurement techniques normally used to ascertain the validity and reliability of assessment instruments based on psychometric theory; it discusses which methods are appropriate for interactive tailored patient assessments and which are not; and finally, it illustrates the application of some of the feasible techniques with a case study that describes how the reliability and validity of the tailored symptom assessment instrument called Choice were evaluated.

[1]  Cornelia M. Ruland,et al.  Research Paper: Handheld Technology to Improve Patient Care: Evaluating a Support System for Preference-based Care Planning at the Bedside , 2002, J. Am. Medical Informatics Assoc..

[2]  Cornelia M. Ruland,et al.  Research Paper: Decision Support for Patient Preference-based Care Planning: Effects on Nursing Care and Patient Outcomes , 1999, J. Am. Medical Informatics Assoc..

[3]  J E Ware,et al.  Overview of the SF-36 Health Survey and the International Quality of Life Assessment (IQOLA) Project. , 1998, Journal of clinical epidemiology.

[4]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[5]  Cornelia M. Ruland,et al.  Research Paper: Effects of a Computerized System to Support Shared Decision Making in Symptom Management of Cancer Patients: Preliminary Results , 2003, J. Am. Medical Informatics Assoc..

[6]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[7]  David L Streiner,et al.  Being Inconsistent About Consistency: When Coefficient Alpha Does and Doesn't Matter , 2003, Journal of personality assessment.

[8]  H. Wittchen.  Reliability and validity studies of the WHO--Composite International Diagnostic Interview (CIDI): a critical review. , 1994, Journal of psychiatric research.

[9]  A. Comrey.  Factor-analytic methods of scale development in personality and clinical psychology. , 1988, Journal of consulting and clinical psychology.

[10]  J S Norbeck.  What constitutes a publishable report of instrument development? , 1985, Nursing research.

[11]  C. Miaskowski.  Gender differences in pain, fatigue, and depression in patients with cancer. , 2004, Journal of the National Cancer Institute. Monographs.

[12]  H. E. Tinsley,et al.  Uses of factor analysis in counseling psychology research. , 1987 .

[13]  R. Lennox,et al.  Conventional wisdom on measurement: A structural equation perspective. , 1991 .

[14]  L. Radloff.  The CES-D Scale , 1977 .

[15]  F. Bull,et al.  Understanding how people process health information: a comparison of tailored and nontailored weight-loss materials. , 1999, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.