The Concept of Reliability in the Context of Observational Assessments

Occupational therapists are interested in clients' performance of activities of daily living in real-life contexts. To maximize the validity of inferences made from assessments, occupational therapists conduct observations in everyday environments: at home, school, and work and in the community. Although these observations in context are arguably more valid, they are potentially less controlled, which affects the reliability of the data obtained. Occupational therapists may wonder about the evidence of reliability when evaluating observational assessments for use or question reliability considerations in developing protocols. This article explores the concept of reliability in the context of observational assessments. We compare two theoretical approaches to reliability; apply the theory to the example of occupational performance, as viewed by occupational therapists; and argue the relevance of internal consistency, observer (rater) reliability, and sample reliability. Finally, the authors examine in-depth factors influencing the sample reliability of observational assessment scores, weighed against the utility of assessments and generalizability of scores.