Naturalistic observation in clinical assessment

SummaryThe development of the Behavioral Coding System (BCS) used by the Social Learning Project has encompassed approximately 8 years of clinical and research experience with naturalistic observation as a clinical assessment tool. The BCS, while originally designed to accomplish certain broad purposes, illustrates a solution to an assessment task that should be applicable to other research and clinical settings in which naturalistic observation of family interactions are needed.A variety of reliability analyses, ranging from traditional interobserver agreement among coders to generalizability analyses, have supported the measurement precision of the BCS scores, for their intended purposes. In conducting this series of investigations, certain problems in psychometric analysis of observation data have arisen and been documented. Most notably, the tradition of estimating reliability via interobserver agreement has been questioned, mainly on the grounds that behavioral complexity intrudes into such analyses in ways that suggest that current observer reliability estimates may be substantially biased. The usefulness of generalizability theory is argued, particularly for observational data collected under varying assessment conditions which may influence behavioral scores.Three types of validity have been reported for BCS scores: content, concurrent, and construct validity. The BCS has favorably withstood these psychometric investigations, showing that the behavioral measures are justified on content grounds, that outside reports of behavior coincide satisfactorily with the BCS scores, and that expected behavioral changes following treatment are readily indexed by the BCS scores.