Cognitive Impact Assessment of Electronic Knowledge Resources: A Mixed Methods Evaluation Study of a Handheld Prototype

RATIONALE We recently proposed a new method to systematically assess the cognitive impact of knowledge resources on health professionals. OBJECTIVE To describe promises and shortcomings of a handheld computer prototype of this method. BACKGROUND We developed an impact scale, and combined this scale with a Computerized Ecological Momentary Assessment technique. METHOD We conducted a mixed methods evaluation study using a 7-item scale within a questionnaire linked to a commercial knowledge resource. Over two months of Family Medicine training, 17 residents assessed the impact of 1,981 information hits retrieved on handheld computer. From observations, log-reports, archives of hits and interviews, we examined issues associated with hardware, software and the questionnaire. FINDINGS Fifteen residents found the questionnaire clearly written, and only one pointed to the questionnaire as a major reason for their low level of use of the resource. Residents reported technical problems (e.g. screen trouble) or limitations (e.g. limited tracking function) and socio-technical issues (e.g. software dependency). CONCLUSION Lessons from this study suggest improvements to guide future implementation of our method for assessing the cognitive impact of knowledge resources on health professionals.