Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials

Objective The research compared and contrasted hand-scoring and computerized methods of evaluating the grade level of patient education materials that are distributed at an academic medical center in east Tennessee and sought to determine if these materials adhered to the American Medical Association’s (AMA’s) recommended reading level of sixth grade. Methods Librarians at an academic medical center located in the heart of Appalachian Tennessee initiated the assessment of 150 of the most used printed patient education materials. Based on the Flesch-Kincaid (F-K) scoring rubric, 2 of the 150 documents were excluded from statistical comparisons due to the absence of text (images only). Researchers assessed the remaining 148 documents using the hand-scored Simple Measure of Gobbledygook (SMOG) method and the computerized F-K grade level method. For SMOG, 3 independent reviewers hand-scored each of the 150 documents. For F-K, documents were analyzed using Microsoft Word. Reading grade levels scores were entered into a database for statistical analysis. Inter-rater reliability was calculated using intra-class correlation coefficients (ICC). Paired t-tests were used to compare readability means. Results Acceptable inter-rater reliability was found for SMOG (ICC=0.95). For the 148 documents assessed, SMOG produced a significantly higher mean reading grade level (M=9.6, SD=1.3) than F-K (M=6.5, SD=1.3; p<0.001). Additionally, when using the SMOG method of assessment, 147 of the 148 documents (99.3%) scored above the AMA’s recommended reading level of sixth grade. Conclusions Computerized health literacy assessment tools, used by many national patient education material providers, might not be representative of the actual reading grade levels of patient education materials. This is problematic in regions like Appalachia because materials may not be comprehensible to the area’s low-literacy patients. Medical librarians have the potential to advance their role in patient education to better serve their patient populations.