Validation study of the USDA's Data Quality Evaluation System

Abstract The Nutrient Data Laboratory (NDL) of USDA conducted a validation study of the USDA Data Quality Evaluation System (DQES). The system evaluates the quality of analytical data on nutrients by rating important documentation concerning the analytical method, analytical quality control, number of samples, sampling plan, and sample handling and creates a “Quality Index” (QI) and “confidence code” (CC) for each nutrient and food. Objectives of the study were: (1) to measure the variability of ratings assigned by different evaluators, (2) to assess the objectivity of the “critical” questions of the DQES categories and (3) to test the robustness of the rating scale. Out of 39 individuals who participated in the International Postgraduate Course for Food Composition and four nutritionists from NDL, 37 completed the evaluation of a research article containing analytical data on vitamin K in lettuce, 25 evaluated an article containing data on catechin in black grapes, and 16 evaluated an article containing data on riboflavin in portabella mushrooms using the DQES. The various rating scores assigned by the participants were analyzed to assess the success of above objectives. The maximum score for each category was 20. There was a greater variation among individuals in the ratings for “number of samples” and “analytical quality control” categories than the other three categories. Overall seventy five percent of the participants assigned a confidence code of “C” to the vitamin K in lettuce, while 70% and 100% assigned confidence codes of “B” and “C” to catechin in black grapes and riboflavin in mushrooms, respectively. The wide range of scores on individual categories by first time evaluators emphasizes the importance of training for evaluators who score analytical values for inclusion in a food composition database. Clear documentation by the authors and training for evaluators to understand basic concepts related to data quality evaluation will be helpful. Future work will assess the assignment of equal rating points for all the categories. Modifications in “sampling plan” category to accommodate country and population size are necessary. The USDA Data Quality Evaluation System represents one of the first efforts to standardize and harmonize the evaluation of analytical data quality across the international food composition network.