Analysis of incomplete and inconsistent clinical survey data

It is common for clinical data in survey trials to be incomplete and inconsistent for several reasons. Inconsistent data occur when more than one set of exclusive alternative questions are answered. One objective of this study was to identify and eliminate inconsistent data as an important data mining preprocessing step. We define three types of incomplete data: missing data due to skip pattern (SPMD), undetermined missing data (UMD), and genuine missing data (GMD). Identifying the type of missing data is another important objective as all missing data types cannot be treated the same. This goal cannot be achieved manually on large data of complex surveys since each subject should be processed individually. The analyses are accomplished in a mathematical framework by exploiting graph theoretic structure inherent in the questionnaire. An undirected graph is built using mutually inconsistent responses as well as its complement. The responses not in the largest maximal clique of complement graph are considered inconsistent. This guarantees removing as few responses as possible so that remaining ones are mutually consistent. Further, all potential paths in questionnaire’s graph are considered, based on the responses of subjects, to identify each type of incomplete data. Experiments are conducted on MESA data. Results show 15.4 % GMD, 9.8 % SPMD, 12.9 % UMD, and 0.021 % inconsistent data. Further utility of the approach is using a) the SPMD for data stratification, and b) inconsistent data for noise estimation. Proposed method is a preprocessing prerequisite for any data mining of clinical survey data.