Contributions to assess the reproducibility and the agreement of respondents in CATA tasks

Abstract In consideration of the widespread use of check-all-that-apply (CATA) questions in sensory and consumer research, the investigation of how the panel performs in this methodology is deemed necessary. Checking the reliability of this type of data is relevant, mostly in situations where, as a rapid method, it is obtained from consumers and in less controlled conditions such as online studies. While an excellent performance is expected and required from trained subjects, when working with consumers it is convenient to explore the data obtained, since this can also hint at how seriously they are taking the task, their levels of fatigue, boredom, etc. In this work, we have developed some complementary tools to existing ones to be able to evaluate statistically the reproducibility of both the respondents and the panel. The assessment of the reliability of the panel would not be complete without other methodologies dedicated to the reproducibility at the panel level of the products and of the terms, and to the assessment of the agreements between pairs of respondents, and between each respondent and the panel solution. For those criteria, methodologies based on multiple factor analysis on contingency tables and McNemar test are presented. To illustrate the methodology developed, the statistical complements were applied on data from two studies. The studies, one involving emotions and another one sensory characterisation data, were conducted in two different test–retest sessions.