Application of a panel performance reliability versus product effect size (PR-ES) framework: A protein powder case study

Abstract Performance reliability of sensory descriptive panels is critical to sensory analysis. Treatment effect size (ES) of products is a main objective of sensory analysis. The intraclass correlation coefficient (ICC) and Cronbach’s coefficient alpha can be used to measure performance reliability, while Thurstonian d-prime and R-index can be used to measure ES. In this paper, an industry case study with a full historical database spanning 10 years of protein studies is shared. This case study uses the analysis framework of Performance Reliability versus Effect Size (PR-ES). R codes for the analysis are developed, used, and provided in the paper.

[1]  Carla Kuesten,et al.  INTRACLASS CORRELATION COEFFICIENT (ICC): A FRAMEWORK FOR MONITORING AND ASSESSING PERFORMANCE OF TRAINED SENSORY PANELS AND PANELISTS , 2012 .

[2]  G. W. Snedecor.  STATISTICAL METHODS , 1967 .

[3]  Sadanori Konishi,et al.  Inferences on Multivariate Measures of Interclass and Intraclass Correlations in Familial Data , 1991 .

[4]  Michael O'Mahony,et al.  UNDERSTANDING DISCRIMINATION TESTS: A USER-FRIENDLY TREATMENT OF RESPONSE BIAS, RATING AND RANKING R-INDEX TESTS AND THEIR RELATIONSHIP TO SIGNAL DETECTION , 1992 .

[5]  H. Ahrens,et al.  Multivariate variance‐covariance components (MVCC) and generalized intraclass correlation coefficient (GICC) , 1976 .

[6]  Alard Verhoef,et al.  Introduction of a quality index, based on Generalizability theory, as a measure of reliability for univariate- and multivariate sensory descriptive data , 2015 .

[7]  L. Thurstone.  A law of comparative judgment. , 1994 .

[8]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[9]  R. A. Bradley,et al.  THE COMPARISON OF THE SENSITIVITIES OF SIMILAR EXPERIMENTS: APPLICATIONS* , 1957 .

[10]  L. Cronbach,et al.  THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY† , 1963 .

[11]  Jian Bi.  Sensory Discrimination Tests and Measurements: Sensometrics in Sensory Evaluation , 2015 .

[12]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[13]  M. Srivastava,et al.  Estimation of Interclass and Intraclass Correlations in Multivariate Familial Data , 1988 .

[14]  P. Brockhoff,et al.  Delta-tilde interpretation of standard linear mixed model results , 2016 .

[15]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[16]  P. Talsma.  Assessing sensory panel performance using generalizability theory , 2016 .

[17]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[18]  Jian Bi,et al.  AGREEMENT AND RELIABILITY ASSESSMENTS FOR PERFORMANCE OF SENSORY DESCRIPTIVE PANEL , 2003 .

[19]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[20]  Jian Bi,et al.  STATISTICAL ANALYSIS OF RECEIVER OPERATING CHARACTERISTIC (ROC) CURVES FOR THE RATINGS OF THE A-NOT A AND THE SAME-DIFFERENT METHODS , 2013 .

[21]  El Mostafa Qannari,et al.  A method for panelists’ consistency assessment in sensory evaluations based on the Cronbach’s alpha coefficient , 2014 .

[22]  M. O'Mahony,et al.  The 9-point hedonic scale: Using R-Index Preference Measurement to compute effect size and eliminate artifactual ties. , 2020, Food research international.