A COMPARISON OF THREE COMMONLY USED METHODS FOR TREATING NO PREFERENCE VOTES

The treatment of no preference votes continues to be an issue in sensory science, especially as the proper treatment of these votes has recently gained importance in advertising claims support. There are currently three main methods in common use: dropping the no preference votes, splitting the votes equally and splitting the votes proportionally according to the results among those who expressed a preference. The analyses then proceed as if the data were binomially distributed. In this paper, we compare these methods with respect to power and type I error. We show that proportional splitting returns more false alarms than expected and hence should not be used. We then discuss the meaningful interpretation of statistical significance in the presence of large numbers of no preference votes before providing general recommendations and indicating a promising direction of future research in this area. PRACTICAL APPLICATIONS When researchers choose a specific method of data analysis, they implicitly choose the effect of that analysis on their conclusions. Thus, it is important to compare rival methods of analysis where they exist. In the case of handling no preference votes, three analytic methods are in common use. These methods – dropping, equal splitting and proportional splitting – are compared in this paper with respect to power and type I error. This comparison, together with the ensuing discussion and recommendations, helps researchers make more informed decisions in their choice of analytic method for data that includes no preference votes.

[1]  G. A. Baker,et al.  ERRORS OF THE SECOND KIND IN ORGANOLEPTIC DIFFERENCE TESTING , 1954 .

[2]  Harry T. Lawless,et al.  Sensory Evaluation of Food , 1999 .

[3]  Jian Bi,et al.  THE POWER OF SENSORY DISCRIMINATION METHODS USED IN REPLICATED DIFFERENCE AND PREFERENCE TESTS , 1999 .

[4]  H. A. David,et al.  Ties in Paired-Comparison Experiments Using a Modified Thurstone-Mosteller Model , 1960 .

[5]  P. Schlich.  Risk tables for discrimination tests , 1993 .

[6]  M. O'Mahony,et al.  Corroborating the 2-AFC and 2-AC Thurstonian models using both a model system and sparkling water , 2004 .

[7]  A. Mood,et al.  The statistical sign test. , 1946, Journal of the American Statistical Association.

[8]  SOME STATISTICAL CONSIDERATIONS IN ORGANOLEPTIC RESEARCH: TRIANGLE, PAIRED, DUO‐TRIO TESTS , 1957 .

[9]  P. V. Rao,et al.  Ties in Paired-Comparison Experiments: A Generalization of the Bradley-Terry Model , 1967 .

[10]  Daniel M. Ennis,et al.  Accounting for no difference/preference responses or ties in choice experiments , 2012 .

[11]  Albrecht Irle,et al.  Note on the Sign Test in the Presence of Ties , 1980 .

[12]  J. Hemelrijk,et al.  A theorem on the sign test when ties are present , 1952 .

[13]  Per B. Brockhoff,et al.  The Design of Replicated Difference Tests , 2002 .

[14]  J. Putter.  The Treatment of Ties in Some Nonparametric Tests , 1955 .

[15]  R. Davidson.  On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments , 1970 .

[16]  N. T. Gridgeman.  Pair Comparison, with and without Ties , 1959 .

[17]  Daniel M. Ennis,et al.  THE POWER OF SENSORY DISCRIMINATION METHODS , 1993 .

[18]  Michael O'Mahony,et al.  Be your own placebo: A double paired preference test approach for establishing expected frequencies , 2007 .

[19]  Michael O'Mahony,et al.  CONSUMERS REPORT PREFERENCES WHEN THEY SHOULD NOT: A CROSS‐CULTURAL STUDY , 2003 .

[20]  Per B. Brockhoff,et al.  Estimation of the Thurstonian model for the 2-AC protocol , 2012 .

[21]  C. W. Coakley,et al.  Versions of the Sign Test in the Presence of Ties , 1996 .

[22]  Daniel M. Ennis,et al.  Relative power of difference testing methods in sensory evaluation , 1990 .

[23]  Michael O'Mahony,et al.  The paired preference test and the 'No Preference' option : Was odesky correct? , 2005 .

[24]  Michael O'Mahony,et al.  PAIRED PREFERENCE TESTS USING PLACEBO PAIRS AND DIFFERENT RESPONSE OPTIONS FOR CHIPS, ORANGE JUICES AND COOKIES , 2008 .

[25]  Harry T. Lawless,et al.  SOURCES OF ERROR AND THE NO-PREFERENCE OPTION IN DAIRY PRODUCT TESTING , 2005 .

[26]  Jian Bi,et al.  SIMILARITY TESTS USING FORCED-CHOICE METHODS IN TERMS OF THURSTONIAN DISCRIMINAL DISTANCE, d′ , 2011 .

[27]  John M. Ennis,et al.  THE POWER OF SENSORY DISCRIMINATION METHODS REVISITED , 2011 .

[28]  H. Meiselman,et al.  Longitudinal measurement of three eating behavior scales during a period of change , 1998 .

[29]  M. O'Mahony,et al.  Comparison of d′ values for the 2-AFC (paired comparison) and 3-AFC discrimination methods: Thurstonian models, sequential sensitivity analysis and power , 1998 .

[30]  Modelling Ties in the Sign Test , 1999, Biometrics.

[31]  Stanford H. Odesky.  Handling the Neutral Vote in Paired Comparison Product Testing , 1967 .