A framework for optimising the cost and performance of concept testing

Abstract To anticipate the likely market demand better and identify the best customers to target with potential new products early in their development, concept tests need to provide adequate data quality for the objectives of measurement. Generalisability theory provides a framework where a generalisability study can be conducted to identify each possible aspect of a measurement as a factor that may be a potential source of variability. When a limited budget is available, optimising measurement designs involves a trade-off between the accuracy of the data (i.e. generalisability coefficients) and cost considerations. Building on previous and ongoing research, this study presents a multivariate optimisation procedure to achieve the most cost-efficient measurement design under the pre-specified generalisability coefficient constraint. We used the available online concept testing data to illustrate how to optimise the measurement cost by sampling along the facets that contribute to the total error variance in different iso-generalisability designs. The findings may help to facilitate decision making for the sampling of respondents, concepts, items, and occasions in the design of concept test. However, the optimisation framework may be applied more generally to improve the effectiveness and efficiency of other customer tests in marketing.

[1]  Gerard A. Athaide,et al.  An Illustration of Potential Sources of Concept‐Test Error* , 2006 .

[2]  Adam Finn,et al.  Whose crystal ball to choose? Individual differences in the generalizability of concept testing , 2010 .

[3]  Ling Peng,et al.  How Cloudy a Crystal Ball: A Psychometric Assessment of Concept Testing* , 2010 .

[4]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[5]  J. W. Taylor,et al.  The Purchase Intention Question in New Product Development: A Field Test , 1975 .

[6]  P. F. Sanders,et al.  Minimizing the number of observations: A generalization of the spearman-brown formula , 1989 .

[7]  V. Mahajan,et al.  New product models: Practice, shortcomings and desired improvements , 1992 .

[8]  J. A. Woodward,et al.  Maximizing the coefficient of generalizability in multi-facet decision studies , 1973 .

[9]  George A. Marcoulides,et al.  The Optimization of Multivariate Generalizability Studies with Budget Constraints , 1992 .

[10]  Charles R. Duke.  Understanding Customer Abilities in Product Concept Tests , 1994 .

[11]  George A. Marcoulides,et al.  Maximizing the Coefficient of Generalizability in Decision Studies , 1991 .

[12]  P. F. Sanders,et al.  Alternative solutions for optimization problems in generalizability theory , 1992 .

[13]  Yoram Wind,et al.  A New Procedure for Concept Evaluation , 1973 .

[14]  Ling Peng,et al.  How far can you Rely on a Concept Test: The Generalizability of Testing over Occasions , 2010 .

[15]  Adam Finn,et al.  Reliability Assessment and Optimization of Marketing Measurement , 1997 .

[16]  Noreen M. Webb,et al.  The Dependability and Interchangeability of Assessment Methods in Science , 2000 .

[17]  J. P. Peter.  Reliability: A Review of Psychometric Basics and Recent Marketing Practices , 1979 .

[18]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[19]  Ling Peng,et al.  Concept testing: the state of contemporary practice , 2008 .

[20]  A. Page,et al.  Developing an effective concept testing program for consumer durables , 1992 .

[21]  Muammer Ozer,et al.  A survey of new product evaluation models , 1999 .

[22]  Joseph O. Rentz.  Generalizability Theory: A Comprehensive Method for Assessing and Improving the Dependability of Marketing Measures , 1987 .

[23]  D. G. Morrison.  Purchase Intentions and Purchase Behavior , 1979 .

[24]  Edward M. Tauber.  Why Concept and Product Tests Fail to Predict New Product Results , 1975 .

[25]  George A. Marcoulides,et al.  Designing Measurement Studies Under Budget Constraints: Controlling Error of Measurement and Power , 1995 .

[26]  George A. Marcoulides,et al.  Selecting the Number of Observations in Multivariate Measurement Studies Under Budget Constraints , 1991 .

[27]  P. F. Sanders,et al.  Maximizing the coefficient of generalizability under the constraint of limited resources , 1991 .

[28]  George A. Marcoulides,et al.  Optimizing Measurement Designs with Budget Constraints: The Variable Cost Case , 1997 .

[29]  H. V. Trijp,et al.  Consumer research in the early stages of new product development: a critical review of methods and techniques , 2005 .

[30]  George A. Marcoulides,et al.  Selecting Weighting Schemes in Multivariate Generalizability Studies , 1994 .

[31]  E. E. Scheuing.  New Product Management , 1974 .