Quantifying survey expectations: What's wrong with the probability approach?

We study a matched sample of individual stock market forecasts consisting of both qualitative and quantitative forecasts. This allows us to test for the quality of forecast quantification methods by comparing quantified qualitative forecasts with actual quantitative forecasts. Focusing mainly on the widely used quantification framework advocated by Carlson and Parkin (1975), the so-called "probability approach", we find that quantified expectations derived from the probability approach display a surprisingly weak correlation with reported quantitative stock return forecasts. We trace the reason for this low correlation to the importance of asymmetric and time-varying thresholds, whereas individual heterogeneity across forecasters seems to play a minor role. Hence, our results suggest that qualitative survey data may not be a very useful device to obtain quantitative forecasts and we suggest ways to remedy this problem when designing qualitative surveys.

[1]  Martin Weale,et al.  The utility of expectational data: Firm-level evidence using matched qualitative-quantitative UK surveys , 2011 .

[2]  Winfried Pohlmeier,et al.  Using Forecasts of Forecasters to Forecast , 2007 .

[3]  M. Weale,et al.  Uncertainty in UK manufacturing: evidence from qualitative survey data , 2007 .

[4]  Gunter Löffler.  Refining the Carlson–Parkin method , 1999 .

[5]  Henri Theil,et al.  On the Time Shape of Economic Microvariables and the Munich Business Test , 1952 .

[6]  M. Hashem Pesaran,et al.  The Limits to Rational Expectations , 1988 .

[7]  Helmut Seitz,et al.  The estimation of inflation forecasts from business survey data , 1988 .

[8]  Michael Schröder,et al.  Heterogeneity in Exchange Rate Expectations: Evidence on the Chartist-Fundamentalist Approach , 2008, Social Science Research Network.

[9]  Andrea Buraschi,et al.  Differences in beliefs and currency risk premiums , 2010 .

[10]  Kajal Lahiri,et al.  A Comparative Study of Alternative Methods of Quantifying Qualitative Survey Responses Using NAPM Data , 1992 .

[11]  Jan Marc Berk,et al.  Measuring inflation expectations: a survey data approach , 1999 .

[12]  J. Wolfers,et al.  Disagreement about Inflation Expectations , 2003, NBER Macroeconomics Annual.

[13]  Eric Ghysels,et al.  The impact of risk and uncertainty on expected returns. , 2009 .

[14]  Steffen R. Henzel,et al.  An Alternative to the Carlson-Parkin Method for the Quantification of Qualitative Inflation Expectations: Evidence from the Ifo World Economic Survey , 2005 .

[15]  James Mitchell,et al.  Forecasting Manufacturing Output Growth Using Firm-Level Survey Data , 2005 .

[16]  Assessing the Rationality of Survey Expectations: The Probability Approach , 2008 .

[17]  M. H. Pesaran,et al.  Formation of Inflation Expectations in British Manufacturing Industries , 1985 .

[18]  K. Lahiri,et al.  On the estimation of inflationary expectations from qualitative responses , 1981 .

[19]  Michela Nardo,et al.  The Quantification of Qualitative Survey Data: A Critical Assessment , 2003 .

[20]  Roy Batchelor,et al.  Inflation Expectations Revisited , 1988 .

[21]  James B. Mitchell,et al.  The Use of Non-Normal Distributions in Quantifying Qualitative Survey Data , 2002 .

[22]  Christian Müller.  You CAN Carlson–Parkin , 2010 .

[23]  James Mitchell,et al.  Quantification of Qualitative Firm-Level Survey Data , 2002 .

[24]  M. Hashem Pesaran,et al.  Chapter 14 Survey Expectations , 2006 .

[25]  Roy Batchelor,et al.  Quantitative v. Qualitative Measures of Inflation Expectations , 2009 .

[26]  Andrew J. Patton,et al.  Why do forecasters disagree? Lessons from the term structure of cross-sectional dispersion , 2010 .

[27]  Óscar Clavería González,et al.  Business and consumer expectations and macroeconomic forecasts , 2006 .

[28]  M. Weale,et al.  Qualitative business surveys: signal or noise? , 2011 .

[29]  Jeff Dominitz,et al.  Comparing predictions and outcomes: theory and application to income changes. , 1999 .

[30]  Roy Batchelor,et al.  Expectations, output and inflation: The European experience , 1982 .

[31]  Allan Timmermann,et al.  Estimation and Testing of Forecast Rationality under Flexible Loss , 2005 .

[32]  Kenneth A. Froot,et al.  Chartists, Fundamentalists, and Trading in the Foreign Exchange Market , 1989 .

[33]  Thomas Maag.  On the Accuracy of the Probability Method for Quantifying Beliefs About Inflation , 2009 .

[34]  Oskar Anderson join.  The Business Test of the IFO-Institute for Economic Research, Munich, and Its Theoretical Model , 1952 .

[35]  Ross A. Williams,et al.  Quantitative versus qualitative measures of price expectations: The evidence from Australian consumer surveys , 1979 .

[36]  U. Fritsche,et al.  The Dynamics of European Inflation Expectations , 2008 .

[37]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .