The economist-as-audience needs a methodology of plausible inference

Economists often try to make plausible inferences from a sizable empirical literature addressing a particular measurement, direction-of-effect, or testing issue. There are serious methodological problems associated with drawing such inferences. This article sets out some of these problems in order to make a case for their importance. After discussing these problems, the paper presents three case study examples of inference difficulties in specific literatures. It then proposes a new hypothesis about the time pattern of publication bias in empirical economics literatures. As support for this hypothesis, it presents evidence that 'reversals in findings' in empirical literatures in economics are not uncommon. Similarities are pointed out between the focus on inference problems in this paper, and the meta-analysis literatures in psychology and medical clinical trials.

[1]  Robert S. Goldfarb,et al.  Does Studying Economics Discourage Cooperation? Watch What We Do, Not What We Say or How We Play , 1996 .

[2]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[3]  Philip Mirowski.  Three ways to think about testing in econometrics , 1995 .

[4]  R. Goldfarb.  If empirical work in economics is not severe testing, what is it? , 1995 .

[5]  Kevin J. Lansing.  Is Public Capital Productive? A Review of the Evidence , 1995 .

[6]  V. Smith,et al.  Can Markets Value Air Quality? A Meta-Analysis of Hedonic Property Value Models , 1995, Journal of Political Economy.

[7]  R. G. Cummings,et al.  Homegrown Values and Hypothetical Surveys: Is the Dichotomous Choice Approach Incentive-Compatible? , 1995 .

[8]  Kevin T. Reilly.  Human Capital and Information: The Employer Size-Wage Effect , 1995 .

[9]  Expanding the role of methodology , 1994 .

[10]  W. Michael Hanemann,et al.  Valuing the Environment through Contingent Valuation , 1994 .

[11]  Betsy Jane Becker,et al.  Combining significance levels. , 1994 .

[12]  Helen R. Neill,et al.  Hypothetical Surveys and Real Economic Commitments , 1994 .

[13]  W. Randolph,et al.  Measuring Permanent Responses to Capital-Gains Tax Changes in Panel Data , 1994 .

[14]  Mark W. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, The American psychologist.

[15]  David Card,et al.  Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania , 1993 .

[16]  V. Kerry Smith,et al.  Hedonic models and air pollution: Twenty-five years and counting , 1993 .

[17]  Thomas Gilovich,et al.  Does Studying Economics Inhibit Cooperation , 1993 .

[18]  B. Mccallum.  Unit Roots in Macroeconomic Time Series: Some Critical Issues , 1993 .

[19]  Ivy E. Broder,et al.  Review of NSF Economics Proposals: Gender and Institutional Patterns , 1993 .

[20]  Kevin Lang,et al.  Are all Economic Hypotheses False? , 1992, Journal of Political Economy.

[21]  James L. Swofford,et al.  The Changing Empirical Definition of Money: A Comment , 1992, Journal of Political Economy.

[22]  Orley Ashenfelter,et al.  Estimates of the Economic Return to Schooling from a New Sample of Twins , 1992 .

[23]  Arnold C. Harberger,et al.  INTERFACE BETWEEN ECONOMIC TECHNIQUES AND ECONOMIC POLICY , 1992 .

[24]  D. Hamermesh.  Unemployment insurance for developing countries , 1992 .

[25]  D. Farley,et al.  Volume-Outcome Relationships and Inhospital Mortality: The Effect of Changes in Volume Over Time , 1992, Medical care.

[26]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[27]  R. Gelber,et al.  Meta-analysis: the fashion of summing-up evidence. Part I. Rationale and conduct. , 1991, Annals of oncology : official journal of the European Society for Medical Oncology.

[28]  J. Cordes,et al.  Policy Watch: Cutting Capital Gains Taxes , 1991 .

[29]  T. Bartik.  The Effects of Property Taxes and Other Local Public Policies on the Intrametropolitan Pattern of Business Location , 1991 .

[30]  J. Tatom,et al.  Public capital and private sector performance , 1991 .

[31]  R. Blank.  The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[32]  Peter F Kostiuk,et al.  Compensating Differentials for Shift Work , 1990, Journal of Political Economy.

[33]  V. Kerry Smith,et al.  Signals or Noise? Explaining the Variation in Recreation Benefit Estimates , 1990 .

[34]  Alicia Haydock Munnell,et al.  Why has productivity growth declined? Productivity and public investment , 1990 .

[35]  Allan Franklin,et al.  Experiment, right or wrong , 1990 .

[36]  Thomas A. Louis,et al.  An Assessment of Publication Bias Using a Sample of Published Clinical Trials , 1989 .

[37]  James A. Chalfant,et al.  The Changing Empirical Definition of Money: Some Estimates from a Model of the Demand for Money Substitutes , 1989, Journal of Political Economy.

[38]  Robert Cameron Mitchell,et al.  Using Surveys to Value Public Goods: The Contingent Valuation Method , 1989 .

[39]  David Alan Aschauer,et al.  IS PUBLIC EXPENDITURE PRODUCTIVE , 1989 .

[40]  David Glasner.  Free Banking and Monetary Reform: The Keynesian revolution and the Monetarist counterrevolution , 1989 .

[41]  F. Denton.  The consequences of economic rhetoric: The significance of significance: Rhetorical aspects of statistical hypothesis testing in economics , 1989 .

[42]  G. Tellis.  The Price Elasticity of Selective Demand: A Meta-Analysis of Econometric Models of Sales , 1988 .

[43]  Larry V. Hedges,et al.  How hard is hard science, how soft is soft science? The empirical cumulativeness of research. , 1987 .

[44]  J. Hausman,et al.  Household Behavior and the Tax Reform Act of 1986 , 1987 .

[45]  P. Perron,et al.  Trends and random walks in macroeconomic time series : Further evidence from a new approach , 1988 .

[46]  R. Hall,et al.  Chapter 17 Cyclical fluctuations in the labor market , 1986 .

[47]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[48]  Andrew Weiss,et al.  Wages, Hiring Standards, and Firm Size , 1984, Journal of Labor Economics.

[49]  J. Farley,et al.  How Advertising Affects Sales: Meta-Analysis of Econometric Results , 1984 .

[50]  R. Rosenthal.  Meta-analytic procedures for social research , 1984 .

[51]  Frederic S. Mishkin.  A Rational Expectations Approach To Macroeconometrics , 1983 .

[52]  Edward E. Leamer,et al.  Let's Take the Con Out of Econometrics , 1983 .

[53]  D. Leimer,et al.  Social Security and Private Saving: New Time-Series Evidence , 1982, Journal of Political Economy.

[54]  Frederic S. Mishkin.  Does Anticipated Monetary Policy Matter? An Econometric Investigation , 1980, Journal of Political Economy.

[55]  C. Nelson,et al.  Trends and random walks in macroeconmic time series: Some evidence and implications , 1982 .

[56]  Charles Brown,et al.  The Effect of the Minimum Wage on Employment and Unemployment: A Survey , 1982 .

[57]  M. Blaug.  The Methodology of Economics , 1981 .

[58]  G. Marwell,et al.  Economists free ride, does anyone else? : Experiments on the provision of public goods, IV , 1981 .

[59]  J. Behrman,et al.  Socioeconomic Success: A Study of the Effects of Genetic Endowments , 1980 .

[60]  A. Flood,et al.  The Relationship Between Intensity and Duration of Medical Services and Outcomes for Hospitalized Patients , 1979, Medical care.

[61]  Kim B. Clark,et al.  Labor Market Dynamics and Unemployment: A Reconsideration , 1979 .

[62]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[63]  Tjalling C. Koopmans,et al.  Economics among the Sciences , 1979 .

[64]  Edward E. Leamer,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[65]  R. Barro.  Unanticipated Money, Output, and the Price Level in the United States , 1978, Journal of Political Economy.

[66]  Anthony M. Santomero,et al.  The Inflation-Unemployment Trade-off: A Critique of the Literature , 1978 .

[67]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[68]  M. Boskin.  Taxation, Saving and the Rate of Interest , 1976 .

[69]  M. Feldstein,et al.  Social Security, Induced Retirement, and Aggregate Capital Accumulation , 1974, Journal of Political Economy.

[70]  H. Johnson.  REFLECTIONS ON CURRENT TRENDS IN ECONOMICS , 1971 .

[71]  H. Johnson,et al.  The Keynesian Revolution and the Monetarist Counter-Revolution , 1971 .

[72]  R. Hall.  Why Is the Unemployment Rate So High at Full Employment , 1970 .

[73]  G. Harman.  The Inference to the Best Explanation , 1965 .

[74]  Shinzaburo Koshimura,et al.  The Relation between Unemployment and the Rate of Change of Money Wage Rates in the United Kingdom, 1862-1957: A Further Analysis , 1960 .

[75]  A. W. Phillips,et al.  A. W. H. Phillips: Collected Works in Contemporary Perspective: The relation between unemployment and the rate of change of money wage rates in the United Kingdom, 1861-1957 , 2000 .