Does an Academic Research Paper Contain Useful Knowledge? No. (p<.05)

Assume that you are handed a paper randomly selected from an academic journal in your field. What is the likelihood that it contains useful knowledge? Along with Ruth Pagell, a librarian for the Goizetta Business School at Emory University, I conducted an analysis to estimate the percentage of published papers in forecasting that contain useful knowledge (Armstrong and Pagell 2003). We defined useful knowledge as evidence that could contribute to better decisions than would have otherwise been made in given situations. We concluded that only 3% met this definition. My opinion, supported by an analysis of literature in marketing (Armstrong 2003), is that the percentage of useful papers in marketing is even lower. When I posted a message to this effect on ELMAR (a listserve for marketing academics and practitioners), some sent messages with concurring opinions, and there were no messages from those who objected. [The essay is in full text at http://jscottarmstrong.com under ELMAR, February 27, 2004.] Now, if less than 3% of published papers are important, should we apply the usual statistical rules and assume that a paper is unimportant since p is less than .05? Practitioners seem to make this assumption, as few read the academic literature. Academicians behave in a similar manner when it comes to applying research studies relevant to their roles as teachers and researchers. For example, despite research showing that teacher evaluations are detrimental to learning, reduce the quality of services rendered to students, reduce student satisfaction, harm morale among faculty, and are expensive, I am unaware of any schools that have acted on this research. (I had asked for such examples in an Essay that I posted on ELMAR.) 1. The Effects of Publishing Rankings of Journals Mort, et al. (2004) provide descriptive evidence on the perception of academic journals. They do not translate this into advice and they recognize that there are concerns about how to use their findings. I expect readers will come away with the conclusions that researchers should send their papers to the most prestigious journals, and that schools should judge the quality of a paper by the reputation of the journal in which it appears. This might be helpful advice for those who are trying to advance their careers, but does it help to advance the development of useful knowledge? I think not. Published rankings of journals are likely to increase the number of submissions to the …