Statistical Testing and Estimation

In Chap. 1, the QDA bridge is presented as an approach to experimental research. In this chapter, a pragmatic approach to statistical testing and estimation (PASTE) is presented. In line with the QDA heuristic introduced in Chap. 1, all statistical testing and estimation is driven by research questions and hypotheses and appropriately accounts for the features of the experimental design and the data acquired in the experiment. To get a first grip on the data, to check which assumptions may be reasonable, and to decide on how to proceed with the numbers, we ought to first carefully inspect our data with graphs and, based on what we see in these graphs, with appropriate simple descriptive statistics. These graphs and descriptive statistics, together with the research questions and hypotheses at hand and the features of the experimental design to be accounted for, help us to decide on appropriate ways to test our hypotheses and to estimate effects of interest. Point estimates are preferably accompanied by CIs and/or CRIs. Statistical testing is preferably done using multiple criteria, though which criteria to use partly depends on the nature of the hypotheses to be tested. In this chapter, different approaches to statistical significance testing as well as different approaches to hypothesis testing using information criteria and Bayesian hypothesis testing are discussed and compared, and concepts of sequential analysis and machine learning are discussed as well. After the introduction of each of the aforementioned concepts and methods, this chapter concludes with a coherent set of general guidelines that may serve as a general pragmatic approach to statistical testing and estimation. This approach, called PASTE, is also used in all subsequent chapters of this book. In a nutshell, PASTE is not about preferring one approach over another (e.g., Bayesian over statistical significance) but about using combinations of methods to make evidence-based decisions with regard to findings at hand and to formulate appropriate recommendations for future research and practice.