Methodological Rigor and Theoretical Foundations of CS Education Research

The problem of the lack of rigor in CS education research has frequently been discussed and examined. Previous reviews of the literature have examined rigor on both theoretical and methodological dimensions, among others. These reviews have also looked at differences in indicators of rigor between conference proceedings and journal publications. However, to date there is no comprehensive review that has examined the intersection of methodological and theoretical quality. This paper reports results from a literature review in which we analyzed both the use of theory and methodological rigor of four years of CS education research from the Computer Science Education (CSE) journal and the proceedings of the International Computing Education Research (ICER) conference. The goal was to provide an updated and expanded picture of the methodological quality and use of theory in the most rigorous CS education publications, as well as to compare between conference proceedings and journal publications on these dimensions. Our focus was on research that draws upon learning theory from education, psychology and other disciplines outside CS education. The results of our review show a different picture than earlier reviews. Focus on empirical results in conference proceedings articles has surpassed that of journal publications, and empirical studies are significantly more likely to make use of theory from outside CS education. Overall, our analysis shows a significant increase in the proportion of articles drawing on theory from outside CS education, compared to earlier literature reviews, whereas indicators of methodological quality show no such change.