Exploring computational thinking assessment in introductory programming courses

Several instruments have been used for assessing Computational Thinking (CT) abilities. In this exploratory and preliminary study, we investigate how appropriate the Bebras challenge is as an instrument to assess and measure CT abilities. Bebras is an international challenge whose goal is to promote Computer Science and CT. The test can be answered without any prior knowledge on computer science. Our broad research question is whether we can evolve Bebras into a full fledged instrument to assess and measure CT abilities. In this paper, we instantiate a few more specific research questions: Is Bebras performance a good predictor of success for students within programming courses? Is there any correlation between Bebras performance and students' grades? Do students improve their performance in Bebras tests when exposed to the contents of a programming course? Our dataset consists of the grades of 138 students who attended introductory programming courses at two Brazilian universities and their performance in two simulated Bebras tests. The first test was applied at the beginning of the term and therefore before any exposition to programming classes. The second one was applied at the end of the term. The results suggest that the performance on Bebras is only moderately correlated to the student grades. We conclude that it is not very likely that CT measures can be derived from the Bebras test as it is currently designed. While further research is needed on how we can leverage the Bebras effort to extend it into a CT assessment instrument, we performed a preliminary study on the use of Item Response Theory (IRT) as a means to improve the selection of questions and the design of the test. We expect the results of this research can contribute both to the development and discussion on CT assessment as well as to the Bebras effort to educate CT.