Comparing the Similarities and Differences of PISA 2003 and TIMSS

This paper makes an in-depth comparison of the PISA (OECD) and TIMSS (IEA) mathematics assessments conducted in 2003. First, a comparison of survey methodologies is presented, followed by an examination of the mathematics frameworks in the two studies. The methodologies and the frameworks in the two studies form the basis for providing explanations for the observed differences in PISA and TIMSS results. At the country level, it appears that Western countries perform relatively better in PISA as compared to their performance in TIMSS. In contrast, Asian and Eastern European countries tend to do better in TIMSS than in PISA. This paper goes beyond making mere conjectures about the observed differences in results between PISA and TIMSS. The paper provides supporting evidence through the use of regression analyses to explain the differences. The analyses showed that performance differences at the country level can be attributed to the content balance of the two tests, as well as the sampling definitions – age-based and grade-based – in PISA and TIMSS respectively. Apart from mathematics achievement, the paper also compares results from the two studies on measures of self-confidence in mathematics. Gender differences are also examined in the light of contrasting results from the two studies. Overall, the paper provides a comprehensive comparison between PISA and TIMSS, and, in doing so, it throws some light on the interpretation of results of large-scale surveys more generally. Le present document etablit une comparaison detaillee des evaluations des mathematiques PISA (OCDE) et TIMSS (IEA), toutes deux menees en 2003. Il presente tout d’abord une comparaison des methodologies d’enquete, puis un examen des cadres d’evaluation des mathematiques. C’est en effet par une analyse des methodologies et cadres d’evaluation des deux etudes que l’on peut expliquer les differences constatees dans les resultats du PISA et ceux du TIMSS. Au niveau des pays, il apparait que les nations occidentales reussissent relativement mieux a l’enquete PISA qu’a l’enquete TIMSS. En revanche, les pays d’Asie et d’Europe de l’Est ont tendance a obtenir de meilleures performances aux evaluations TIMSS qu’aux evaluations PISA. Au-dela de simples conjectures sur les differences observees, le present document fournit des elements de preuves en utilisant des analyses de regression qui expliquent les disparites de resultats. Les analyses ont demontre que les variations de performance au niveau national peuvent etre imputees a l’equilibre des contenus des deux tests, ainsi qu’aux definitions d’echantillonnage – fondees sur l’âge ou fondees sur la classe – pour PISA et pour TIMSS, respectivement. Outre les performances en mathematiques, le present document compare les resultats des deux etudes sur les mesures de la confiance en soi en mathematiques. Les differences entre les sexes sont egalement examinees a la lumiere des resultats contrastes des deux enquetes. Dans l’ensemble, ce document offre une comparaison complete entre PISA et TIMSS