Validation Studies of a Questionnaire Developed to Measure Students’ Engagement With Systems Thinking

The purpose of this paper was to develop and validate a new theoretically based scale to measure students’ learning of systems thinking in relation to the affective domain in the context of systems engineering education. Two variant questionnaires are reported here, one using only questions constructed using positive grammar and the other using a mix of positive and negative constructs, each applied to a different sample. The first group of 186 participants completed the positive version of the questionnaire, and, the second group of 163 completed the mixed version. Construct validity was examined through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA was conducted to find the factors underlying each questionnaire. CFA was conducted to confirm the better questionnaire version and to confirm the factors which underlie both versions. The results indicate that a three factor, 16 item, scale with a mix of positive and negative wording is the better instrument with which to measure students’ engagement with systems thinking. The results also indicate that the three factor, 16 item construct is a better representative of both versions of the questionnaire, whether the questionnaire has only positive questions or a mix of positive and negative questions.

[1]  Linda D. Grooms,et al.  Development of an instrument to measure perceived cognitive, affective, and psychomotor learning in traditional and virtual classroom higher education settings , 2009, Internet High. Educ..

[2]  Martin Reynolds,et al.  Response to paper "Systems thinking" by D. Cabrera et al.: systems thinking from a critical systems perspective. , 2008, Evaluation and program planning.

[3]  H. Marsh,et al.  Assessing Goodness of Fit: Is Parsimony Always Desirable? , 1996 .

[4]  Richard Shelton Kirby.  Engineering in History , 1956 .

[5]  Moti Frank.  Assessing the interest for systems engineering positions and other engineering positions' required capacity for engineering systems thinking (CEST) , 2010 .

[6]  Meredith D. Gall,et al.  Educational Research: An Introduction , 1965 .

[7]  Caroline Marie Lamb,et al.  Collaborative systems thinking : an exploration of the mechanisms enabling team systems thinking , 2009 .

[8]  J. B. Norman,et al.  Educational Research Planning and Economic Change in Hungary , 1982 .

[9]  L. Christensen,et al.  Educational Research: Quantitative and Qualitative Approaches , 2000 .

[10]  Bill Cameron,et al.  Systems for Philosophy , 1995 .

[11]  Richard Stevens,et al.  Systems engineering : coping with complexity , 1998 .

[12]  Dick Schoech,et al.  Neuroscience, the Unconscious and Professional Decision Making: Implications for ICT , 2010 .

[13]  Nadia Kellam,et al.  Complex Systems: What Are They And Why Should We Care? , 2009 .

[14]  Ute Harms,et al.  Assessing System Thinking Through Different Concept-Mapping Practices , 2012 .

[15]  A. Bechara.  The role of emotion in decision-making: Evidence from neurological patients with orbitofrontal damage , 2004, Brain and Cognition.

[16]  Peter Checkland,et al.  Systems Thinking, Systems Practice , 1981 .

[17]  C. Robert Kenley,et al.  Exploring the Role of Design in Systems Engineering , 2015 .

[18]  Ervan G. Garrison,et al.  A History of Engineering and Technology Artful Methods , 1991 .

[19]  S. Quartz,et al.  Reason, emotion and decision-making: risk and reward computation with feeling , 2009, Trends in Cognitive Sciences.

[20]  David H. Cropley,et al.  Measuring Creativity for Innovation Management , 2011 .

[21]  Howard B. Lee,et al.  A first course in factor analysis , 1973 .

[22]  N. Ntoumanis,et al.  The Coach–Athlete Relationship Questionnaire (CART‐Q): development and initial validation , 2004, Scandinavian journal of medicine & science in sports.

[23]  P. Harmer,et al.  Confirmatory Factor Analysis of the Group Environment Questionnaire with an Intercollegiate Sample , 1996 .

[24]  D. Goncalves,et al.  Developing systems engineers , 2008, PICMET '08 - 2008 Portland International Conference on Management of Engineering & Technology.

[25]  Russell L. Ackoff,et al.  Creating the Corporate Future: Plan or be Planned For , 1981 .

[26]  Daniela Carlucci,et al.  Applying a systems thinking framework to assess knowledge assets dynamics for business performance improvement , 2012, Expert Syst. Appl..

[27]  Robert Cloutier,et al.  Comparing Perceptions of Competency Knowledge Development in Systems Engineering Curriculum: A Case Study , 2011 .

[28]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[29]  R. Dolan,et al.  Brain, emotion and decision making: the paradigmatic example of regret , 2007, Trends in Cognitive Sciences.

[30]  E. Jantsch.  The design of inquiring systems,: by C. West Churchman New York, Basic Books, 1972. , 1972 .

[31]  Derek Cabrera,et al.  A Unifying Theory of Systems Thinking with Psychosocial Applications , 2015 .

[32]  S. Turner,et al.  The SAGE Handbook of Social Science Methodology , 2007 .

[33]  Heidi L. Davidz,et al.  Enabling systems thinking to accelerate the development of senior systems engineers , 2008, Syst. Eng..

[34]  A. Smith,et al.  Research Methodology: A Step-by-step Guide for Beginners , 2012 .

[35]  Rolph E. Anderson,et al.  Multivariate Data Analysis (7th ed. , 2009 .

[36]  Heidi L. Davidz,et al.  Enabling systems thinking to accelerate the development of senior systems engineers , 2008 .

[37]  Iean Russell,et al.  Getting the big picture in natural resource management - Systems thinking as 'method' for scientists, policy makers and other stakeholders , 2007 .

[38]  Guangming Cao,et al.  A Path Model Linking Business Analytics, Data-Driven Culture, and Competitive Advantage , 2014, ECIS.

[39]  Jay W. Forrester,et al.  System dynamics, systems thinking, and soft OR , 1994 .

[40]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[41]  Yeshika Alversia,et al.  Doing quantitative research in education with SPSS , 2011 .

[42]  Andy Field,et al.  Discovering statistics using SPSS: and sex and drugs and rock 'n' roll, 3rd Edition , 2009 .

[43]  P. M'Pherson.  A perspective on systems science and systems philosophy , 1974 .

[44]  John Mingers,et al.  Critical Heuristics of Social Planning: A New Approach to Practical Philosophy , 1983 .

[45]  Shi Lei.  Introduction to system engineering of green process , 2003 .

[46]  Darren George,et al.  SPSS for Windows Step by Step: A Simple Guide and Reference , 1998 .

[47]  D. Cabrera,et al.  Systems thinking. , 2008, Evaluation and program planning.

[48]  Michael Griffin.  System Engineering and the "Two Cultures" of Engineering , 2007, IEEE Engineering Management Review.

[49]  Jason W. Osborne,et al.  Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. , 2005 .

[50]  S. J. Kline,et al.  Conceptual Foundations for Multidisciplinary Thinking , 2020 .

[51]  Art Pyster,et al.  Graduate Reference Curriculum for Systems Engineering (GRCSE) (presentation) , 2011 .

[52]  Yamil Sanchez Rivera.  Promoting Motivation through Mode of Instruction: The Relationship between Use of Affective Teaching Techniques and Motivation to Learn Science. , 2010 .

[53]  Timothy L. J. Ferris,et al.  Development and Initial Validation of an Instrument to Measure Students' Learning About Systems Thinking: The Affective Domain , 2018, IEEE Systems Journal.

[54]  W. Trochim,et al.  Practical challenges of systems thinking and modeling in public health. , 2006, American journal of public health.

[55]  Matthew D. Lammi.  Student Achievement and Affective Traits in Electrical Engineering Laboratories Using Traditional and Computer-Based Instrumentation , 2008 .

[56]  John Boardman,et al.  Systems Thinking: Coping with 21st Century Problems , 2008 .