Assessing Civic Competency and Engagement in Higher Education: Research Background, Frameworks, and Directions for Next‐Generation Assessment

Civic learning is increasingly recognized as important by the higher education and workforce communities. The development of high-quality assessments that can be used to evaluate students' civic learning during the college years has become a priority. This paper presents a comprehensive review of existing frameworks, definitions, and assessments of civic-related constructs from approximately 30 projects relevant to higher education, and includes a discussion of the challenges related to assessment design and implementation. Synthesizing information from the review, we propose an assessment framework to guide the design of a next-generation assessment of individuals' civic learning that takes advantage of recent advances in assessment methods. The definition identifies 2 key domains within civic learning: civic competency and civic engagement. Civic competency encompasses 3 areas (civic knowledge; analytic skills; and participatory and involvement skills), and civic engagement also captures 3 areas (motivations, attitudes, and efficacy; democratic norms and values; and participation and activities). We discuss item formats and task types that would ensure fair and reliable scoring for the assessment. The review of definitions of civic learning and its components developed by organizations, the proposed assessment framework, and assessment considerations presented here have potential benefits for a range of higher education institutions. This includes institutions that currently have students engaged in relevant curricular or cocurricular activities and also institutions that would find assessments of civic competency and engagement helpful in program development or in evaluating students' accomplishments.

[1]  O. Liu MEASURING LEARNING OUTCOMES IN HIGHER EDUCATION USING THE MEASURE OF ACADEMIC PROFICIENCY AND PROGRESS (MAPP) , 2008 .

[2]  C. Shealy A model and method for "making" a Combined-Integrated psychologist: equilintegration (EI) theory and the beliefs, events, and values inventory (BEVI). , 2004, Journal of clinical psychology.

[3]  J. Pekar,et al.  Meta-analysis of Go/No-go tasks demonstrating that fMRI activation associated with response inhibition is task-dependent , 2008, Neuropsychologia.

[4]  Roger Tourangeau,et al.  What They See Is What We Get , 2004 .

[5]  Nilza Costa,et al.  Bridging the Gap Between Conceptualisation & Assessment of Intercultural Competence☆ , 2012 .

[6]  J. Trefil,et al.  The new dictionary of cultural literacy , 2002 .

[7]  Mariana Norel,et al.  Explicit and Implicit Assessment of Intercultural Competence , 2013 .

[8]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[9]  Mark C. Nicholas,et al.  Faculty Approaches to Assessing Critical Thinking in the Humanities and the Natural and Social Sciences: Implications for General Education , 2013, The Journal of General Education.

[10]  V. Jeffries Handbook of public sociology , 2009 .

[11]  Heather Butler Halpern Critical Thinking Assessment Predicts Real-World Outcomes of Critical Thinking , 2012 .

[12]  Assessing the Psychometric Properties of Civic Values , 2011 .

[13]  Descriptors Higher Annual Meeting of the International Communication Association , 1974 .

[14]  M. Bazerman Judgment in Managerial Decision Making , 1990 .

[15]  Jennifer Hanson,et al.  Communicating across Cultures. , 1995 .

[16]  Y. Y. Kim Becoming Intercultural: An Integrative Theory of Communication and Cross-Cultural Adaptation , 2000 .

[17]  E. Rr Synthesizing Frameworks of Higher Education Student Learning Outcomes , 2013 .

[18]  P. Facione Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. Research Findings and Recommendations. , 1990 .

[19]  Robert L. Bangert-Drowns,et al.  Meta-Analysis of Effects of Explicit Instruction for Critical Thinking. , 1990 .

[20]  David A. Frisbie,et al.  Reliability of Scores From Teacher‐Made Tests , 1988 .

[21]  Gary Oddou,et al.  Measuring Global Leader Intercultural Competency: Development and Validation of the Global Competencies Inventory (GCI) , 2014 .

[22]  P. Earley,et al.  Cultural Intelligence: Individual Interactions Across Cultures , 2020 .

[23]  Jase R. Ramsey,et al.  Commitment to the Study of International Business and Cultural Intelligence: A Multilevel Model , 2014 .

[24]  Understanding the Cultural Orientations Approach : An Overview of the Development and Updates to the COA , 2012 .

[25]  Jessica T. Feezell,et al.  The Civic and Political Significance of Online Participatory Cultures among Youth Transitioning to Adulthood , 2013 .

[26]  Michael J. Jensen,et al.  Digital Media and Political Engagement Worldwide: Acknowledgments , 2012 .

[27]  Format effects on critical thinking test performance , 1995 .

[28]  Schon Beechler,et al.  What we talk about when we talk about ‘global mindset’: Managerial cognition in multinational corporations , 2007 .

[29]  M. Erez,et al.  Cultural Intelligence and Global Identity in Multicultural Teams , 2015 .

[30]  D. Butin,et al.  The Engaged Campus: Certificates, Minors, and Majors as the New Community Engagement , 2012 .

[31]  Gary N. Burns,et al.  Reconsidering Forced-Choice Item Formats for Applicant Personality Assessment , 2005 .

[32]  G. Hofstede,et al.  Culture′s Consequences: International Differences in Work-Related Values , 1980 .

[33]  Wayne Jacobson,et al.  Portfolio assessment of intercultural competence , 1999 .

[34]  R. Loo,et al.  A Psychometric Investigation of Scores on the Watson-Glaser Critical Thinking Appraisal New Form S , 1999 .

[35]  Tora K. Bikson,et al.  New Challenges for International Leadership: Lessons from Organizations with Global Missions , 2003 .

[36]  Phillip M. Mangos,et al.  Applicant Faking, Social Desirability, and the Prediction of Counterproductive Work Behaviors , 2011 .

[37]  David L. Amsbury,et al.  Detrital dolomite in central Texas , 1962 .

[38]  Paul R. Sackett,et al.  A META-ANALYSIS OF CONDITIONAL REASONING TESTS OF AGGRESSION , 2010 .

[39]  M. Banaji,et al.  PREDICTIVE VALIDITY OF THE IAT 1 RUNNING HEAD : PREDICTIVE VALIDITY OF THE IAT Understanding and Using the Implicit Association Test : III . Meta-analysis of Predictive Validity , 2006 .

[40]  Maarten A. S. Boksem,et al.  Please Scroll down for Article Social Neuroscience Facing Disapproval: Performance Monitoring in a Social Context Facing Disapproval: Performance Monitoring in a Social Context Performance Monitoring in a Social Context , 2022 .

[41]  R. Fischer Cross-cultural training effects on cultural essentialism beliefs and cultural intelligence. , 2011 .

[42]  Trudy H. Bers,et al.  The Disposition to Think Critically Among Community College Students: The California Critical Thinking Dispositions Inventory. , 1996 .

[43]  Robert J. Mislevy,et al.  DESIGN PATTERNS FOR IMPROVING ACCESSIBILITY FOR TEST TAKERS WITH DISABILITIES , 2008 .

[44]  Robert E. Ployhart,et al.  The State of Research on Situational Judgment Tests: A Content Analysis and Directions for Future Research , 2014 .

[45]  Thomas Ehrlich Civic Learning: "Democracy and Education" Revisited. , 1997 .

[46]  L. Engwall The Internationalisation of Higher Education , 2016, European Review.

[47]  Marjaana Gunkel,et al.  Working in the “Global Village”: The Influence of Cultural Intelligence on the Intention to Work Abroad , 2013 .

[48]  Chet Robie,et al.  Faking and the validity of conscientiousness: a Monte Carlo investigation. , 2008, The Journal of applied psychology.

[49]  Jonathan J. Mohr,et al.  Factor Structure and Short Form of the Miville-Guzman Universality-Diversity Scale , 2000 .

[50]  Michael G. Harvey,et al.  Executive Insights: An Intercultural Communication Model for Use in Global Interorganizational Networks , 2001 .

[51]  M. Erez,et al.  A Dynamic, Multi‐Level Model of Culture: From the Micro Level of the Individual to the Macro Level of a Global Culture , 2004 .

[52]  K. Rosenbusch,et al.  How global leaders develop , 2013 .

[53]  R. Eric Landrum,et al.  APA Guidelines for the Undergraduate Psychology Major, Version 2.0 , 2016 .

[54]  Richard L. Griffith,et al.  The Failure of Social Desirability Measures to Capture Applicant Faking Behavior , 2008, Industrial and Organizational Psychology.

[55]  Richard L. Griffith,et al.  Do applicants fake? An examination of the frequency of applicant faking behavior , 2007 .

[56]  Klaus J. Templer,et al.  Cultural Intelligence: Its Measurement and Effects on Cultural Judgment and Decision Making, Cultural Adaptation and Task Performance , 2007, Management and Organization Review.

[57]  Ying Zhang Expatriate Development for Cross-Cultural Adjustment , 2012 .

[58]  Meghan W. Brenneman,et al.  SYNTHESIZING FRAMEWORKS OF HIGHER EDUCATION STUDENT LEARNING OUTCOMES , 2013 .

[59]  DolanKathleen Do Women and Men Know Different Things? Measuring Gender Differences in Political Knowledge , 2011 .

[60]  Jennifer Suzanne Earl,et al.  Digital Media and Political Engagement Worldwide: A Comparative Study , 2014 .

[61]  B. Ruben Assessing Communication Competency for Intercultural Adaptation , 1976 .

[62]  Linking Diversity and Civic-Minded Practices with Student Outcomes: New Evidence from National Surveys , 2012 .

[63]  Paul F. Lazarsfeld,et al.  Voting. A Study of Opinion Formation in a Presidential Campaign , 1956 .

[64]  Jan Pieter van Oudenhoven,et al.  The multicultural personality questionnaire: a multidimensional instrument of multicultural effectiveness , 2000 .

[65]  S. Keeter,et al.  What Americans Know about Politics and Why It Matters , 1996 .

[66]  Kathryn S. Steinberg,et al.  Educating for Informed Community Involvement , 2010, American journal of community psychology.

[67]  David Matsumoto,et al.  Development and validation of a measure of intercultural adjustment potential in Japanese sojourners: the Intercultural Adjustment Potential Scale (ICAPS) , 2001 .

[68]  P. Behrens,et al.  The Watson-Glaser Critical Thinking Appraisal and academic performance of diploma school students. , 1996, The Journal of nursing education.

[69]  M. Maznevski,et al.  Globalization: Hercules Meets Buddha , 2017 .

[70]  Douglas R. Glasnapp,et al.  Predictive validity of critical thinking skills for initial clinical dental hygiene performance. , 2003, Journal of dental education.

[71]  Dana L. Kelly,et al.  International Association for the Evaluation of Educational Achievement , 1998 .

[72]  Jill Casner-Lotto,et al.  Are They Really Ready to Work? Employers' Perspectives on the Basic Knowledge and Applied Skills of New Entrants to the 21st Century U.S. Workforce. , 2006 .

[73]  Joseph A. Rios,et al.  Assessing Intercultural Competence in Higher Education: Existing Research and Future Directions , 2016 .

[74]  D. Halpern Thought and Knowledge: An Introduction to Critical Thinking , 1995 .

[75]  Paul L. Gaston,et al.  The Degree Qualifications Profile 2.0: Defining US Degrees through Demonstration and Documentation of College Learning. , 2014 .

[76]  P. Ewell,et al.  The Degree Qualifications Profile. Defining Degrees: A New Direction for American Higher Education to Be Tested and Developed in Partnership with Faculty, Students, Leaders and Stakeholders. , 2011 .

[77]  M. Adams,et al.  Critical thinking as an educational outcome: an evaluation of current tools of measurement. , 1996, Nurse educator.

[78]  Rebecca L. Stephens,et al.  Reliability and Validity of the Watson-Glasere Critical Thinking Appraisal-Forms for Different Academic Groups * , 2006 .

[79]  Shelley E. Taylor,et al.  Illusion and well-being: a social psychological perspective on mental health. , 1988, Psychological bulletin.

[80]  Guo-Ming Chen,et al.  The Development and Validation of the Intercultural Sensitivity Scale. , 2000 .

[81]  Howard Wainer,et al.  COMBINING MULTIPLE-CHOICE AND CONSTRUCTED RESPONSE TEST SCORES: TOWARD A MARXIST THEORY OF TEST CONSTRUCTION , 1992 .

[82]  Libby Gerard,et al.  Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles , 2014, Educational Measurement: Issues and Practice.

[83]  Mary K. Enright,et al.  Analytical Reasoning Skills in Graduate Study: Perceptions of Faculty in Six Fields. , 1987 .

[84]  S. Keeter,et al.  A New Engagement?: Political Participation, Civic Life, and the Changing American Citizen , 2006 .

[85]  Steven M. Downing,et al.  Handbook of test development , 2006 .

[86]  Joan Dejaeghere,et al.  Development of intercultural competence among US American teachers: professional development factors that enhance competence , 2008 .

[87]  M. W. Lustig,et al.  Intercultural Competence: Interpersonal Communication Across Cultures , 1998 .

[88]  William B. Gudykunst Cross-Cultural and Intercultural Communication , 2003 .

[89]  Robert G. Hanvey An Attainable Global Perspective. , 1982 .

[90]  L. Davies Global citizenship: abstraction or framework for action? , 2006 .

[91]  Mary F. Howard-Hamilton,et al.  Promoting Multicultural Education: A Holistic Approach. , 1998 .

[92]  D. Spencer The frankness of subjects on personality measures. , 1938 .

[93]  Gary N. Burns,et al.  Methods of Measuring Faking Behavior , 2011 .

[94]  P. Turner,et al.  Education for Citizenship and the Teaching of Democracy in Schools* , 1999 .

[95]  Winfred Arthur,et al.  The Magnitude and Extent of Cheating and Response Distortion Effects on Unproctored Internet-Based Tests of Cognitive Ability and Personality , 2010 .

[96]  Ryan A. Miller,et al.  Uncovering Categories of Civically Engaged College Students: A Latent Class Analysis , 2013 .

[97]  W. Stephan Intergroup Anxiety , 2014, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[98]  R. Callister,et al.  Is There a Global Warming Toward Women in Academia , 2005 .

[99]  A. Ryder,et al.  Measuring Campus Climate for Personal and Social Responsibility. , 2013 .

[100]  R. Heck,et al.  Evaluating the Impact of Multicultural Counseling Training , 1991 .

[101]  Kurt T. Taube Critical Thinking Ability and Disposition as Factors of Performance on a Written Critical Thinking Test. , 1995 .

[102]  김신아,et al.  Intercultural Competence , 2020, The SAGE Encyclopedia of Higher Education.

[103]  R. Bies,et al.  Organizational Citizenship Behavior: The Good Soldier Syndrome , 1989 .

[104]  P. M. King,et al.  A Developmental Model of Intercultural Maturity , 2005 .

[105]  Gerine M. A. Lodder,et al.  Loneliness and the social monitoring system: Emotion recognition and eye gaze in a real-life conversation. , 2016, British journal of psychology.

[106]  Bruna Zani,et al.  Political and civic engagement : multidisciplinary perspectives , 2014 .

[107]  D. Marlowe,et al.  A new scale of social desirability independent of psychopathology. , 1960, Journal of consulting psychology.

[108]  R. Traub,et al.  NCME Instructional Module: Understanding Reliability. , 1991 .

[109]  D. Paulhus Two-component models of socially desirable responding. , 1984 .

[110]  J. Conant General Education In A Free Society Report Of The Harvard Committee , 2011 .

[111]  J. Hochschild DEMOCRACY REMIXED: BLACK YOUTH AND THE FUTURE OF AMERICAN POLITICS , 2012 .

[112]  Martin Leever,et al.  Educating Citizens: Preparing America’s Undergraduates for Lives of Moral and Civic Responsibility , 2006 .

[113]  J. Stewart Black,et al.  The Influence of the Spouse on American Expatriate Adjustment and Intent to Stay in Pacific Rim Overseas Assignments , 1989 .

[114]  Robert H. Ennis Critical thinking assessment , 1993 .

[115]  G. Robinson,et al.  A Practical Guide for Integrating Civic Responsibility into the Curriculum. Second Edition. , 2002 .

[116]  Reginald Worthley,et al.  From Contact to Development in Experiential Cultural Intelligence Education: The Mediating Influence of Expectancy Disconfirmation , 2013 .

[117]  Otmar E. Varela,et al.  The Development of the Global Manager: An Empirical Study on the Role of Academic International Sojourns , 2014 .

[118]  L. Lawton,et al.  Intercultural Development: Study Abroad vs. On-Campus Study. , 2011 .

[119]  S. T. Hunt,et al.  How effective are people at faking on personality questionnaires , 2002 .

[120]  H. D. Haan Can internationalisation really lead to institutional competitive advantage? : a study of 16 Dutch public higher education institutions , 2014 .

[121]  Howard P. Smith,et al.  A Scale to Measure World-Minded Attitudes , 1957 .

[122]  K. Fellows,et al.  What’s your CQ? A thought leadership exploration of cultural intelligence in contemporary institutions of higher learning , 2016 .

[123]  B. Thacker Assessing Critical Thinking , 2007, Critical Thinking and Reasoning.

[124]  Bernd Kupka Creation of an instrument to assess intercultural communication competence for strategic international human resource management , 2008 .

[125]  Hugh Starkey,et al.  Vzgoja za demokraticno dr?avljanstvo: Pregled raziskav, politik in praks 1995-2005 (Education for Democratic Citizenship: a review of research, policy and practice 1995-2005) , 2006 .

[126]  Rebecca L. Stephens,et al.  Prediction of GPA with Educational Psychology Grades and Critical Thinking Scores , 2002 .

[127]  Philippa Collin,et al.  The civic web: young people, the internet and civic participation , 2015 .

[128]  Yi-Chun Lin,et al.  Does your intelligence help to survive in a foreign jungle? The effects of cultural intelligence and emotional intelligence on cross-cultural adjustment , 2012 .

[129]  Marianne Teräs,et al.  The Development of Teachers’ Intercultural Competence Using a Change Laboratory Method , 2013 .

[130]  Robert Gibson,et al.  The Sage Handbook of Intercultural Competence , 2011 .

[131]  Sandip Sinharay,et al.  How Often Do Subscores Have Added Value? Results from Operational and Simulated Data , 2010 .

[132]  R. Fischer,et al.  Assessing the predictive validity of cultural intelligence over time , 2011 .

[133]  R. Wiseman,et al.  Measuring intercultural sensitivity: The intercultural development inventory , 2003 .

[134]  Stefanie Rathje Intercultural Competence: The Status and Future of a Controversial Concept , 2007 .

[135]  Tracy Rundstrom Williams,et al.  Exploring the Impact of Study Abroad on Students’ Intercultural Communication Skills: Adaptability and Sensitivity , 2005 .

[136]  Kathleen M. Sheehan,et al.  Some Paths Toward Making Praxis Scores More Useful , 1998 .

[137]  T. B. Rogers,et al.  Development of a scale to measure cross-cultural sensitivity in the Canadian context. , 1993 .

[138]  R. Wiseman,et al.  Predictors of intercultural communication competence , 1989 .

[139]  Cathy Cohen Democracy Remixed: Black Youth and the Future of American Politics , 2010 .

[140]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[141]  James M. LeBreton,et al.  Measurement issues associated with conditional reasoning tests: indirect measurement and test faking. , 2007, The Journal of applied psychology.

[142]  Henry E. Brady,et al.  Voice and Equality: Civic Voluntarism in American Politics , 1996 .

[143]  P. Lazarsfeld,et al.  Voting: A Study of Opinion Formation in a Presidential Campaign. , 1955 .

[144]  M. Byram Teaching and Assessing Intercultural Communicative Competence , 1997 .

[145]  T. Urdan,et al.  Assessing Secondary Students’ Disposition toward Critical Thinking: Development of the California Measure of Mental Motivation , 2004 .

[146]  Thomas Ehrlich,et al.  Highlights from Educating for Democracy: Preparing Undergraduates for Responsible Political Engagement. , 2007 .

[147]  Michael A. McDaniel,et al.  Use of situational judgment tests to predict job performance: a clarification of the literature. , 2001, The Journal of applied psychology.

[148]  Brian Steensland :Citizen Speak: The Democratic Imagination in American Life , 2008 .

[149]  Thomas D. Snyder,et al.  Digest of Education Statistics, 2011. NCES 2012-001. , 2012 .

[150]  H. Kaufmann,et al.  Tailoring Cross‐Cultural Competence Training , 2014 .

[151]  A. Graf,et al.  Intercultural Training: Six Measures Assessing Training Needs. , 2009 .

[152]  M. Erez,et al.  Leadership emergence in multicultural teams: The power of global characteristics , 2013 .

[153]  F. Lievens,et al.  Situational judgment tests: A review of recent research , 2008 .

[154]  P. Earley,et al.  The Elusive Cultural Chameleon: Cultural Intelligence as a New Approach to Intercultural Training for the Global Manager , 2004 .

[155]  Julia Brandl,et al.  Applying cognitive adjustment theory to cross-cultural training for global virtual teams , 2009 .

[156]  S. Keeter,et al.  A New Engagement , 2006 .

[157]  Paula M. Winke,et al.  Investigating the Reliability of the Civics Component of the U.S. Naturalization Test , 2011 .

[158]  Bruno Fimm,et al.  A test battery for attentional performance , 2002 .

[159]  G. White,et al.  What Does It Mean to Be Globally Competent? , 2006 .

[160]  C. Murray,et al.  Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research , 2003, American Political Science Review.

[161]  Soon Ang,et al.  Personality Correlates of the Four-Factor Model of Cultural Intelligence , 2006 .

[162]  J. Lott Predictors of Civic Values: Understanding Student-Level and Institutional-Level Effects , 2013 .

[163]  Susan L. Davis,et al.  A Factor Analytic Study of the Cross-Cultural Adaptability Inventory , 2006 .

[164]  T. Salthouse,et al.  Influence of social desirability on age differences in self-reports of mood and personality. , 2011, Journal of personality.

[165]  Brian D. Blume,et al.  Individual- and organizational-level consequences of organizational citizenship behaviors: A meta-analysis. , 2009, The Journal of applied psychology.

[166]  S. Lane Validity of High‐Stakes Assessment: Are Students Engaged in Complex Thinking? , 2005 .

[167]  E. Rr Investigating 10-Year Trends of Learning Outcomes at Community Colleges , 2013 .

[168]  Guo-Ming Chen,et al.  Intercultural Communication Competence: A Synthesis , 1996 .

[169]  R. Paige,et al.  Education for the Intercultural Experience , 1994 .

[170]  Lily A. Arasaratnam The Development of a New Instrument of Intercultural Communication Competence , 2009, Journal of Intercultural Communication.

[171]  Susan A. Ostrander Democracy, Civic Participation, and the University: A Comparative Study of Civic Engagement on Five Campuses , 2004 .

[172]  H. Haste Citizenship Education: A Critical Look at a Contested Field , 2010 .

[173]  Craig D White,et al.  Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion. , 2014, The Journal of applied psychology.

[174]  Joseph A. Rios,et al.  Validity evidence based on internal structure. , 2014, Psicothema.

[175]  John A. Saltmarsh The Civic Promise of Service Learning. , 2005 .

[176]  F. Lievens,et al.  Situational Judgment Tests and their Predictiveness of College Students’ Success: The Influence of Faking , 2005 .

[177]  Seth Pollack,et al.  Critical Civic Literacy: Knowledge at the Intersection of Career and Community , 2013 .

[178]  Karen Hapgood,et al.  Intercultural learning in the classroom : crossing borders , 1997 .

[179]  Richard K. Rein Educational Testing Service. The Examiner Examined. , 1974 .

[180]  Michael C. Rodriguez,et al.  Developing and Validating Test Items , 2013 .

[181]  C. Leppa Standardized measures of critical thinking. Experience with the California Critical Thinking Tests. , 1997, Nurse educator.

[182]  S. Shavitt,et al.  Survey response styles across cultures , 2010 .

[183]  Charles J. Gelso,et al.  Appreciating similarities and valuing differences : The Miville-Guzman Universality-Diversity scale , 1999 .

[184]  Measurement issues associated with conditional reasoning tests: An examination of faking , 2013 .

[185]  N. Schmitt,et al.  Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. , 1997, The Journal of applied psychology.

[186]  Wolfram Schulz,et al.  IEA Civic Education Study: Technical Report , 2004 .

[187]  Leonard S. Cahen,et al.  Educational Testing Service , 1970 .

[188]  Psychometric Properties of the California Critical Thinking Tests , 2001, Journal of Nursing Measurement.

[189]  M. Kane The Argument-Based Approach to Validation , 1990 .

[190]  Jon Grahe,et al.  A National Survey of American Higher Education Capstone Practices in Sociology and Psychology , 2012 .

[191]  J. Harrison,et al.  Individual and combined effects of behavior modeling and the cultural assimilator in cross-cultural management training. , 1992 .

[192]  D. Halpern Teaching critical thinking for transfer across domains. Dispositions, skills, structure training, and metacognitive monitoring. , 1998, The American psychologist.

[193]  Kelly Y. L. Ku,et al.  Assessing Students' Critical Thinking Performance: Urging for Measurements Using Multi-Response Format. , 2009 .

[194]  A. Everett,et al.  Moscow on the Hudson: assessing expatriates' affective fit in host cultures with the intercultural communication affinity scale , 2008 .

[195]  J. Torney-Purta,et al.  Students' Conceptual Knowledge and Process Skills in Civic Education: Identifying Cognitive Profiles and Classroom Correlates , 2012 .

[196]  Michael C. Rodriguez Construct Equivalence of Multiple-Choice and Constructed-Response Items: A Random Effects Synthesis of Correlations , 2003 .

[197]  Gerard J. Fogarty,et al.  Predicting performance on a situational judgement test: the role of communication skills, listening skills, and expertise , 2005 .

[198]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[199]  Douglas Walton,et al.  Advances in the Theory of Argumentation Schemes and Critical Questions , 2008 .

[200]  Doreen Meier,et al.  Educating For Democracy Preparing Undergraduates For Responsible Political Engagement , 2016 .

[201]  Brent Bridgeman,et al.  Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country , 2012 .

[202]  Douglas R. Glasnapp,et al.  Predictive validity of critical thinking skills and disposition for the national board dental hygiene examination: a preliminary investigation. , 2006, Journal of dental education.

[203]  Richard D. Roberts,et al.  Is Intelligence Critical Thinking? Why We Need a New Definition of Intelligence , 2007 .

[204]  Terrel L. Rhodes Assessing outcomes and improving achievement : tips and tools for using rubrics , 2010 .

[205]  Deidra J. Schleicher,et al.  Cross-Cultural Social Intelligence: An Assessment for Employees Working in Cross-National Contexts , 2008 .

[206]  Michael X. Delli Carpini,et al.  Measuring Political Knowledge: Putting First Things First , 1993 .

[207]  G. Hofstede The GLOBE debate: Back to relevance , 2010 .

[208]  Allison Abbe,et al.  Cross-Cultural Competence in Army Leaders: A Conceptual and Empirical Foundation , 2007 .

[209]  Judith Meyers,et al.  Cross‐Cultural Adaptability Inventory , 2008 .

[210]  Julie A. Hatcher Assessing Civic Knowledge and Engagement. , 2011 .

[211]  C. Brewster,et al.  The Dimensions of Expatriate Adjustment , 2013 .

[212]  Todd M. Davis,et al.  Open Doors: Report on International Educational Exchange. , 2000 .

[213]  Randy Elliot Bennett,et al.  Online Assessment in Mathematics and Writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. , 2005 .

[214]  Paul Kline,et al.  A Psychometrics Primer , 2000 .

[215]  Richard L. Griffith,et al.  The Rules of Evidence and the Prevalence of Applicant Faking , 2011 .

[216]  Robert L. Engle,et al.  The impact of international experience on cultural intelligence: an application of contact theory in a structured short-term programme , 2014 .

[217]  E. Cleaver Citizenship Education Longitudinal Study: Literature Review-Citizenship Education One Year On – What Does it Mean?: Emerging Definitions and Approaches in the First Year of National Curriculum Citizenship in England , 2016 .

[218]  J. Tosh Public history. civic engagement and the historical profession in Britain , 2014 .

[219]  John E. Ingulsrud,et al.  The assessment of cross-cultural experience: measuring awareness through critical text analysis , 2002 .

[220]  I. Ajzen Nature and operation of attitudes. , 2001, Annual review of psychology.

[221]  Jon Bruschke,et al.  A test of anxiety/uncertainty management theory: The intercultural adaptation context , 1998 .

[222]  G. Stricker American Psychological Association Code of Ethics , 2010 .

[223]  M. Davidson,et al.  The Scale of Ethnocultural Empathy: Development, Validation, and Reliability , 2003 .

[224]  Joseph Kahne,et al.  What Kind of Citizen? The Politics of Educating for Democracy , 2004 .

[225]  Robert H. Ennis Critical Thinking and Subject Specificity: Clarification and Needed Research , 1989 .

[226]  P. Scott Globalisation and Higher Education: Challenges for the 21st Century , 2000 .

[227]  A. Perrin Citizen Speak: The Democratic Imagination in American Life , 2006 .

[228]  Miriam Erez,et al.  Global Work Culture and Global Identity, as a Platform for a Shared Understanding in Multicultural Teams , 2006 .

[229]  F. Brodbeck,et al.  The Influence of Social Identity and Personality on Outcomes of Cultural Diversity in Teams , 2004 .

[230]  Colleen Ward,et al.  The measurement of sociocultural adaptation , 1999 .

[231]  R. Brislin,et al.  The measurement of intercultural sensitivity using the concepts of individualism and collectivism , 1992 .

[232]  Gautam Puhan,et al.  Subscores Based on Classical Test Theory: To Report or Not to Report , 2007 .

[233]  C. M. Walsh,et al.  Factor Structure Stability of the California Critical Thinking Disposition Inventory across Sex and Various Students' Majors , 1997 .

[234]  J. Gross The Emerging Field of Emotion Regulation: An Integrative Review , 1998 .

[235]  Xiaoming Xi,et al.  A three-stage approach to the automated scoring of spontaneous spoken responses , 2011, Comput. Speech Lang..

[236]  Laura S. Hamilton,et al.  Measuring 21st Century Competencies: Guidance for Educators , 2013 .

[237]  Milton J. Bennett A developmental approach to training for intercultural sensitivity , 1986 .

[238]  S. Haberman,et al.  An NCME Instructional Module on Subscores , 2011 .

[239]  David M. Williamson,et al.  A Framework for Evaluation and Use of Automated Scoring , 2012 .

[240]  Robert M. Bernard,et al.  Exploring the structure of the Watson–Glaser Critical Thinking Appraisal: One scale or many subscales? , 2008 .

[241]  Elizabeth C. Ravlin,et al.  Development of the Cultural Intelligence Assessment , 2012 .

[242]  A. Meade Psychometric problems and issues involved with creating and using ipsative measures for selection , 2004 .

[243]  Marisol Navas,et al.  Relative Acculturation Extended Model (RAEM): New contributions with regard to the study of acculturation , 2005 .

[244]  J. J. Donovan,et al.  Do Warnings Not to Fake Reduce Faking? , 2003 .

[245]  Julie A. Hatcher,et al.  Civic-Minded Graduate: A North Star , 2011 .

[246]  L. Shepard The Role of Assessment in a Learning Culture , 2000 .

[247]  Tim Moore,et al.  Critical thinking and disciplinary thinking: a continuing debate , 2011 .

[248]  Ellen Middaugh,et al.  Democracy for Some: The Civic Opportunity Gap in High School. Circle Working Paper 59. , 2008 .

[249]  B. Brenner,et al.  Can Business Schools Make Students Culturally Competent? Effects of Cross-Cultural Management Courses on Cultural Intelligence , 2013 .

[250]  M. Berg The Framework for Qualifications of the European Higher Education Area , 2005 .

[251]  Leonie Koban,et al.  Brain systems underlying the affective and social monitoring of actions: An integrative review , 2014, Neuroscience & Biobehavioral Reviews.

[252]  R. Bataineh,et al.  Jordanian TEFL Graduate Students' Use of Critical Thinking Skills (as Measured by the Cornell Critical Thinking Test, Level Z) , 2006 .

[253]  Milton J. Bennett Towards Ethnorelativism: A Developmental Model of Intercultural Sensitivity , 1993 .

[254]  Lisa A. Seldomridge,et al.  California Critical Thinking Disposition Inventory: Further Factor Analytic Examination , 2007, Perceptual and motor skills.

[255]  Brent Bridgeman,et al.  Measuring Learning Outcomes in Higher Education , 2012 .

[256]  Mitchell R. Hammer THE INTERCULTURAL DEVELOPMENT INVENTORY A New Frontier in Assessment and Development of Intercultural Competence , 2012 .

[257]  E. Poutsma,et al.  Global management competencies: a theoretical foundation , 2010 .

[258]  J. Torney-Purta,et al.  Developing Citizenship Competencies from Kindergarten through Grade 12: A Background Paper for Policymakers and Educators , 2006 .

[259]  Alvino E. Fantini,et al.  Introduction-language, culture and world view: Exploring the nexus , 1995 .

[260]  Gene W. Gloeckner,et al.  The relationship of critical thinking to performance on the NCLEX-RN. , 2005, The Journal of nursing education.

[261]  I. Ajzen The theory of planned behavior , 1991 .

[262]  J. Bernstein Using “Think-Alouds” to Understand Variations in Political Thinking , 2010 .

[263]  Amy K. Syvertsen,et al.  Civic Measurement Models: Tapping Adolescents' Civic Engagement. CIRCLE Working Paper 55. , 2007 .

[264]  D. Buckingham,et al.  The Civic Web: Young People, the Internet, and Civic Participation , 2013 .

[265]  Ann Marie Ryan,et al.  Toward an Integrated Model of Applicant Faking Behavior , 2006 .

[266]  水野 里恵,et al.  脱慣習水準のモラル思考法の発達に「教育」が及ぼす影響--DIT-2(Defining Issues Test 2)測定による青年のモラル判断基準に関する研究 , 2009 .

[267]  Margaret G. Olebe,et al.  Exploring the cross-cultural equivalence of the behavioral assessment scale for intercultural communication , 1989 .

[268]  Marya L. Doerfel,et al.  Intercultural communication competence: Identifying key components from multicultural perspectives , 2005 .

[269]  Stanley S. Jacobs The Equivalence of Forms A and B of the California Critical Thinking Skills Test. , 1999 .

[270]  Nicholas A. Bowman Promoting Participation in a Diverse Democracy , 2011 .

[271]  J. Dewey Democracy and education : an introduction to the philosophy of education , 1961 .

[272]  M. Barrett The PIDOP Project: An overview , 2012 .

[273]  Mitchell R. Hammer Additional cross-cultural validity testing of the Intercultural Development Inventory ☆ , 2011 .

[274]  Dharm P. S. Bhawuk,et al.  Evolution of culture assimilators , 2001 .

[275]  U. Kim,et al.  Acculturation Attitudes in Plural Societies , 1989 .

[276]  T. Haladyna,et al.  Content-Related Validity Evidence in Test Development , 2011 .

[277]  Amy K. Syvertsen,et al.  Civic Measurement Models: Tapping Adolescents' Civic Engagement , 2007 .

[278]  Thomas Ehrlich,et al.  Educating Citizens: Preparing America's Undergraduates for Lives of Moral and Civic Responsibility , 2003 .

[279]  Phil Wood Confirmatory Factor Analysis for Applied Research , 2008 .

[280]  Mary L. Lanigan,et al.  Relational model of intercultural communication competence , 1989 .

[281]  Frank H Wilhelm,et al.  Seeing the Silver Lining: Cognitive Reappraisal Ability Moderates the Relationship between Stress and Depressive Symptoms Cognitive Reappraisal , 2022 .

[282]  D. Walton Argumentation Schemes for Presumptive Reasoning , 1995 .

[283]  D. Eignor The standards for educational and psychological testing. , 2013 .

[284]  Chris Reed,et al.  Argumentation Schemes , 2008 .

[285]  Wolfram Schulz,et al.  International civic and citizenship education study : assessment framework , 2008 .

[286]  M. Lazarova,et al.  Expatriate adjustment and performance: A critical review. , 2006 .

[287]  Robert G. Bringle,et al.  The Measure of Service Learning: Research Scales to Assess Student Experiences , 2003 .

[288]  Paula Caligiuri,et al.  Dynamic cross-cultural competencies and global leadership effectiveness , 2012 .

[289]  Alexandra F. Corning,et al.  Individual Orientation Toward Engagement in Social , 2002 .

[290]  Richard L. Griffith,et al.  Theoretical and Practical Advances in the Assessment of Cross-Cultural Competence , 2012 .

[291]  Stella Ting-Toomey,et al.  Facework competence in intercultural conflict: an updated face-negotiation theory , 1998 .

[292]  Geert Loosveldt,et al.  An Evaluation of the Effect of Response Formats on Data Quality in Web Surveys , 2002 .

[293]  Peter A. Facione,et al.  The California Critical Thinking Skills Test--College Level. Technical Report #4. Interpreting the CCTST, Group Norms, and Sub-Scores. , 1990 .

[294]  N. Nguyen,et al.  A Validation Study of the Cross-Cultural Adaptability Inventory , 2010 .

[295]  N. Frederiksen The real test bias: Influences of testing on teaching and learning. , 1984 .

[296]  Kimberly Hawkins Thinking and Reading Among College Undergraduates: An Examination of the Relationship between Critical Thinking Skills and Voluntary Reading , 2012 .

[297]  J. Torney-Purta,et al.  Promoting Political Competence and Engagement in College Students: An Empirical Study , 2006 .

[298]  Guo-ming Chen A Test of Intercultural Communication Competence , 1992 .

[299]  D. Jackson,et al.  The Impact of Faking on Employment Tests: Does Forced Choice Offer a Solution? , 2000 .

[300]  Gary Oddou,et al.  Defining the content domain of intercultural competence for global leaders , 2010 .

[301]  Paul L. Gaston,et al.  The Degree Qualifications Profile: A Learning-Centered Framework for What College Graduates Should Know and Be Able to Do to Earn the Associate, Bachelor's or Master's Degree. , 2014 .

[302]  Ou Lydia Liu,et al.  Assessing Critical Thinking in Higher Education: Current State and Directions for Next-Generation Assessment. Research Report. ETS RR-14-10. , 2014 .

[303]  Christopher J. Einolf,et al.  Quarterly Nonprofit and Voluntary Sector Nonprofit and Voluntary Sector Quarterly Gender Differences in the Correlates of Volunteering and Charitable Giving on Behalf Of: Association for Research on Nonprofit Organizations and Voluntary Action Gender Differences in the Correlates of Volunteering and , 2022 .

[304]  Chockalingam Viswesvaran,et al.  Meta-Analyses of Fakability Estimates: Implications for Personality Measurement , 1999 .

[305]  Elizabeth Beaumont The Challenge of Assessing Civic Engagement: What We Know and What We Still Need to Learn about Civic Education in College , 2005 .

[306]  S. Sinharay A Note on Assessing the Added Value of Subscores , 2013 .

[307]  A. Higgins-D’alessandro,et al.  Youth Civic Engagement: Normative Issues , 2010 .

[308]  Michael C. Pyryt Human cognitive abilities: A survey of factor analytic studies , 1998 .

[309]  Bruce W. Davidson,et al.  Assessing EFL Student Progress in Critical Thinking with the Ennis-Weir Critical Thinking Essay Test. , 1996 .

[310]  Elizabeth C. Ravlin,et al.  Past Experience, Cultural Intelligence, and Satisfaction With International Business Studies , 2013 .

[311]  P. Facione,et al.  Critical thinking disposition as a measure of competent clinical judgment: the development of the California Critical Thinking Disposition Inventory. , 1994, The Journal of nursing education.

[312]  Yi Cao,et al.  Developing U.S. teachers’ intercultural competence: Does professional development matter? , 2009 .

[313]  S. Beechler,et al.  Leading with a Global Mindset , 2007 .

[314]  K. Gasior,et al.  OECD: Education at a Glance 2012: OECD Indicators , 2013 .

[315]  Wolfram Schulz,et al.  Citizenship and Education in Twenty-Eight Countries: Civic Knowledge and Engagement at Age Fourteen. Executive Summary. , 2001 .

[316]  Thomas S. Barrows College Students' Knowledge and Beliefs: A Survey of Global Understanding. The Final Report of the Global Understanding Project. , 1981 .

[317]  Steven Andrew Light,et al.  Assessing Student Learning Outcomes and Documenting Success through a Capstone Course , 2010, PS: Political Science & Politics.

[318]  Martin Davies,et al.  Critical thinking and the disciplines reconsidered , 2013 .

[319]  K. Stanovich,et al.  On the relative independence of thinking biases and cognitive ability. , 2008, Journal of personality and social psychology.

[320]  Henry W. Lane,et al.  The Blackwell Handbook of Global Management: A Guide to Managing Complexity , 2006 .

[321]  Robert P. Tett,et al.  Personality tests at the crossroads: A response to Morgeson, Campion, Dipboye, Hollenbeck, Murphy, and Schmitt (2007). , 2007 .

[322]  Margaret G. Olebe,et al.  The behavioral assessment scale for intercultural communication effectiveness , 1988 .

[323]  R. Coelen The Internationalisation of Higher Education, 2.0 , 2013 .

[324]  Richard W. Brislin,et al.  Cross‐cultural Training: A Review , 2000 .

[325]  Peter T. Ewell,et al.  The Degree Qualifications Profile: What It Is and Why We Need It Now , 2013 .

[326]  Stephen J. Thoma,et al.  The defining issues test of moral judgment development , 2014 .

[327]  Ashley P. Finley How Reliable Are the VALUE Rubrics , 2011 .

[328]  Marcia C. Linn,et al.  Validating Measurement of Knowledge Integration in Science Using Multiple-Choice and Explanation Items , 2011 .

[329]  Michael A. McDaniel,et al.  Situational judgment tests: An overview of current research , 2009 .