Technology and Assessment in Selection

As technology evolves and is incorporated into testing and assessment applications, psychologists have opportunities to improve testing processes making scores on measurement tools more accurate, the administration process more efficient, and often the assessment process more realistic. At the same time, they also must contend with fundamental changes in the assumptions made about good testing practices and confront new problems that are created by technological enhancements. Issues such as distractions and their implications for test performance, changes to the applicant pool and their effect on adverse impact, and cheating and its impact on test scores must be explored. This article provides examples of how technology is being used in tests and assessments, discusses the advantages and disadvantages of relying on technology, and offers an agenda for future research.

[1]  Robert E. Ployhart,et al.  WEB‐BASED AND PAPER‐AND‐PENCIL TESTING OF APPLICANTS IN A PROCTORED SETTING: ARE PERSONALITY, BIODATA, AND SITUATIONAL JUDGMENT TESTS COMPARABLE? , 2003 .

[2]  John E. Hunter,et al.  Job sample vs. paper-and-pencil trades and technical tests: Adverse impact and examinee attitudes. , 1977 .

[3]  Ben-Roy Do,et al.  Research on Unproctored Internet Testing , 2009, Industrial and Organizational Psychology.

[4]  K. Murphy,et al.  The Decisions Job Applicants Must Make: Insights From a Bayesian Perspective , 2004 .

[5]  Michael S. Fetzer,et al.  Computer Adaptive Testing (CAT): A Faster, Smarter, and More Secure Approach to Pre-Employment Testing , 2011 .

[6]  Philip L. Roth,et al.  A META‐ANALYSIS OF WORK SAMPLE TEST VALIDITY: UPDATING AND INTEGRATING SOME CLASSIC LITERATURE , 2005 .

[7]  Fritz Drasgow,et al.  Two-Step Testing in Employee Selection: Is Score Inflation a Problem? , 2008 .

[8]  C. Glas,et al.  Unproctored Internet Test Verification , 2011 .

[9]  Nicole L. Toldi.  Job applicants favor video interviewing in the candidate‐selection process , 2011 .

[10]  John L. Smith,et al.  Using the Internet for psychological research: personality testing on the World Wide Web. , 1999, British journal of psychology.

[11]  Traditional tests and job simulations: minority and majority performance and test validities. , 2001 .

[12]  D. Bartram.  The International Test Commission Guidelines on Computer-Based and Internet-Delivered Testing , 2009, Industrial and Organizational Psychology.

[13]  N. Schmitt,et al.  Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. , 1997, The Journal of applied psychology.

[14]  Bradford S. Bell,et al.  Justice Expectations and Applicant Perceptions , 2004 .

[15]  John A. Weiner,et al.  Unproctored Online Testing: Environmental Conditions and Validity , 2009, Industrial and Organizational Psychology.

[16]  J. Salgado,et al.  Fairness Reactions to Personnel Selection Techniques in Spain and Portugal , 2004 .

[17]  David J. Weiss,et al.  Improving Measurement Quality and Efficiency with Adaptive Testing , 1982 .

[18]  Paul R. Sackett,et al.  Fairness in selection: Current developments and perspectives , 1993 .

[19]  Filip Lievens,et al.  The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. , 2011, The Journal of applied psychology.

[20]  Kevin R. Murphy,et al.  When your top choice turns you down: Effect of rejected offers on the utility of selection tests. , 1986 .

[21]  Christopher D. Nye,et al.  Cheating on Proctored Tests: The Other Side of the Unproctored Debate , 2009, Industrial and Organizational Psychology.

[22]  Howard C. Nusbaum,et al.  Neuroimaging as a New Tool in the Toolbox of Psychological Science , 2008 .

[23]  Paul E. Levy,et al.  Form, content, and function: An evaluative methodology for corporate employment web sites , 2004 .

[24]  Kenneth Pearlman.  Unproctored Internet Testing: Practical, Legal, and Ethical Concerns , 2009, Industrial and Organizational Psychology.

[25]  Gary J. Lautenschlager,et al.  Computer administration of questions: More desirable or more social desirability? , 1990 .

[26]  Jinyan Fan,et al.  Testing the efficacy of a new procedure for reducing faking on personality tests within selection contexts. , 2012, The Journal of applied psychology.

[27]  John P. Hausknecht,et al.  Applicant Reactions to Selection Procedures: An Updated Model and Meta-Analysis , 2004 .

[28]  Michael M. Harris.  Internet Testing: The Examinee Perspective , 2008 .

[29]  T. Judge,et al.  Fairness Reactions to Personnel Selection Techniques in Greece: The Role of Core Self-Evaluations , 2007 .

[30]  P. Bobko,et al.  Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests. , 1997, The Journal of applied psychology.

[31]  Ann Marie Ryan,et al.  Designing and Implementing Global Selection Systems , 2009 .

[32]  Eugene Burke,et al.  Preserving the Integrity of Online Testing , 2009, Industrial and Organizational Psychology.

[33]  Jorge N. Tendeiro,et al.  Using Cumulative Sum Statistics to Detect Inconsistencies in Unproctored Internet Testing , 2013 .

[34]  M. Born,et al.  How Applicants Want and Expect to Be Treated: Applicants' Selection Treatment Beliefs and the Development of the Social Process Questionnaire on Selection , 2004 .

[35]  D. D. Steiner,et al.  Fairness Reactions to Selection Methods: An Italian Study , 2007 .

[36]  Denise Potosky,et al.  A Conceptual Framework for the Role of the Administration Medium in the Personnel Assessment Process , 2008 .

[37]  Dennis Doverspike,et al.  The Use of Mobile Devices in High‐Stakes Remotely Delivered Assessments and Testing , 2014 .

[38]  Stephen A. Dwight,et al.  A Quantitative Review of the Effect of Computerized Testing on the Measurement of Social Desirability , 2000 .

[39]  Christopher L. Martin,et al.  Some effects of computerized interviewing on job applicant responses , 1989 .

[40]  S. Rynes,et al.  Applicant Attraction Strategies: An Organizational Perspective , 1990 .

[41]  Filip Lievens,et al.  Dealing with the threats inherent in unproctored Internet testing of cognitive ability: Results from a large‐scale operational test program , 2011 .

[42]  A. Ryan,et al.  Reactions to Computerized Testing in Selection Contexts , 2003 .

[43]  U. Kanning,et al.  From the Subjects' Point of View , 2006 .

[44]  Donald P. Schwab,et al.  INDIVIDUAL REACTIONS TO ORGANIZATIONAL RECRUITING: A REVIEW , 1980 .

[45]  L. McFarland.  Warning Against Faking on a Personality Test: Effects on Applicant Reactions and Personality Test Scores , 2003 .

[46]  Jeffrey A. Miles,et al.  The effects of videoconference, telephone, and face-to-face media on interviewer and applicant judgments in employment interviews , 2001 .

[47]  Anton J. Villado,et al.  The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. , 2008, The Journal of applied psychology.

[48]  Fritz Drasgow,et al.  A Meta-Analytic Study of Social Desirability Distortion in Computer- Administered Questionnaires, Traditional Questionnaires, and Interviews , 1999 .

[49]  Fritz Drasgow,et al.  INTERACTIVE VIDEO ASSESSMENT OF CONFLICT RESOLUTION SKILLS , 1998 .

[50]  Donald M. Truxillo,et al.  The Importance of Organizational Justice in Personnel Selection: Defining When Selection Fairness Really Matters , 2004 .

[51]  A. Ryan,et al.  Applicant self-selection: correlates of withdrawal from a multiple hurdle process. , 2000, The Journal of applied psychology.

[52]  Neil A. Morelli,et al.  Establishing the Measurement Equivalence of Online Selection Assessments Delivered on Mobile Versus Nonmobile Devices , 2014 .

[53]  Selection fairness information and applicant reactions: a longitudinal field study. , 2002 .

[54]  Robert E. Ployhart,et al.  Using Attributions to Understand the Effects of Explanations on Applicant Reactions: Are Reactions Consistent With the Covariation Principle?1 , 2005 .

[55]  Phillip W. Braddy,et al.  Internet Recruiting , 2003 .

[56]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[57]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing , 1999 .

[58]  Donald M. Truxillo,et al.  Applicant Reactions to Different Selection Technology: Face-to-Face, Interactive Voice Response, and Computer-Assisted Telephone Screening Interviews , 2004 .

[59]  James W. Smither,et al.  APPLICANT REACTIONS TO SELECTION PROCEDURES , 2006 .

[60]  C. Steele,et al.  Stereotype threat and the intellectual test performance of African Americans. , 1995, Journal of personality and social psychology.

[61]  David Foster.  Secure, Online, High-Stakes Testing: Science Fiction or Business Reality?1 , 2009, Industrial and Organizational Psychology.

[62]  S. Gilliland,et al.  Fairness reactions to personnel selection techniques in France and the United States , 1996 .

[63]  Jane Webster,et al.  Applicant reactions to face-to-face and technology-mediated interviews: a field investigation. , 2003, The Journal of applied psychology.

[64]  Kelly A Kaminski,et al.  To Proctor or Not To Proctor? Balancing Business Needs With Validity in Online Assessment , 2009 .

[65]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[66]  Philip Bobko,et al.  Selection Testing via the Internet: Practical Considerations and Exploratory Empirical Findings* , 2004 .

[67]  Fritz Drasgow,et al.  Personality assessment: Does the medium matter? No☆ , 2006 .

[68]  Fritz Drasgow,et al.  Proctored Versus Unproctored Internet Tests: Are Unproctored Noncognitive Tests as Predictive of Job Performance? , 2011 .

[69]  Stephen B. Dunbar,et al.  Complex, Performance-Based Assessment: Expectations and Validation Criteria , 1991 .

[70]  R. Landers,et al.  Offsetting Performance Losses Due to Cheating in Unproctored Internet‐Based Testing by Increasing the Applicant Pool , 2012 .

[71]  S. Gully,et al.  Fairness reactions to personnel selection techniques in Singapore and the United States , 2002 .

[72]  K. Murphy,et al.  Psychological Testing: Principles and Applications , 1988 .

[73]  J. J. Donovan,et al.  Do Warnings Not to Fake Reduce Faking? , 2003 .

[74]  A. Ryan,et al.  Not much more than platitudes? A critical look at the utility of applicant reactions research , 2008 .

[75]  F. Drasgow,et al.  An IRT Approach to Constructing and Scoring Pairwise Preference Items Involving Stimuli on Different Dimensions: The Multi-Unidimensional Pairwise-Preference Model , 2005 .

[76]  Bernd Marcus.  Attitudes Towards Personnel Selection Methods: A Partial Replication and Extension in a German Sample , 2003 .

[77]  F. Lievens,et al.  Video-based versus written situational judgment tests: a comparison in terms of predictive validity. , 2006, The Journal of applied psychology.

[78]  John W. Boudreau,et al.  Role of recruitment in staffing utility analysis. , 1985 .

[79]  Neil Anderson,et al.  Technology and Discourse: A Comparison of Face‐to‐face and Telephone Employment Interviews , 2003 .

[80]  Adam W. Meade,et al.  Are Internet and Paper-and-Pencil Personality Tests Truly Comparable? , 2007 .

[81]  Fritz Drasgow,et al.  Identifying Cheating on Unproctored Internet Tests: The Z-Test and the Likelihood Ratio Test , 2010 .

[82]  Anton J. Villado,et al.  Unproctored Internet-Based Tests of Cognitive Ability and Personality: Magnitude of Cheating and Response Distortion , 2009, Industrial and Organizational Psychology.

[83]  Neil Anderson,et al.  Fairness Reactions to Personnel Selection Methods: An International Comparison between the Netherlands, the United States, France, Spain, Portugal, and Singapore , 2008 .

[84]  N. Tippins.  Where Is the Unproctored Internet Testing Train Headed Now? , 2009, Industrial and Organizational Psychology.

[85]  Filip Lievens,et al.  Research on Internet recruiting and testing: Current status and future directions , 2005 .

[86]  Jeffrey M. Cucina,et al.  Video‐Based Testing at U.S. Customs and Border Protection , 2011 .

[87]  Robert E. Gibby,et al.  Moving Beyond the Challenges to Make Unproctored Internet Testing a Reality , 2009 .

[88]  Juan Ling,et al.  Aesthetic properties and message customization: navigating the dark side of web recruitment. , 2007, The Journal of applied psychology.

[89]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[90]  N. Tippins.  Overview of Technology‐Enhanced Assessments , 2011 .

[91]  Chockalingam Viswesvaran,et al.  Meta-Analyses of Fakability Estimates: Implications for Personality Measurement , 1999 .

[92]  R. Landers,et al.  Retesting after initial failure, coaching rumors, and warnings against faking in online personality measures for selection. , 2011, The Journal of applied psychology.

[93]  Silvia Moscoso,et al.  Internet-based Personality Testing: Equivalence of Measures and Assesses' Perceptions and Reactions , 2003 .