Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks

In this paper we develop a framework for analysing the impact of AI on occupations. Leaving aside the debates on robotisation, digitalisation and online platforms as well as workplace automation, we focus on the occupational impact of AI that is driven by rapid progress in machine learning. In our framework we map 59 generic tasks from several worker surveys and databases to 14 cognitive abilities (that we extract from the cognitive science literature) and these to a comprehensive list of 328 AI benchmarks used to evaluate progress in AI techniques. The use of these cognitive abilities as an intermediate mapping, instead of mapping task characteristics to AI tasks, allows for an analysis of AI’s occupational impact that goes beyond automation. An application of our framework to occupational databases gives insights into the abilities through which AI is most likely to affect jobs and allows for a ranking of occupation with respect to AI impact. Moreover, we find that some jobs that were traditionally less affected by previous waves of automation may now be subject to relatively higher AI impact.

[1]  David Autor,et al.  Skills, Tasks and Technologies: Implications for Employment and Earnings , 2010 .

[2]  P. Lockhart INTRODUCTION TO GEOMETRY , 2007 .

[3]  José Hernández-Orallo,et al.  Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement , 2017, Artificial Intelligence Review.

[4]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[5]  Murray Campbell,et al.  Deep Blue , 2002, Artif. Intell..

[6]  Miles Brundage,et al.  Modeling Progress in AI , 2015, AAAI Workshop: AI, Ethics, and Society.

[7]  H. Eyre Introduction I , 1979, New Surveys in the Classics.

[8]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[9]  Petar Velickovic,et al.  The PlayStation Reinforcement Learning Environment (PSXLE) , 2019, ArXiv.

[10]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[11]  José Hernández-Orallo,et al.  A multidisciplinary task-based perspective for evaluating the impact of AI autonomy and generality on the future of work , 2018, IJCAI 2018.

[12]  S. Nuti,et al.  The Use of Google Trends in Health Care Research: A Systematic Review , 2014, PloS one.

[13]  Tom M. Mitchell,et al.  What can machine learning do? Workforce implications , 2017, Science.

[14]  A. Manning,et al.  Explaining Job Polarization: Routine-Biased Technological Change and Offshoring , 2014 .

[15]  Alex Coad,et al.  The firm-level employment effects of innovations in high-tech US manufacturing industries , 2011 .

[16]  H. Stanley,et al.  Quantifying Trading Behavior in Financial Markets Using Google Trends , 2013, Scientific Reports.

[17]  José Hernández-Orallo,et al.  Tracking AI: The Capability Is (Not) Near , 2020, ECAI.

[18]  Mark Chen,et al.  Scaling Laws for Autoregressive Generative Modeling , 2020, ArXiv.

[19]  Daniele Archibugi,et al.  Patenting as an indicator of technological innovation: a review , 1992 .

[20]  José Hernández-Orallo,et al.  AI Extenders: The Ethical and Societal Implications of Humans Cognitively Extended by AI , 2019, AIES.

[21]  Adam B. Jaffe,et al.  Patent citation data in social science research: Overview and best practices , 2016, J. Assoc. Inf. Sci. Technol..

[22]  E. Brynjolfsson,et al.  The Productivity J-Curve: How Intangibles Complement General Purpose Technologies , 2018, American Economic Journal: Macroeconomics.

[23]  Nick Bostrom,et al.  Future progress in artificial intelligence: a poll among experts , 2014, SIGAI.

[24]  Donald G. Saari,et al.  Complexity and the geometry of voting , 2008, Math. Comput. Model..

[25]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[26]  José Hernández-Orallo,et al.  The Measure of All Minds: Evaluating Natural and Artificial Intelligence , 2017 .

[27]  E. Wasserman,et al.  Comparative cognition : experimental explorations of animal intelligence , 2009 .

[28]  José Hernández-Orallo,et al.  Dual Indicators to Analyze AI Benchmarks: Difficulty, Discrimination, Ability, and Generality , 2020, IEEE Transactions on Games.

[29]  Tom M. Mitchell,et al.  What Can Machines Learn, and What Does It Mean for Occupations and the Economy? , 2018 .

[30]  Dawn Song,et al.  Measuring Massive Multitask Language Understanding , 2020, ICLR.

[31]  Michael Webb,et al.  The Impact of Artificial Intelligence on the Labor Market , 2019, SSRN Electronic Journal.

[32]  David Autor,et al.  The Growth of Low Skill Service Jobs and the Polarization of the U.S. Labor Market , 2009, SSRN Electronic Journal.

[33]  E. Monte,et al.  Economic Uncertainty: A Geometric Indicator of Discrepancy Among Experts’ Expectations , 2018, Social Indicators Research.

[34]  M. Virgillito,et al.  The Innovation‐Employment Nexus: A Critical Survey of Theory and Empirics , 2016 .

[35]  D. Deming,et al.  The Growing Importance of Social Skills in the Labor Market , 2017 .

[36]  Michael C. Pyryt Human cognitive abilities: A survey of factor analytic studies , 1998 .

[37]  M. Savona,et al.  The Effect of R&D Growth on Employment and Self-Employment in Local Labour Markets , 2018 .

[38]  Enrique Fernández-Macías,et al.  A Taxonomy of Tasks for Assessing the Impact of New Technologies on Work , 2020 .

[39]  Shimon Whiteson,et al.  Protecting against evaluation overfitting in empirical reinforcement learning , 2011, 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).

[40]  N. Dalkey,et al.  An Experimental Application of the Delphi Method to the Use of Experts , 1963 .

[41]  Elizabeth Lyons,et al.  Digitization and the Contract Labor Market: A Research Agenda , 2013 .

[42]  Erik Brynjolfsson,et al.  Artificial Intelligence and the Modern Productivity Paradox: A Clash of Expectations and Statistics , 2017 .

[43]  Marlos C. Machado,et al.  Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents , 2017, J. Artif. Intell. Res..

[44]  C. Adcock,et al.  Primary Mental Abilities. , 1971, The Journal of general psychology.

[45]  W. Naudé,et al.  Artificial Intelligence, Income Distribution and Economic Growth , 2020 .

[46]  José Hernández-Orallo,et al.  Does AI Qualify for the Job?: A Bidirectional Model Mapping Labour and AI Intensities , 2020, AIES.

[47]  M. Arntz,et al.  The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis , 2016 .

[48]  Ernest Davis,et al.  Commonsense reasoning and commonsense knowledge in artificial intelligence , 2015, Commun. ACM.

[49]  Rebecca Henderson,et al.  The Impact of Artificial Intelligence on Innovation , 2018, The Economics of Artificial Intelligence.

[50]  Avi Goldfarb,et al.  Prediction Machines: The Simple Economics of Artificial Intelligence , 2018 .

[51]  Michael A. Osborne,et al.  The future of employment: How susceptible are jobs to computerisation? , 2017 .

[52]  A. Manning,et al.  Job Polarization in Europe , 2009 .

[53]  David Autor The “task approach” to labor markets: an overview , 2013, SSRN Electronic Journal.

[54]  Daron Acemoglu,et al.  The Race between Man and Machine: Implications of Technology for Growth, Factor Shares, and Employment , 2018, American Economic Review.

[55]  D. Budescu Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. , 1993 .

[56]  José Hernández-Orallo,et al.  Accounting for the Neglected Dimensions of AI Progress , 2018, ArXiv.

[57]  Isabelle Guyon,et al.  Comparison of classifier methods: a case study in handwritten digit recognition , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[58]  Trupti M. Kodinariya,et al.  Review on determining number of Cluster in K-Means Clustering , 2013 .

[59]  James Bessen,et al.  Ai and Jobs: The Role of Demand , 2017 .

[60]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[61]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[62]  John F. Sowa,et al.  Mapping the Landscape of Human-Level Artificial General Intelligence , 2012, AI Mag..

[63]  John Salvatier,et al.  When Will AI Exceed Human Performance? Evidence from AI Experts , 2017, ArXiv.

[64]  Noam Brown,et al.  Superhuman AI for multiplayer poker , 2019, Science.

[65]  Z. Griliches Patent Statistics as Economic Indicators: a Survey , 1990 .

[66]  David Autor,et al.  Putting Tasks to the Test: Human Capital, Job Tasks, and Wages , 2009, Journal of Labor Economics.

[67]  David A. Ferrucci,et al.  Introduction to "This is Watson" , 2012, IBM J. Res. Dev..

[68]  D. W. Fiske Consistency of the factorial structures of personality ratings from different sour sources. , 1949, Journal of abnormal psychology.

[69]  G. Dosi Sources, Procedures, and Microeconomic Effects of Innovation , 1988 .

[70]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[71]  José Hernández-Orallo,et al.  A New AI Evaluation Cosmos: Ready to Play the Game? , 2017, AI Mag..

[72]  Tom Schaul,et al.  StarCraft II: A New Challenge for Reinforcement Learning , 2017, ArXiv.

[73]  David H. Autor,et al.  Replication data for: Why Are There Still So Many Jobs? The History and Future of Workplace Automation , 2019 .

[74]  Edward W. Felten,et al.  A Method to Link Advances in Artificial Intelligence to Occupational Abilities , 2018 .

[75]  Bruce G. Buchanan,et al.  A Virtual Archive for the History of AI , 2013, AI Mag..

[76]  Joel Mokyr,et al.  The history of technological anxiety and the future of economic growth: Is this time different? , 2015 .

[77]  David Autor,et al.  Polanyi&Apos;S Paradox and the Shape of Employment Growth , 2014 .

[78]  F. Bogliacino,et al.  R&D and Employment: An Application of the LSDVC Estimator Using European Microdata , 2012 .

[79]  M. Polanyi The Logic of Tacit Inference , 1966, Philosophy.

[80]  James Bessen Automation and Jobs: When Technology Boosts Employment , 2019, Economic Policy.

[81]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[82]  G. Quintini,et al.  Automation, skills use and training , 2018 .

[83]  Nick Bostrom,et al.  Future Progress in Artificial Intelligence: A Survey of Expert Opinion , 2013, PT-AI.

[84]  D. Acemoglu Technical Change, Inequality, and the Labor Market , 2000 .

[85]  Omer Levy,et al.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems , 2019, NeurIPS.

[86]  Timothy Z. Keith,et al.  Cattell-Horn-Carroll Abilities and Cognitive Tests: What We've Learned from 20 Years of Research. , 2010 .