AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries

As artificial intelligence (AI) systems are increasingly deployed, principles for ethical AI are also proliferating. Certification offers a method to both incentivize the adoption of these principles and substantiate that they have been implemented in practice. This article draws from management literature on certification and reviews current AI certification programs and proposals. Successful programs rely on both emerging technical methods and specific design considerations. In order to avoid two common failures of certification, program designs should ensure that the symbol of the certification is substantially implemented in practice and that the program achieves its stated goals. The review indicates that the field currently focuses on self-certification and third-party certification of systems, individuals, and organizations—to the exclusion of process management certifications. Additionally, this article considers prospects for future AI certification programs. Ongoing changes in AI technology suggest that AI certification regimes should be designed to emphasize governance criteria of enduring value, such as ethics training for AI developers, and to adjust technical criteria as the technology changes. Overall, certification can play a valuable mix in the portfolio of AI governance tools.

[1]  J. Tschirhart,et al.  Alternatives to traditional regulation , 1989 .

[2]  Seth D. Baum,et al.  Medium-Term Artificial Intelligence and Society , 2020, Inf..

[3]  Inioluwa Deborah Raji,et al.  Model Cards for Model Reporting , 2018, FAT.

[4]  Nancy Tuana,et al.  The Role of the National Science Foundation Broader Impacts Criterion in Enhancing Research Ethics Pedagogy , 2009 .

[5]  Omer Levy,et al.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems , 2019, NeurIPS.

[6]  T. Walsh.  The effective and ethical development of artificial intelligence: an opportunity to improve our wellbeing , 2019 .

[7]  R. Penrose,et al.  How Long Until Human-Level AI ? Results from an Expert Assessment , 2011 .

[8]  Sameer Singh,et al.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList , 2020, ACL.

[9]  A. Spiller,et al.  The Reliability of Certification: Quality Labels as a Consumer Policy Tool , 2005 .

[10]  L. Lakhal.  The Relationship Between ISO 9000 Certification, TQM Practices, and Organizational Performance , 2014 .

[11]  O. Boiral.  Managing with ISO Systems: Lessons from Practice , 2011 .

[12]  Seth D. Baum.  Reconciliation between factions focused on near-term and long-term artificial intelligence , 2017, AI & SOCIETY.

[13]  Kush R. Varshney,et al.  Increasing Trust in AI Services through Supplier's Declarations of Conformity , 2018, IBM J. Res. Dev..

[14]  Sanford J. Grossman,et al.  The Costs and Benefits of Ownership: A Theory of Vertical and Lateral Integration , 1986 .

[15]  Seth D. Baum,et al.  Corporate Governance of Artificial Intelligence in the Public Interest , 2021, Inf..

[16]  Michael J. Lenox,et al.  The Strategic Use of Decentralized Institutions: Exploring Certification With the ISO 14001 Management Standard , 2005 .

[17]  Luciano Floridi,et al.  From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices , 2019, Science and Engineering Ethics.

[18]  M. Cannarsa.  Ethics Guidelines for Trustworthy AI , 2021, The Cambridge Handbook of Lawyering in the Digital Age.

[19]  Stuart Armstrong,et al.  The errors, insights and lessons of famous AI predictions – and what they mean for the future , 2014, J. Exp. Theor. Artif. Intell..

[20]  Michael J. Lenox,et al.  Industry Self-Regulation Without Sanctions: The Chemical Industry's Responsible Care Program , 2000 .

[21]  Olivier Boiral,et al.  ISO 9000: Outside the Iron Cage , 2003, Organ. Sci..

[22]  Elizabeth Gibney.  The battle for ethical AI at the world’s biggest machine-learning conference , 2020, Nature.

[23]  Matthew U. Scherer.  Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies , 2015 .

[24]  T. S. Ragu-Nathan,et al.  Does ISO 9000 have an effect on quality management practices? An international empirical study , 1997 .

[25]  Cynthia A. Williams,et al.  The New Role for Assurance Services in Global Commerce , 2008 .

[26]  George A. Akerlof,et al.  The Market for “Lemons”: Quality Uncertainty and the Market Mechanism , 1970 .

[27]  Anna Jobin,et al.  The global landscape of AI ethics guidelines , 2019, Nature Machine Intelligence.

[28]  Brent Mittelstadt,et al.  Principles alone cannot guarantee ethical AI , 2019, Nature Machine Intelligence.

[29]  Edward J. Zajac,et al.  Substance and Symbolism in CEOs' Long-Term Incentive Plans , 1994 .

[30]  Hongyi Sun.  Diffusion and contribution of total quality management: an empirical study in Norway , 1999 .

[31]  Andrew A. King,et al.  The effect of certification with the ISO 9000 Quality Management Standard: A signaling approach , 2006 .

[32]  Roman V Yampolskiy,et al.  Safety Engineering for Artificial General Intelligence , 2012, Topoi.

[33]  J. O'Rourke.  Bridgestone/Firestone, Inc. and Ford Motor Company: How a Product Safety Crisis Ended a Hundred-Year Relationship , 2001 .

[34]  Hesan A. Quazi,et al.  Impact of ISO 9000 certification on quality management practices: A comparative study , 2002 .

[35]  Petra Christmann,et al.  Firm self-regulation through international certifiable standards: determinants of symbolic versus substantive implementation , 2006 .

[36]  John Salvatier,et al.  When Will AI Exceed Human Performance? Evidence from AI Experts , 2017, ArXiv.

[37]  A. Bradford.  The Brussels Effect: How the European Union Rules the World , 2020 .

[38]  Linda S. Munilla,et al.  Marketing and environmental registration/certification: What industrial marketers should understand about ISO 14000 , 1997 .

[39]  James D. Wright,et al.  A World of Standards but not a Standard World: Toward a Sociology of Standards and Standardization ∗ , 2013 .

[40]  Timnit Gebru,et al.  Datasheets for datasets , 2018, Commun. ACM.

[41]  Madhulika Srikumar,et al.  Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI , 2020, SSRN Electronic Journal.

[42]  D. Ernst,et al.  The Common Criteria for Information Technology Security Evaluation – Implications for China’s Policy on Information Security Standards , 2009 .

[43]  日本規格協会.  環境マネジメントシステム : 要求事項及び利用の手引 = Environmental management systems : requirements with guidance for use , 2002 .

[44]  Walter W. Powell,et al.  From Smoke and Mirrors to Walking the Talk: Decoupling in the Contemporary World , 2012 .

[45]  Andrew McCallum,et al.  Energy and Policy Considerations for Deep Learning in NLP , 2019, ACL.

[46]  F. Wijen.  Means Versus Ends in Opaque Institutional Fields: Trading Off Compliance and Achievement in Sustainability Standard Adoption , 2013 .

[47]  Miles Brundage,et al.  The Role of Cooperation in Responsible AI Development , 2019, ArXiv.

[48]  Inioluwa Deborah Raji,et al.  Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing , 2020, FAT*.

[49]  Peer C. Fiss,et al.  The Symbolic Management of Strategic Change: Sensegiving Via Framing and Decoupling , 2006 .

[50]  Morten T. Hansen,et al.  Faking It or Muddling Through? Understanding Decoupling in Response to Stakeholder Pressures , 2012 .

[51]  M. Alvesson,et al.  A Stupidity‐Based Theory of Organizations , 2012 .

[52]  Jorge Rivera,et al.  Is Greener Whiter? Voluntary Environmental Performance of Western Ski Areas , 2004 .

[53]  Peter Henderson,et al.  Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims , 2020, ArXiv.

[54]  John W. Meyer,et al.  Institutionalized Organizations: Formal Structure as Myth and Ceremony , 1977, American Journal of Sociology.

[55]  Ahmed Hosny,et al.  The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards , 2018, Data Protection and Privacy.

[56]  日本規格協会.  情報技術-セキュリティ技術-情報セキュリティマネジメントシステム-要求事項 : 国際規格ISO/IEC 27001 = Information technology-Security techniques-Information security management systems-Requirements : ISO/IEC 27001 , 2005 .

[57]  N. Bostrom.  Strategic Implications of Openness in AI Development , 2017, Artificial Intelligence Safety and Security.