论文信息 - TuringBox: An Experimental Platform for the Evaluation of AI Systems

TuringBox: An Experimental Platform for the Evaluation of AI Systems

We introduce TuringBox, a platform to democratize the study of AI. On one side of the platform, AI contributors upload existing and novel algorithms to be studied scientifically by others. On the other side, AI examiners develop and post machine intelligence tasks to evaluate and characterize the outputs of algorithms. We outline the architecture of such a platform, and describe two interactive case studies of algorithmic auditing on the platform.

[1] Christo Wilson,et al. An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace , 2016, WWW.

[2] Simone Muench,et al. Queue , 2020 .

[3] Paul Voosen,et al. The AI detectives. , 2017, Science.

[4] Timnit Gebru,et al. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification , 2018, FAT.

[5] Brendan T. O'Connor,et al. Demographic Dialectal Variation in Social Media: A Case Study of African-American English , 2016, EMNLP.

[6] David Lazer,et al. Measuring Price Discrimination and Steering on E-commerce Web Sites , 2014, Internet Measurement Conference.

[7] Latanya Sweeney,et al. Discrimination in online ad delivery , 2013, CACM.

[8] Helen Nissenbaum,et al. Bias in computer systems , 1996, TOIS.

[9] Tony Doyle,et al. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy , 2017, Inf. Soc..

[10] Maria Leonor Pacheco,et al. of the Association for Computational Linguistics: , 2001 .

[11] Seth Flaxman,et al. EU regulations on algorithmic decision-making and a "right to explanation" , 2016, ArXiv.

[12] Adam Tauman Kalai,et al. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[13] Jieyu Zhao,et al. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints , 2017, EMNLP.