Leveraging the Potentials of Dedicated Collaborative Interactive Learning: Conceptual Foundations to Overcome Uncertainty by Human-Machine Collaboration

When a learning system learns from data that was previously assigned to categories, we say that the learning system learns in a supervised way. By “supervised”, we mean that a higher entity, for example a human, has arranged the data into categories. Fully categorizing the data is cost intensive and time consuming. Moreover, the categories (labels) provided by humans might be subject to uncertainty, as humans are prone to error. This is where dedicate collaborative interactive learning (D-CIL) comes together: The learning system can decide from which data it learns, copes with uncertainty regarding the categories, and does not require a fully labeled dataset. Against this background, we create the foundations of two central challenges in this early development stage of D-CIL: task complexity and uncertainty. We present an approach to “crowdsourcing traffic sign labels with self-assessment” that will support leveraging the potentials of D-CIL.

[1]  Robert F Murphy,et al.  An active role for machine learning in drug development. , 2011, Nature chemical biology.

[2]  Johannes Stallkamp,et al.  Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition , 2012, Neural Networks.

[3]  Robert O. Briggs,et al.  A Six-Layer Model of Collaboration , 2015 .

[4]  Ricardo M. Marcacini,et al.  An active learning approach to frequent itemset-based text clustering , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[5]  Xingquan Zhu,et al.  Active learning with uncertain labeling knowledge , 2014, Pattern Recognit. Lett..

[6]  Alan R. Hevner,et al.  POSITIONING AND PRESENTING DESIGN SCIENCE RESEARCH FOR MAXIMUM IMPACT 1 , 2013 .

[7]  Alan R. Hevner,et al.  The Three Cycle View of Design Science , 2007, Scand. J. Inf. Syst..

[8]  Xindong Wu,et al.  Multi-Class Ground Truth Inference in Crowdsourcing with Clustering , 2016, IEEE Transactions on Knowledge and Data Engineering.

[9]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[10]  Yu Zhang,et al.  An Active Learning Approach for Jointly Estimating Worker Performance and Annotation Reliability with Crowdsourced Data , 2014, ArXiv.

[11]  Ulrich Bretschneider,et al.  Towards Successful Crowdsourcing Projects: Evaluating the Implementation of Governance Mechanisms , 2015, ICIS.

[12]  Mausam,et al.  Re-Active Learning: Active Learning with Relabeling , 2016, AAAI.

[13]  Jan Marco Leimeister,et al.  Crowd Work , 2016, Business & Information Systems Engineering.

[14]  Paul Lukowicz,et al.  From Active Learning to Dedicated Collaborative Interactive Learning , 2016 .

[15]  Xindong Wu,et al.  Self-Taught Active Learning from Crowds , 2012, 2012 IEEE 12th International Conference on Data Mining.

[16]  Paul N. Bennett,et al.  Dual Strategy Active Learning , 2007, ECML.

[17]  Xiaoyu Zhang,et al.  Bidirectional Active Learning: A Two-Way Exploration Into Unlabeled and Labeled Data Set , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Robert O. Briggs,et al.  Defining Key Concepts for Collaboration Engineering , 2006, AMCIS.

[19]  Robert F. Murphy,et al.  Efficient discovery of responses of proteins to compounds using active learning , 2013, BMC Bioinformatics.

[20]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[21]  Jingjing Li,et al.  Learning from crowds with active learning and self-healing , 2017, Neural Computing and Applications.

[22]  Jun Du,et al.  Active Learning with Human-Like Noisy Oracle , 2010, 2010 IEEE International Conference on Data Mining.

[23]  Amihai Motro,et al.  Uncertainty Management in Information Systems: From Needs to Solution , 1996 .

[24]  Antanas Verikas,et al.  Agreeing to disagree: active learning with noisy labels without crowdsourcing , 2017, International Journal of Machine Learning and Cybernetics.

[25]  Russell Greiner,et al.  Optimistic Active-Learning Using Mutual Information , 2007, IJCAI.

[26]  Aristidis Likas,et al.  An incremental training method for the probabilistic RBF network , 2006, IEEE Trans. Neural Networks.

[27]  Johannes Stallkamp,et al.  The German Traffic Sign Recognition Benchmark: A multi-class classification competition , 2011, The 2011 International Joint Conference on Neural Networks.

[28]  Shirley Gregor,et al.  The Anatomy of a Design Theory , 2007, J. Assoc. Inf. Syst..