Knowledge-driven Active Learning

The deployment of Deep Learning (DL) models is still precluded in those contexts where the amount of supervised data is limited. To answer this issue, active learning strategies aim at minimizing the amount of labelled data required to train a DL model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. These techniques are theoretically sound, but an understanding of the selected samples based on their content is not straightforward, further driving non-experts to consider DL as a black-box. For the first time, here we propose to take into consideration common domain-knowledge and enable non-expert users to train a model with fewer samples. In our Knowledge-driven Active Learning (KAL) framework, rule-based knowledge is converted into logic constraints and their violation is checked as a natural guide for sample selection. We show that even simple relationships among data and output classes offer a way to spot predictions for which the model need supervision. We empirically show that KAL (i) outperforms many active learning strategies, particularly in those contexts where domain knowledge is rich, (ii) it discovers data distribution lying far from the initial training data, (iii) it ensures domain experts that the provided knowledge is acquired by the model, (iv) it is suitable for regression and object recognition tasks unlike uncertainty-based strategies, and (v) its computational demand is low.

[1]  Thomas Lukasiewicz,et al.  Deep Learning with Logical Constraints , 2022, IJCAI.

[2]  Antoni B. Chan,et al.  A Comparative Survey of Deep Active Learning , 2022, ArXiv.

[3]  Fabien L. Gandon,et al.  Learning and Reasoning for Cultural Metadata Quality: Coupling Symbolic AI and Machine Learning over a Semantic Web Knowledge Graph to Support Museum Curators in Improving the Quality of Cultural Metadata and Information Retrieval , 2022, ACM Journal on Computing and Cultural Heritage.

[4]  M. Gori,et al.  Logic Explained Networks , 2021, NeSy.

[5]  Antoni B. Chan,et al.  A Comparative Survey: Benchmarking for Pool-based Active Learning , 2021, IJCAI.

[6]  M. Gori,et al.  Entropy-based Logic Explanations of Neural Networks , 2021, AAAI.

[7]  I. Tsang,et al.  Bayesian Active Learning by Disagreements: A Geometric Perspective , 2021, ArXiv.

[8]  Hyuk-Jae Lee,et al.  Active Learning for Deep Object Detection via Probabilistic Modeling , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Nicolas Thome,et al.  Confidence Estimation via Auxiliary Models , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Zhihui Li,et al.  A Survey of Deep Active Learning , 2020, ACM Comput. Surv..

[11]  Marco Maggini,et al.  Human-Driven FOL Explanations of Deep Learning , 2020, IJCAI.

[12]  Been Kim,et al.  Concept Bottleneck Models , 2020, ICML.

[13]  Tae-Kyun Kim,et al.  Sequential Graph Convolutional Network for Active Learning , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Arun Das,et al.  Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey , 2020, ArXiv.

[15]  F. Roli,et al.  Domain Knowledge Alleviates Adversarial Attacks in Multi-Label Classifiers , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Michele Fenzi,et al.  Scalable Active Learning for Object Detection , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[17]  Haifeng Shen,et al.  Adaptive Object Detection with Dual Multi-Label Prediction , 2020, ECCV.

[18]  C. Rudin,et al.  Concept whitening for interpretable image recognition , 2020, Nature Machine Intelligence.

[19]  Alessandro Bay,et al.  On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks NeurIPS Reproducibility Challenge 2019 , 2019 .

[20]  Bin Liu,et al.  Using multi-label classification to improve object detection , 2019, Neurocomputing.

[21]  Xiaoshuang Shi,et al.  Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss , 2019, Comput. Medical Imaging Graph..

[22]  Yarin Gal,et al.  BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning , 2019, NeurIPS.

[23]  John Langford,et al.  Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds , 2019, ICLR.

[24]  Gopinath Chennupati,et al.  On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks , 2019, NeurIPS.

[25]  In So Kweon,et al.  Learning Loss for Active Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Trevor Darrell,et al.  Variational Adversarial Active Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Marco Gori,et al.  LYRICS: A General Interface Layer to Integrate Logic Inference and Deep Learning , 2019, ECML/PKDD.

[28]  James Zou,et al.  Towards Automatic Concept-based Explanations , 2019, NeurIPS.

[29]  Fedor Zhdanov,et al.  Diverse mini-batch Active Learning , 2019, ArXiv.

[30]  Remus Pop,et al.  Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles , 2018, ArXiv.

[31]  Andreas Nürnberger,et al.  The Power of Ensembles for Active Learning in Image Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Lalana Kagal,et al.  Explaining Explanations: An Overview of Interpretability of Machine Learning , 2018, 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA).

[33]  Franco Turini,et al.  Local Rule-Based Explanations of Black Box Decision Systems , 2018, ArXiv.

[34]  Carlos Guestrin,et al.  Anchors: High-Precision Model-Agnostic Explanations , 2018, AAAI.

[35]  Frédéric Precioso,et al.  Adversarial Active Learning for Deep Networks: a Margin Based Approach , 2018, ArXiv.

[36]  Gary Marcus,et al.  Deep Learning: A Critical Appraisal , 2018, ArXiv.

[37]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[38]  Silvio Savarese,et al.  Active Learning for Convolutional Neural Networks: A Core-Set Approach , 2017, ICLR.

[39]  Razvan Pascanu,et al.  A simple neural network module for relational reasoning , 2017, NIPS.

[40]  Zoubin Ghahramani,et al.  Deep Bayesian Active Learning with Image Data , 2017, ICML.

[41]  Marco Gori,et al.  Semantic-based regularization for learning and inference , 2017, Artif. Intell..

[42]  Sergio Gomez Colmenarejo,et al.  Hybrid computing using a neural network with dynamic external memory , 2016, Nature.

[43]  Artur S. d'Avila Garcez,et al.  Logic Tensor Networks: Deep Learning and Logical Reasoning from Data and Knowledge , 2016, NeSy@HLAI.

[44]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  L. D. Raedt,et al.  Probabilistic (logic) programming concepts , 2015, Machine Learning.

[46]  Yinda Zhang,et al.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop , 2015, ArXiv.

[47]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Marcello Sanguineti,et al.  Foundations of Support Constraint Machines , 2015, Neural Computation.

[49]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[50]  Dan Wang,et al.  A new active labeling method for deep learning , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[51]  Sanja Fidler,et al.  Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[53]  Marco Gori,et al.  Constraint Verification With Kernel Machines , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[54]  Zoubin Ghahramani,et al.  Bayesian Active Learning for Classification and Preference Learning , 2011, ArXiv.

[55]  Pietro Perona,et al.  The Caltech-UCSD Birds-200-2011 Dataset , 2011 .

[56]  Xiaojin Zhu,et al.  Introduction to Semi-Supervised Learning , 2009, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[57]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[59]  Ross D. King,et al.  Active Learning for Regression Based on Query by Committee , 2007, IDEAL.

[60]  Klaus Brinker,et al.  Incorporating Diversity in Active Learning with Support Vector Machines , 2003, ICML.

[61]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2000, J. Mach. Learn. Res..

[62]  Greg Schohn,et al.  Less is More: Active Learning with Support Vector Machines , 2000, ICML.

[63]  Andrew McCallum,et al.  Employing EM and Pool-Based Active Learning for Text Classification , 1998, ICML.

[64]  Frédéric Precioso,et al.  Active learning strategy for CNN combining batchwise Dropout and Query-By-Committee , 2017, ESANN.

[65]  Philipp Koehn,et al.  Cognitive Psychology , 1992, Ageing and Society.

[66]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[67]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[68]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[69]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[70]  Christian Eitzinger,et al.  Triangular Norms , 2001, Künstliche Intell..

[71]  Laurian M. Chirica,et al.  The entity-relationship model: toward a unified view of data , 1975, SIGF.