Mass Personalization of Deep Learning

We discuss training techniques, objectives and metrics toward mass personalization of deep learning models. In machine learning, personalization refers to the fact that every trained model should be targeted towards an individual by optimizing one or several performance metrics and often obeying additional constraints. We investigate three methods for personalized training of neural networks. They constitute three forms of curriculum learning. The methods are partially inspired by the "shaping" concept from psychology. Interestingly, we discover that extensive exposure to a limited set of training data in terms of class diversity \emph{early} in the training can lead to an irreversible reduction of the capability of a network to learn from more diverse training data. This is in close alignment with existing theories in human development. In contrast, training on a small data set covering all classes \emph{early} in the training can lead to better performance.

[1]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[2]  Ruhi Sarikaya.  An overview of the system architecture and key components The Technology Behind Personal Digital Assistants , 2022 .

[3]  Deyu Meng,et al.  Small Sample Learning in Big Data Era , 2018, ArXiv.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[6]  Johanes Schneider,et al.  Personalized explanation in machine learning , 2019, ArXiv.

[7]  Farinaz Koushanfar,et al.  Privacy-Preserving Deep Learning and Inference , 2018, 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[8]  Shiguang Shan,et al.  Self-Paced Learning with Diversity , 2014, NIPS.

[9]  Jinshuo Dong,et al.  Deep Learning with Gaussian Differential Privacy , 2020, Harvard data science review.

[10]  Bernhard Schölkopf,et al.  Correcting Sample Selection Bias by Unlabeled Data , 2006, NIPS.

[11]  Matt J. Kusner,et al.  Counterfactual Fairness , 2017, NIPS.

[12]  Gustavo Rossi,et al.  Designing personalized web applications , 2001, WWW '01.

[13]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[14]  Xu Chen,et al.  Explainable Recommendation: A Survey and New Perspectives , 2018, Found. Trends Inf. Retr..

[15]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[16]  Graham W. Taylor,et al.  Deep Multimodal Learning: A Survey on Recent Advances and Trends , 2017, IEEE Signal Processing Magazine.

[17]  John R. Anderson,et al.  MACHINE LEARNING An Artificial Intelligence Approach , 2009 .

[18]  Yoshua Bengio,et al.  Deep Learning of Representations for Unsupervised and Transfer Learning , 2011, ICML Unsupervised and Transfer Learning.

[19]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[20]  Martial Hebert,et al.  Growing a Brain: Fine-Tuning by Increasing Model Capacity , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Alex Graves,et al.  Automated Curriculum Learning for Neural Networks , 2017, ICML.

[22]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[23]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[24]  Martial Hebert,et al.  Learning by Asking Questions , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Stefano Ermon,et al.  Label-Free Supervision of Neural Networks with Physics and Domain Knowledge , 2016, AAAI.

[26]  G. Peterson.  A day of great illumination: B. F. Skinner's discovery of shaping. , 2004, Journal of the experimental analysis of behavior.

[27]  Johannes Schneider,et al.  Principles of Green Data Mining , 2019, HICSS.

[28]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[29]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[30]  Hsinchun Chen,et al.  Web mining: Machine learning for web applications , 2005, Annu. Rev. Inf. Sci. Technol..

[31]  Charles A. Nelson,et al.  Romania’s Abandoned Children: Deprivation, Brain Development, and the Struggle for Recovery , 2014 .

[32]  Marshall Scott Poole,et al.  What Is Personalization? Perspectives on the Design and Implementation of Personalization in Information Systems , 2006, J. Organ. Comput. Electron. Commer..

[33]  Reuben Binns,et al.  Fairness in Machine Learning: Lessons from Political Philosophy , 2017, FAT.

[34]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.