Bias learning, knowledge sharing

Biasing properly the hypothesis space of a learner has been shown to improve generalization performance. Methods for achieving this goal have been proposed, that range from designing and introducing a bias into a learner to automatically learning the bias. Multitask learning methods fall into the latter category. When several related tasks derived from the same domain are available, these methods use the domain-related knowledge coded in the training examples of all the tasks as a source of bias. We extend some of the ideas presented in this field and describe a new approach that identifies a family of hypotheses, represented by a manifold in hypothesis space, that embodies domain-related knowledge. This family is learned using training examples sampled from a group of related tasks. Learning models trained on these tasks are only allowed to select hypotheses that belong to the family. We show that the new approach encompasses a large variety of families which can be learned. A statistical analysis on a class of related tasks is performed that shows significantly improved performances when using this approach.

[1]  K Y Liang,et al.  Longitudinal data analysis for discrete and continuous outcomes. , 1986, Biometrics.

[2]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[3]  Geoffrey E. Hinton,et al.  A time-delay neural network architecture for isolated word recognition , 1990, Neural Networks.

[4]  Raymond J. Mooney,et al.  Improving Shared Rules in Multiple Category Domain Theories , 1991, ML.

[5]  William W. Cohen.  Compiling prior knowledge into an explicit basis , 1992, ICML 1992.

[6]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[7]  Geoffrey E. Hinton,et al.  Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[8]  Dennis F. Kibler,et al.  Concept Sharing: A Means to Improve Multi-Concept Learning , 1993, ICML.

[9]  Yaser S. Abu-Mostafa,et al.  Hints and the VC Dimension , 1993, Neural Computation.

[10]  Tom M. Mitchell,et al.  Experience with a learning personal assistant , 1994, CACM.

[11]  Stephen M. Omohundro.  Family Discovery , 1995, NIPS.

[12]  Jonathan Baxter,et al.  Learning Model Bias , 1995, NIPS.

[13]  Nathan Intrator,et al.  How to Make a Low-Dimensional Representation Suitable for Diverse Tasks , 1996 .

[14]  Sebastian Thrun,et al.  Discovering Structure in Multiple Learning Tasks: The TC Algorithm , 1996, ICML.

[15]  Rich Caruana,et al.  Algorithms and Applications for Multitask Learning , 1996, ICML.

[16]  Jonathan Baxter,et al.  A Bayesian/information theoretic model of bias learning , 2019, COLT '96.

[17]  Daniel L. Silver,et al.  The Parallel Transfer of Task Knowledge Using Dynamic Learning Rates Based on a Measure of Relatedness , 1996, Connect. Sci..

[18]  Joseph Sill,et al.  Monotonicity Hints , 1996, NIPS.

[19]  Yoshua Bengio,et al.  Multi-Task Learning for Stock Selection , 1996, NIPS.

[20]  Tom Heskes,et al.  Solving a Huge Number of Similar Tasks: A Combination of Multi-Task Learning and a Hierarchical Bayesian Approach , 1998, ICML.

[21]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[22]  Hilan Bensusan.  Odd bites into bananas don''t make you blind: learning about simplicity and attribute addition , 1998 .

[23]  Nathan Intrator,et al.  Making a Low-Dimensional Representation Suitable for Diverse Tasks , 1998, Learning to Learn.

[24]  Sebastian Thrun,et al.  Learning to Learn , 1998, Springer US.

[25]  Yoshua Bengio,et al.  Bias learning, knowledge sharing , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[26]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[27]  Jonathan Baxter,et al.  A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling , 1997, Machine Learning.

[28]  Jürgen Schmidhuber,et al.  Shifting Inductive Bias with Success-Story Algorithm, Adaptive Levin Search, and Incremental Self-Improvement , 1997, Machine Learning.