Virtual to Real-World Transfer Learning: A Systematic Review

Machine learning has become an important research area in many domains and real-world applications. The prevailing assumption in traditional machine learning techniques, that training and testing data should be of the same domain, is a challenge. In the real world, gathering enough training data to create high-performance learning models is not easy. Sometimes data are not available, very expensive, or dangerous to collect. In this scenario, the concept of machine learning does not hold up to its potential. Transfer learning has recently gained much acclaim in the field of research as it has the capability to create high performance learners through virtual environments or by using data gathered from other domains. This systematic review defines (a) transfer learning; (b) discusses the recent research conducted; (c) the current status of transfer learning and finally, (d) discusses how transfer learning can bridge the gap between the virtual and the real.

[1]  Mei Wang,et al.  Deep Visual Domain Adaptation: A Survey , 2018, Neurocomputing.

[2]  Alexander Fabisch,et al.  A Survey of Behavior Learning Applications in Robotics - State of the Art and Perspectives , 2019, ArXiv.

[3]  Henrik I. Christensen,et al.  How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? , 2019, ArXiv.

[4]  Bruce A. Francis,et al.  A Preliminary Study of Transfer Learning between Unicycle Robots , 2016, AAAI Spring Symposia.

[5]  D. Moher,et al.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement , 2009, BMJ : British Medical Journal.

[6]  Thomas Wolf,et al.  TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents , 2019, ArXiv.

[7]  Danica Kragic,et al.  Fast Adaptation with Meta-Reinforcement Learning for Trust Modelling in Human-Robot Interaction , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  Wojciech Zaremba,et al.  Domain randomization for transferring deep neural networks from simulation to the real world , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Jinwoo Shin,et al.  Learning What and Where to Transfer , 2019, ICML.

[10]  Yiqiang Chen,et al.  Transfer Learning with Dynamic Distribution Adaptation , 2019, ACM Trans. Intell. Syst. Technol..

[11]  Sergey Levine,et al.  (CAD)$^2$RL: Real Single-Image Flight without a Single Real Image , 2016, Robotics: Science and Systems.

[12]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[13]  Sergey Levine,et al.  Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning , 2018, ICLR.

[14]  Razvan Pascanu,et al.  Sim-to-Real Robot Learning from Pixels with Progressive Nets , 2016, CoRL.

[15]  Rajat Raina,et al.  Self-taught learning: transfer learning from unlabeled data , 2007, ICML '07.

[16]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[17]  Alberto Broggi,et al.  From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[18]  Amos Storkey,et al.  Meta-Learning in Neural Networks: A Survey , 2020, IEEE transactions on pattern analysis and machine intelligence.

[19]  Jianping Gou,et al.  Knowledge Distillation: A Survey , 2020, International Journal of Computer Vision.

[20]  Gregory Hager,et al.  “Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer , 2019, IEEE Robotics and Automation Letters.

[21]  Andrew G. Barto,et al.  Autonomous shaping: knowledge transfer in reinforcement learning , 2006, ICML.

[22]  Alan Fern,et al.  Transfer Learning in Sequential Decision Problems: A Hierarchical Bayesian Approach , 2012, ICML Unsupervised and Transfer Learning.

[23]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[24]  Jiayu Zhou,et al.  Transfer Learning in Deep Reinforcement Learning: A Survey , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Dongpu Cao,et al.  Virtual-to-Real Knowledge Transfer for Driving Behavior Recognition: Framework and a Case Study , 2019, IEEE Transactions on Vehicular Technology.

[26]  Jorge Pena Queralta,et al.  Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey , 2020, 2020 IEEE Symposium Series on Computational Intelligence (SSCI).

[27]  Lorenz Wellhausen,et al.  Learning quadrupedal locomotion over challenging terrain , 2020, Science Robotics.

[28]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[29]  Param S. Rajpura,et al.  Transfer Learning for Object Detection using State-of-the-Art Deep Neural Networks , 2018, 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN).

[30]  Jung-Min Park,et al.  Independent Joint Learning: A novel task-to-task transfer learning scheme for robot models , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[32]  Thomas Chaffre,et al.  Sim-to-Real Transfer with Incremental Environment Complexity for Reinforcement Learning of Depth-Based Robot Navigation , 2020, ICINCO.

[33]  Weiwei Shang,et al.  Transfer of Robot Perception Module With Adversarial Learning , 2019, IEEE Access.

[34]  Sanja Fidler,et al.  Meta-Sim: Learning to Generate Synthetic Datasets , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Antoine Cully,et al.  Robots that can adapt like animals , 2014, Nature.

[36]  Taghi M. Khoshgoftaar,et al.  A survey of transfer learning , 2016, Journal of Big Data.

[37]  David Jacobs,et al.  Adversarially robust transfer learning , 2020, ICLR.

[38]  Daniel Szafir,et al.  Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[39]  Gongfa Li,et al.  Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning , 2021, Sensors.

[40]  H. Jin Kim,et al.  Zero-Shot Transfer Learning of a Throwing Task via Domain Randomization , 2020, 2020 20th International Conference on Control, Automation and Systems (ICCAS).

[41]  Ramesh Nallapati,et al.  A Comparative Study of Methods for Transductive Transfer Learning , 2007 .

[42]  Lin Xie,et al.  From Simulation to Real-World Robotic Mobile Fulfillment Systems , 2019, Logist. Res..

[43]  Wojciech Zaremba,et al.  Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model , 2016, ArXiv.

[44]  Silvia Rossi,et al.  Personalized models for facial emotion recognition through transfer learning , 2020, Multimedia Tools and Applications.

[45]  Bernt Schiele,et al.  Meta-Transfer Learning Through Hard Tasks , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Malik Ghallab,et al.  Deliberation for autonomous robots: A survey , 2017, Artif. Intell..

[47]  Stephen Tyree,et al.  Sim-to-Real Transfer of Accurate Grasping with Eye-In-Hand Observations and Continuous Control , 2017, ArXiv.

[48]  Raymond J. Mooney,et al.  Transfer Learning from Minimal Target Data by Mapping across Relational Domains , 2009, IJCAI.

[49]  Yevgen Chebotar,et al.  Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[50]  Ioannis P. Vlahavas,et al.  Transfer Learning in Multi-Agent Reinforcement Learning Domains , 2011, EWRL.

[51]  Cewu Lu,et al.  Virtual to Real Reinforcement Learning for Autonomous Driving , 2017, BMVC.

[52]  Shaogang Gong,et al.  Unsupervised Transfer Learning with Self-Supervised Remedy , 2020, ArXiv.

[53]  Hod Lipson,et al.  Evolutionary Robotics for Legged Machines: From Simulation to Physical Reality , 2006, IAS.

[54]  Anca D. Dragan,et al.  SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards , 2019, ICLR.

[55]  Aliasghar Arab,et al.  Optimal Control of Wheeled Mobile Robots: From Simulation to Real World , 2020, 2020 American Control Conference (ACC).

[56]  Hooman Samani,et al.  Object Detection using Transfer Learning for Underwater Robot , 2020, 2020 International Conference on Advanced Robotics and Intelligent Systems (ARIS).

[57]  Jan Peters,et al.  Alignment-based transfer learning for robot models , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[58]  Marcin Andrychowicz,et al.  Sim-to-Real Transfer of Robotic Control with Dynamics Randomization , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[59]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[60]  Angela P. Schoellig,et al.  Multi-robot transfer learning: A dynamical system perspective , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[61]  Joel Veness,et al.  A Combinatorial Perspective on Transfer Learning , 2020, NeurIPS.

[62]  Marcin Andrychowicz,et al.  Solving Rubik's Cube with a Robot Hand , 2019, ArXiv.

[63]  Martial Hebert,et al.  Learning Transferable Policies for Monocular Reactive MAV Control , 2016, ISER.

[64]  Ian Lenz.  Deep Learning For Robotics , 2016 .

[65]  Bin Fang,et al.  Autoencoder-based transfer learning in brain–computer interface for rehabilitation robot , 2019, International Journal of Advanced Robotic Systems.

[66]  Sergey Levine,et al.  Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning , 2017, ICLR.

[67]  Jeff G. Schneider,et al.  Flexible Transfer Learning under Support and Model Shift , 2014, NIPS.

[68]  Peter Stone,et al.  Transfer Learning via Inter-Task Mappings for Temporal Difference Learning , 2007, J. Mach. Learn. Res..

[69]  Peter Stone,et al.  Transfer learning for reinforcement learning on a physical robot , 2010, AAMAS 2010.

[70]  Jose Ruiz-Pinales,et al.  Transfer Learning for Humanoid Robot Appearance-Based Localization in a Visual Map , 2021, IEEE Access.

[71]  Girish Chowdhary,et al.  Cross-Domain Transfer in Reinforcement Learning Using Target Apprentice , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).