Learning to Control Direct Current Motor for Steering in Real Time via Reinforcement Learning

Model free techniques have been successful at optimal control of complex systems at an expense of copious amounts of data and computation. However, it is often desired to obtain a control policy in a short period of time with minimal data use and computational burden. To this end, we make use of the NFQ algorithm for steering position control of a golf cart in both a real hardware and a simulated environment that was built from real-world interaction. The controller learns to apply a sequence of voltage signals in the presence of environmental uncertainties and inherent non-linearities that challenge the the control task. We were able to increase the rate of successful control under four minutes in simulation and under 11 minutes in real hardware.

[1]  Ming C. Lin,et al.  A Survey on Visual Traffic Simulation: Models, Evaluations, and Applications in Autonomous Driving , 2019, Comput. Graph. Forum.

[2]  Muhammad Aamir,et al.  On Replacing PID Controller with ANN Controller for DC Motor Position Control , 2013, ArXiv.

[3]  Weizi Li,et al.  Black-box Adversarial Attacks on Network-wide Multi-step Traffic State Prediction Models , 2021, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC).

[4]  Ming C. Lin,et al.  ADAPS: Autonomous Driving Via Principled Simulations , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[5]  Benjamin Van Roy.  Neuro-Dynamic Programming: Overview and Recent Trends , 2002 .

[6]  S. Levine,et al.  Conservative Q-Learning for Offline Reinforcement Learning , 2020, NeurIPS.

[7]  Nisreen Khamas,et al.  Model predictive control for DC motors , 2018, 2018 1st International Scientific Conference of Engineering Sciences - 3rd Scientific Conference of Engineering Science (ISCES).

[8]  S. Levine,et al.  Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems , 2020, ArXiv.

[9]  Wouter Caarls,et al.  Parallel Online Temporal Difference Learning for Motor Control , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Jui-Hong Horng,et al.  Neural Adaptive Tracking Control of a DC Motor , 1999, Inf. Sci..

[11]  Lingqiao Qin,et al.  Vehicle Trajectory Prediction Using LSTMs With Spatial–Temporal Attention Mechanisms , 2022, IEEE Intelligent Transportation Systems Magazine.

[12]  Srinivas Peeta,et al.  Predicting Station-Level Bike-Sharing Demands Using Graph Convolutional Neural Network , 2020, 2004.08723.

[13]  Ming C. Lin,et al.  Citywide Estimation of Traffic Dynamics via Sparse GPS Traces , 2017, IEEE Intelligent Transportation Systems Magazine.

[14]  Ming C. Lin,et al.  City-scale traffic animation using statistical learning and metamodel-based optimization , 2017, ACM Trans. Graph..

[15]  Jay H. Lee,et al.  Model predictive control: Review of the three decades of development , 2011 .

[16]  Martin A. Riedmiller.  Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method , 2005, ECML.

[17]  Martijn Wisse,et al.  The design of LEO: A 2D bipedal walking robot for online autonomous Reinforcement Learning , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[18]  Srinivas Peeta,et al.  Efficient Data Collection and Accurate Travel Time Estimation in a Connected Vehicle Environment Via Real-Time Compressive Sensing , 2019, Journal of Big Data Analytics in Transportation.

[19]  Martin A. Riedmiller,et al.  Learning to Drive a Real Car in 20 Minutes , 2007, 2007 Frontiers in the Convergence of Bioscience and Information Technologies.

[20]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[21]  Ming C. Lin,et al.  Estimating urban traffic states using iterative refinement and Wardrop equilibria , 2018, IET Intelligent Transport Systems.

[22]  Marco Rivera,et al.  Model Predictive Control for Power Converters and Drives: Advances and Trends , 2017, IEEE Transactions on Industrial Electronics.

[23]  P. Poongodi,et al.  Position Control of DC Motor Using Genetic Algorithm Based PID Controller , 2009 .

[24]  Arief Syaichu-Rohman,et al.  Model predictive control implementation on a programmable logic controller for DC motor speed control , 2011, Proceedings of the 2011 International Conference on Electrical Engineering and Informatics.

[25]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[26]  Alexander Carballo,et al.  A Survey of Autonomous Driving: Common Practices and Emerging Technologies , 2019, IEEE Access.

[27]  Bidyadhar Subudhi,et al.  Optimal speed control of DC motor using linear quadratic regulator and model predictive control , 2015, 2015 International Conference on Energy, Power and Environment: Towards Sustainable Growth (ICEPE).

[28]  Ming C. Lin,et al.  Virtualized Traffic at Metropolitan Scales , 2015, Front. Robot. AI.

[29]  Mohammad El-Bardini,et al.  Embedded Adaptive Fuzzy Controller Based on Reinforcement Learning for DC Motor with Flexible Shaft , 2015 .

[30]  Rohit G. Kanojiya,et al.  Tuning of PID controller using Ziegler-Nichols method for speed control of DC motor , 2012, IEEE-International Conference On Advances In Engineering, Science And Management (ICAESM -2012).

[31]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[32]  Martin A. Riedmiller.  10 Steps and Some Tricks to Set up Neural Reinforcement Controllers , 2012, Neural Networks: Tricks of the Trade.