Reinforcement Learning for Active Flow Control in Experiments

We demonstrate experimentally the feasibility of applying reinforcement learning (RL) in flow control problems by automatically discovering active control strategies without any prior knowledge of the flow physics. We consider the turbulent flow past a circular cylinder with the aim of reducing the cylinder drag force or maximizing the power gain efficiency by properly selecting the rotational speed of two small diameter cylinders, parallel to and located downstream of the larger cylinder. Given properly designed rewards and noise reduction techniques, after tens of towing experiments, the RL agent could discover the optimal control strategy, comparable to the optimal static control. While RL has been found to be effective in recent computer flow simulation studies, this is the first time that its effectiveness is demonstrated experimentally, paving the way for exploring new optimal active flow control strategies in complex fluid mechanics applications.

[1]  Lakshminarayanan Mahadevan,et al.  Controlled gliding and perching through deep-reinforcement-learning , 2019, Physical Review Fluids.

[2]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[3]  Alexander Kuhnle,et al.  Direct shape optimization through deep reinforcement learning , 2019, Journal of Computational Physics.

[4]  Babak Hejazialhosseini,et al.  Reinforcement Learning and Wavelet Adapted Vortex Methods for Simulations of Self-propelled Swimmers , 2014, SIAM J. Sci. Comput..

[5]  Jean-François Beaudoin,et al.  Drag reduction of a bluff body using adaptive control methods , 2006 .

[6]  B. R. Noack,et al.  Closed-Loop Turbulence Control: Progress and Challenges , 2015 .

[7]  Jean Rabault,et al.  Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control , 2018, Journal of Fluid Mechanics.

[8]  Petros Koumoutsakos,et al.  Machine Learning for Fluid Mechanics , 2019, Annual Review of Fluid Mechanics.

[9]  Herke van Hoof,et al.  Addressing Function Approximation Error in Actor-Critic Methods , 2018, ICML.

[10]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[11]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[12]  L. Prandtl,et al.  Report on Investigation of Developed Turbulence , 1949 .

[13]  Petros Koumoutsakos,et al.  Efficient collective swimming by harnessing vortices through deep reinforcement learning , 2018, Proceedings of the National Academy of Sciences.

[14]  Gabriel D. Weymouth.  Lily Pad: towards real-time interactive computational fluid dynamics , 2015 .

[15]  M. Triantafyllou,et al.  Flow control with rotating cylinders , 2017, Journal of Fluid Mechanics.

[16]  Hongjun Zhu,et al.  Simultaneous CFD evaluation of VIV suppression using smaller control cylinders , 2015 .

[17]  Paul Zarchan,et al.  Fundamentals of Kalman Filtering: A Practical Approach , 2001 .

[18]  Antonio Celani,et al.  Flow Navigation by Smart Microswimmers via Reinforcement Learning , 2017, Physical review letters.

[19]  Bo Ren,et al.  Fluid directed rigid body control using deep reinforcement learning , 2018, ACM Trans. Graph..