A Reinforcement Learning Based 3D Guided Drilling Method: Beyond Ground Control

The current drilling guide operation relies on the two-way transmission of signals between the downhole drilling tools and the ground control center. However, the downhole environment is sometimes not conducive to such real-time signal transmission, and the analysis and decision-making in the ground involves complex human expert analysis and fine management. To deal with these problems, this paper proposes a downhole self-steering guided drilling method based on a reinforcement learning framework to achieve the 3D well trajectory design and control in real-time. In every time interval of the drilling process, the proposed system evaluates the drilling status and gives the adjustment action of drill bit in 3D space according to the received data, guiding the drill bit to the target reservoir without the involvement of human. The main module is a modified deep Q network using Sarsa algorithm for online self-learning. The experimental results show that after training, the drill bit is increasingly able to select control actions closer to the target reservoir. The frequency of effective actions is approximately 258% higher after the algorithm converges. The proposed system has the ability of online self-learning, which can automatically adjust the evaluation and decision models without manual monitoring.

[1]  Samantha Hansen,et al.  Using Deep Q-Learning to Control Optimization Hyperparameters , 2016, ArXiv.

[2]  Tom Schaul,et al.  Reinforcement Learning with Unsupervised Auxiliary Tasks , 2016, ICLR.

[3]  Benjamin Van Roy,et al.  Deep Exploration via Bootstrapped DQN , 2016, NIPS.

[4]  Li Zhaofeng,et al.  Application of 3D geosteering in geology-engineering integration practice , 2016 .

[5]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[6]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[7]  Paolo Ferraris,et al.  Geosteering And/Or Reservoir Characterization The Prowess Of New-Generation Lwd Tools , 2010 .

[8]  John Johnstone,et al.  The Value of Rotary Closed - Loop Drilling Technology on Chevron Alba's North Sea Horizontal Field Development , 2001 .

[9]  Trevor Darrell,et al.  Loss is its own Reward: Self-Supervision for Reinforcement Learning , 2016, ICLR.

[10]  S. G. Ponnambalam,et al.  Reinforcement learning: exploration–exploitation dilemma in multi-agent foraging task , 2012 .

[11]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[12]  Ramesh Raskar,et al.  Designing Neural Network Architectures using Reinforcement Learning , 2016, ICLR.

[13]  Ming Liu,et al.  Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[14]  Tzuu-Hseng S. Li,et al.  Backward Q-learning: The combination of Sarsa algorithm and Q-learning , 2013, Eng. Appl. Artif. Intell..

[15]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[16]  Zeb Kurth-Nelson,et al.  Learning to reinforcement learn , 2016, CogSci.

[17]  T. M. Warren.  Trends toward rotary steerable directional systems , 1997 .