Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review

Algorithmic stock trading has become a staple in today’s financial market, the majority of trades being now fully automated. Deep Reinforcement Learning (DRL) agents proved to be to a force to be reckon with in many complex games like Chess and Go. We can look at the stock market historical price series and movements as a complex imperfect information environment in which we try to maximize return profit and minimize risk. This paper reviews the progress made so far with deep reinforcement learning in the subdomain of AI in finance, more precisely, automated low-frequency quantitative stock trading. Many of the reviewed studies had only proof-of-concept ideals with experiments conducted in unrealistic settings and no real-time trading applications. For the majority of the works, despite all showing statistically significant improvements in performance compared to established baseline strategies, no decent profitability level was obtained. Furthermore, there is a lack of experimental testing in real-time, online trading platforms and a lack of meaningful comparisons between agents built on different types of DRL or human traders. We conclude that DRL in stock trading has showed huge applicability potential rivalling professional traders under strong assumptions, but the research is still in the very early stages of development.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  K. Šrédl,et al.  Commodity Channel Index: Evaluation of Trading Rule of Agricultural Commodities , 2016 .

[3]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[4]  G. Ritter.  Machine Learning for Trading , 2017 .

[5]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[6]  Andrew Clare,et al.  An Evaluation of Alternative Equity Indices - Part 1: Heuristic and Optimised Weighting Schemes , 2013 .

[7]  Amir Mosavi,et al.  Deep Learning for Stock Market Prediction , 2020, Entropy.

[8]  Jia WU,et al.  Quantitative Trading on Stock Market Based on Deep Reinforcement Learning , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[9]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[10]  Lin Chen,et al.  Application of Deep Reinforcement Learning on Automated Stock Trading , 2019, 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS).

[11]  T. Chong,et al.  Profitability of the On-Balance Volume Indicator , 2009 .

[12]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[13]  W. Ongsakul,et al.  Sortino ratio based portfolio optimization considering EVs and renewable energy in microgrid power market , 2017, 2017 IEEE Power & Energy Society General Meeting.

[14]  James C. Bezdek,et al.  Measuring fuzzy uncertainty , 1994, IEEE Trans. Fuzzy Syst..

[15]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[16]  Youyong Kong,et al.  Deep Direct Reinforcement Learning for Financial Signal Representation and Trading , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Chin-Teng Lin,et al.  Neural-Network-Based Fuzzy Logic Control and Decision System , 1991, IEEE Trans. Computers.

[18]  Brad M. Barber,et al.  The Behavior of Individual Investors , 2011 .

[19]  Yao Hua Ooi,et al.  Time Series Momentum , 2011 .

[20]  Thomas G. Fischer,et al.  Reinforcement learning in financial markets - a survey , 2018 .

[21]  Matthew Saffell,et al.  Learning to trade via direct reinforcement , 2001, IEEE Trans. Neural Networks.

[22]  W. Sharpe.  The Sharpe Ratio , 1994 .

[23]  Xiao-Yang Liu,et al.  Practical Deep Reinforcement Learning Approach for Stock Trading , 2018, ArXiv.

[24]  Farzan Soleymani,et al.  Financial portfolio optimization with online deep reinforcement learning and restricted stacked autoencoder - DeepBreath , 2020, Expert Syst. Appl..

[25]  Campbell R. Harvey,et al.  Dissecting Investment Strategies in the Cross Section and Time Series , 2015 .

[26]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[27]  Hiok Chai Quek,et al.  Stock trading with cycles: A financial application of ANFIS and reinforcement learning , 2011, Expert Syst. Appl..

[28]  Luigi Troiano,et al.  Adaptive stock trading strategies with deep reinforcement learning methods , 2020, Inf. Sci..

[29]  Wei Wang,et al.  Generalized Autoencoder: A Neural Network Framework for Dimensionality Reduction , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[30]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[31]  Marco Corazza,et al.  Testing different Reinforcement Learning con?gurations for ?nancial trading: Introduction and applications , 2018 .

[32]  Robert W. Colby,et al.  The Encyclopedia of Technical Market Indicators , 1988 .

[33]  Youyong Kong,et al.  Sparse Coding-Inspired Optimal Trading System for HFT Industry , 2015, IEEE Transactions on Industrial Informatics.

[34]  Stefan Zohren,et al.  Enhancing Time-Series Momentum Strategies Using Deep Neural Networks , 2019, The Journal of Financial Data Science.

[35]  J. Wilder.  New Concepts in Technical Trading Systems , 1978 .

[36]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[37]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[38]  Xiao-Yang Liu,et al.  Deep reinforcement learning for automated stock trading: an ensemble strategy , 2020, ICAIF.

[39]  Zihao Zhang,et al.  Deep Reinforcement Learning for Trading , 2019, The Journal of Financial Data Science.

[40]  Ikhlaas Gurrib.  “ Performance of the Average Directional Index as a market timing tool for the most actively traded USD based currency pairs ” , 2019 .

[41]  Chien-Yi Huang,et al.  Financial Trading as a Game: A Deep Reinforcement Learning Approach , 2018, ArXiv.

[42]  P. Arumugam,et al.  Financial Stock Market Forecast using Data Mining Techniques , 2010 .

[43]  Yanran Li,et al.  Adversarial Deep Reinforcement Learning in Portfolio Management , 2018 .

[44]  Rudy Prabowo,et al.  Sentiment analysis: A combined approach , 2009, J. Informetrics.

[45]  Zhang Xiong,et al.  AlphaStock: A Buying-Winners-and-Selling-Losers Investment Strategy using Interpretable Deep Reinforcement Attention Networks , 2019, KDD.

[46]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[47]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[48]  Marcus B. Perry,et al.  The Exponentially Weighted Moving Average , 2010 .

[49]  Yongduan Song,et al.  Fault-Tolerant Adaptive Control of High-Speed Trains Under Traction/Braking Failures: A Virtual Parameter-Based Approach , 2014, IEEE Transactions on Intelligent Transportation Systems.

[50]  Mamatha V. Jadhav,et al.  Stock Trading Bot Using Deep Reinforcement Learning , 2019 .