Risk aware portfolio construction using deep deterministic policy gradients

Allocation of liquid capital to the financial instruments in a portfolio is typically done using a two-step process. In the first step, predictive techniques are used to determine the future risk and rewards for the instrument. In the subsequent step, a quadratic optimization problem is solved to obtain the allocation that maximizes a relevant measure of the portfolio performance computed using a combination of the risks and the rewards. Deep Reinforcement Learning (DRL) eliminates the need for a two step process to find the allocation across the instruments that will optimize a measure of portfolio performance obtained from the market. DRL based portfolio construction autonomously adjusts to a change in the environment unlike traditional machine learning algorithms used in prediction. The existing DRL methods suffer from the challenges of stability, and do not lend themselves well to the portfolio construction problem that has a continuous action space. Proposed in 2015, Deep Deterministic Policy Gradients (DDPG) is a type of actorcritic DRL algorithm that provides support for continuous action space which is encountered in portfolio construction. This paper evaluates the use of DDPG to solve the problem of risk aware portfolio construction. Simulations are done on a portfolio of twenty stocks and the use of both Rate of Return and Sortino ratio as a measure of portfolio performance are evaluated. Results are presented that demonstrate the effectiveness of DDPG for risk aware portfolio construction. The simulation results presented in this paper show that having a risk-aware measure of portfolio performance such as Sortino ratio give a portfolio with superior return and lower variance.

[1]  G. Hunanyan,et al.  Portfolio Selection , 2019, Finanzwirtschaft, Banken und Bankmanagement I Finance, Banks and Bank Management.

[2]  David W. Lu,et al.  Agent Inspired Trading Using Recurrent Reinforcement Learning and LSTM Neural Networks , 2017, 1707.07338.

[3]  Zhengyao Jiang,et al.  A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem , 2017, ArXiv.

[4]  Navdeep Jaitly,et al.  Discrete Sequential Prediction of Continuous Actions for Deep RL , 2017, ArXiv.

[5]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[6]  Marina Weber The Ascent Of Money A Financial History Of The World , 2016 .

[7]  Alexandra Gabriela Ţiţan The Efficient Market Hypothesis: Review of Specialized Literature and Empirical Research☆ , 2015 .

[8]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[9]  D. Chiszar The ascent of money: A financial history of the world , 2009 .

[10]  Louis Wehenkel,et al.  Risk-aware decision making and dynamic programming , 2008 .

[11]  Matthew Saffell,et al.  Learning to trade via direct reinforcement , 2001, IEEE Trans. Neural Networks.

[12]  Lai-Wan Chan,et al.  An Algorithm for Trading and Portfolio Management Using Q-learning and Sharpe Ratio Maximization , 2000 .

[13]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[14]  Ralph Neuneier,et al.  Optimal Asset Allocation using Adaptive Dynamic Programming , 1995, NIPS.

[15]  E. Fama EFFICIENT CAPITAL MARKETS: A REVIEW OF THEORY AND EMPIRICAL WORK* , 1970 .