A deep reinforcement learning approach to mountain railway alignment optimization

The design and planning of railway alignments is the dominant task in railway construction. However, it is difficult to achieve self‐learning and learning from human experience with manual as well as automated design methods. Also, many existing approaches require predefined numbers of horizontal points of intersection or vertical points of intersection as input. To address these issues, this study employs deep reinforcement learning (DRL) to optimize mountainous railway alignments with the goal of minimizing construction costs. First, in the DRL model, the state of the railway alignment optimization environment is determined, and the action and reward function of the optimization agent are defined along with the corresponding alignment constraints. Second, we integrate a recent DRL algorithm called the deep deterministic policy gradient with optional human experience to obtain the final optimized railway alignment, and the influence of human experience is demonstrated through a sensitivity analysis. Finally, this methodology is applied to a real‐world case study in a mountainous region, and the results verify that the DRL approach used here can automatically explore and optimize the railway alignment, decreasing the construction cost by 17.65% and 7.98%, compared with the manual alignment and with the results of a method based on the distance transform, respectively, while satisfying various alignment constraints.

[1]  Bernard E. Howard,et al.  Optimum Curvature Principle In Highway Routing , 1968 .

[2]  Robert W Hayman.  OPTIMIZATION OF VERTICAL ALIGNMENT FOR HIGHWAYS THROUGH MATHEMATICAL PROGRAMMING , 1970 .

[3]  A Keith Turner,et al.  The GCARS System - a Computer-Assisted Method of Regional Route Location : Technical Paper , 1970 .

[4]  Alan J. Nicholson.  A variational approach to optimal route location , 1976 .

[5]  Neville A. Parker.  Rural highway route corridor selection , 1977 .

[6]  Jocelyn F. B. Shaw,et al.  COMPARISON OF TWO INTEGRATION METHODS IN TRANSPORTATION ROUTING , 1981 .

[7]  Jocelyn F. B. Shaw,et al.  Expressway Route Optimization by OCP , 1982 .

[8]  Ek Peng Chew,et al.  Discrete and continuous models for computation of optimal vertical highway alignment , 1988 .

[9]  Said M. Easa,et al.  Selection of roadway grades that minimize earthwork cost using linear programming , 1988 .

[10]  J. F. Sykes,et al.  Route selection through a dynamic ice field using the maximum principle , 1988 .

[11]  T. F. FwA,et al.  HIGHWAY VERTICAL ALIGNMENT ANALYSIS BY DYNAMIC PROGRAMMING , 1989 .

[12]  T. Fwa,et al.  SIMULTANEOUS OPTIMIZATION OF HORIZONTAL AND VERTICAL ALIGNMENTS FOR HIGHWAYS , 1989 .

[13]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[14]  Jeff R. Wright,et al.  Civil And Environmental Systems Engineering , 1996 .

[15]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[16]  J. C. Jong,et al.  Optimizing Highway Alignments with Genetic Algorithms , 1998 .

[17]  Paul Schonfeld,et al.  Integrating Genetic Algorithms and Geographic Information System to Optimize Highway Alignments , 2000 .

[18]  Paul Schonfeld,et al.  Using GIS, Genetic Algorithms, and Visualization in Highway Development , 2001 .

[19]  P. Schonfeld,et al.  An evolutionary model for simultaneously optimizing three-dimensional highway alignments , 2003 .

[20]  Lawrence Mandow,et al.  Sindi: an intelligent assistant for highway design , 2004, Expert Syst. Appl..

[21]  Paul Schonfeld,et al.  A highway alignment optimization model using geographic information systems , 2004 .

[22]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[23]  Charles Audet,et al.  Mesh Adaptive Direct Search Algorithms for Constrained Optimization , 2006, SIAM J. Optim..

[24]  Michael J. De Smith.  Determination of Gradient and Curvature Constrained Optimal Paths , 2006, Comput. Aided Civ. Infrastructure Eng..

[25]  Said M. Easa,et al.  Optimizing Design of Highway Horizontal Alignments: New Substantive Safety Approach , 2008, Comput. Aided Civ. Infrastructure Eng..

[26]  Yusin Lee,et al.  Optimization Method for Highway Horizontal Alignment Design , 2009 .

[27]  Manoj K. Jha,et al.  Multi‐objective highway alignment optimization using a genetic algorithm , 2009 .

[28]  Hojjat Adeli,et al.  Enhanced probabilistic neural network with local decision circles: A robust classifier , 2010, Integr. Comput. Aided Eng..

[29]  Lance Sherry,et al.  Accuracy of reinforcement learning algorithms for predicting aircraft taxi-out times: A case-study of Tampa Bay departures , 2010 .

[30]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[31]  Marco Wiering,et al.  Reinforcement Learning , 2014, Adaptation, Learning, and Optimization.

[32]  Paul Schonfeld,et al.  Applicability of highway alignment optimization models , 2012 .

[33]  Paul Schonfeld,et al.  Optimization of Rail Transit Alignments considering Vehicle Dynamics , 2012 .

[34]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[35]  Gaetano Bosurgi,et al.  A PSO Highway Alignment Optimization Algorithm Considering Environmental Constraints , 2013 .

[36]  Wei Li,et al.  Approach for Optimizing 3D Highway Alignments Based on Two-stage Dynamic Programming , 2013, J. Softw..

[37]  Yousef Shafahi,et al.  A Customized Particle Swarm Method to Solve Highway Alignment Optimization Problem , 2013, Comput. Aided Civ. Infrastructure Eng..

[38]  Shahadat Hossain,et al.  Models and strategies for efficiently determining an optimal vertical alignment of roads , 2014, Comput. Oper. Res..

[39]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[40]  Paul Schonfeld,et al.  Multi-objective highway alignment optimization incorporating preference information , 2014 .

[41]  Honglak Lee,et al.  Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning , 2014, NIPS.

[42]  Warren Hare,et al.  A mixed-integer linear programming model to optimize the vertical alignment considering blocks and side-slopes in road construction , 2015, Eur. J. Oper. Res..

[43]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[44]  Warren Hare,et al.  Optimizing horizontal alignment of roads in a specified corridor , 2015, Comput. Oper. Res..

[45]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[46]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[47]  Warren Hare,et al.  Multiple-path selection for new highway alignments using discrete algorithms , 2015, Eur. J. Oper. Res..

[48]  Mohammad Hossein Rafiei,et al.  A Novel Machine Learning Model for Estimation of Sale Prices of Real Estate Units , 2016 .

[49]  Paul Schonfeld,et al.  Methodology for optimizing constrained 3-dimensional railway alignments in mountainous terrain , 2016 .

[50]  Chunlin Chen,et al.  A novel DDPG method with prioritized experience replay , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[51]  Martin A. Riedmiller,et al.  Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards , 2017, ArXiv.

[52]  Wei Li,et al.  Mountain Railway Alignment Optimization with Bidirectional Distance Transform and Genetic Algorithm , 2017, Comput. Aided Civ. Infrastructure Eng..

[53]  Moshe Zukerman,et al.  Multiobjective Path Optimization for Critical Infrastructure Links with Consideration to Seismic Resilience , 2017, Comput. Aided Civ. Infrastructure Eng..

[54]  Peter Stone,et al.  Reinforcement learning , 2019, Scholarpedia.

[55]  Dushyant Rao,et al.  Large-scale cost function learning for path planning using deep inverse reinforcement learning , 2017, Int. J. Robotics Res..

[56]  Pieter Abbeel,et al.  Constrained Policy Optimization , 2017, ICML.

[57]  Ali Farhadi,et al.  Target-driven visual navigation in indoor scenes using deep reinforcement learning , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[58]  Shahram Sarkani,et al.  Many-objective stochastic path finding using reinforcement learning , 2017, Expert Syst. Appl..

[59]  Miguel E. Vázquez-Méndez,et al.  Optimization of horizontal alignment geometry in road design and reconstruction , 2017 .

[60]  Shahadat Hossain,et al.  Multi-haul quasi network flow model for vertical alignment optimization , 2017, 1701.01768.

[61]  Aleksandr I. Panov,et al.  Grid Path Planning with Deep Reinforcement Learning: Preliminary Results , 2017, BICA.

[62]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[63]  Saman K. Halgamuge,et al.  Optimal road design through ecologically sensitive areas considering animal migration dynamics , 2017 .

[64]  Jing Xin,et al.  Application of deep reinforcement learning in mobile robot path planning , 2017, 2017 Chinese Automation Congress (CAC).

[65]  Hojjat Adeli,et al.  A New Neural Dynamic Classification Algorithm , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[66]  Zhian Zhang,et al.  Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning , 2018, J. Robotics.

[67]  Miguel E. Vázquez-Méndez,et al.  A 3D Model for Optimizing Infrastructure Costs in Road Design , 2018, Comput. Aided Civ. Infrastructure Eng..

[68]  Hojjat Adeli,et al.  Novel Machine-Learning Model for Estimating Construction Costs Considering Economic Variables and Indexes , 2018, Journal of Construction Engineering and Management.

[69]  Yang Yang,et al.  Robot Path Planning in Dynamic Environments Based on Deep Reinforcement Learning , 2019 .

[70]  João Paulo Papa,et al.  FEMa: a finite element machine for fast learning , 2019, Neural Computing and Applications.

[71]  Wei Li,et al.  Concurrent optimization of mountain railway alignment and station locations using a distance transform algorithm , 2018, Comput. Ind. Eng..

[72]  Hojjat Adeli,et al.  A dynamic ensemble learning algorithm for neural networks , 2019, Neural Computing and Applications.

[73]  Wei Li,et al.  Mountain railway alignment optimization using stepwise & hybrid particle swarm optimization incorporating genetic operators , 2019, Appl. Soft Comput..

[75]  Jianping Hu,et al.  A three‐dimensional distance transform for optimizing constrained mountain railway alignments , 2019, Comput. Aided Civ. Infrastructure Eng..

[76]  Yuan Wang,et al.  Reinforcement learning approach for optimal control of multiple electric locomotives in a heavy-haul freight train: A Double-Switch-Q-network architecture , 2020, Knowl. Based Syst..

[77]  Avijit Maji,et al.  A modified motion planning algorithm for horizontal highway alignment development , 2020, Comput. Aided Civ. Infrastructure Eng..

[78]  Warren Hare,et al.  Fast feasibility check of the multi-material vertical alignment problem in road design , 2020, Comput. Optim. Appl..

[79]  Paul Schonfeld,et al.  Multi-objective railway alignment optimization considering costs and environmental impacts , 2020, Appl. Soft Comput..

[80]  Hong Zhang,et al.  Mountain railway alignment optimization considering geological impacts: A cost‐hazard bi‐objective model , 2020, Comput. Aided Civ. Infrastructure Eng..

[81]  Paul Schonfeld,et al.  Bi‐objective mountain railway alignment optimization incorporating seismic risk assessment , 2020, Comput. Aided Civ. Infrastructure Eng..