A Survey on Visual Traffic Simulation: Models, Evaluations, and Applications in Autonomous Driving

Virtualized traffic via various simulation models and real‐world traffic data are promising approaches to reconstruct detailed traffic flows. A variety of applications can benefit from the virtual traffic, including, but not limited to, video games, virtual reality, traffic engineering and autonomous driving. In this survey, we provide a comprehensive review on the state‐of‐the‐art techniques for traffic simulation and animation. We start with a discussion on three classes of traffic simulation models applied at different levels of detail. Then, we introduce various data‐driven animation techniques, including existing data collection methods, and the validation and evaluation of simulated traffic flows. Next, we discuss how traffic simulations can benefit the training and testing of autonomous vehicles. Finally, we discuss the current states of traffic simulation and animation and suggest future research directions.

[1]  WonkaPeter,et al.  Interactive procedural street modeling , 2008 .

[2]  Sterling J. Anderson,et al.  Design and Development of an Optimal-Control-Based Framework for Trajectory Planning, Threat Assessment, and Semi-autonomous Control of Passenger Vehicles in Hazard Avoidance Scenarios , 2009, ISRR.

[3]  Zuduo Zheng,et al.  Incorporating human-factors in car-following models : a review of recent developments and research needs , 2014 .

[4]  Eder Santana,et al.  Learning a Driving Simulator , 2016, ArXiv.

[5]  Daniel G. Aliaga,et al.  Designing large‐scale interactive traffic animations for urban modeling , 2014, Comput. Graph. Forum.

[6]  Michel Rascle,et al.  Resurrection of "Second Order" Models of Traffic Flow , 2000, SIAM J. Appl. Math..

[7]  Chung Choo Chung,et al.  Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[8]  Francisco J. Ros,et al.  A survey on modeling and simulation of vehicular networks: Communications, mobility, and tools , 2014, Comput. Commun..

[9]  Gita Alaghband,et al.  Scene-LSTM: A Model for Human Trajectory Prediction , 2018, ArXiv.

[10]  Wentong Cai,et al.  Crowd modeling and simulation technologies , 2010, TOMC.

[11]  Christos Katrakazas,et al.  Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions , 2015 .

[12]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ming C. Lin,et al.  WarpDriver: context-aware probabilistic motion prediction for crowd simulation , 2016, ACM Trans. Graph..

[14]  Zhigang Deng,et al.  A data-driven model for lane-changing in traffic simulation , 2016, Symposium on Computer Animation.

[15]  Ming C. Lin,et al.  City-scale traffic animation using statistical learning and metamodel-based optimization , 2017, ACM Trans. Graph..

[16]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[17]  H. Haj-Salem,et al.  The Aw-Rascle and Zhang's model: Vacuum problems, existence and regularity of the solutions of the Riemann problem , 2007 .

[18]  D. Helbing Traffic and related self-driven many-particle systems , 2000, cond-mat/0012229.

[19]  Lawrence A. Klein,et al.  Traffic Detector Handbook: Third Edition - Volume I , 2006 .

[20]  Robert Herman,et al.  Vehicular Traffic Flow , 1963 .

[21]  Ming C. Lin,et al.  Virtualized Traffic at Metropolitan Scales , 2015, Front. Robot. AI.

[22]  Fang Xu,et al.  Synthesizing Personalized Training Programs for Improving Driving Habits via Virtual Reality , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[23]  Daniel G. Aliaga,et al.  Example‐Driven Procedural Urban Roads , 2016, Comput. Graph. Forum.

[24]  Mohan M. Trivedi,et al.  Convolutional Social Pooling for Vehicle Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Ming C. Lin,et al.  Continuum Traffic Simulation , 2010, Comput. Graph. Forum.

[26]  B. Piccoli,et al.  Traffic Flow on a Road Network Using the Aw–Rascle Model , 2006 .

[27]  MaoTianlu,et al.  An efficient lane model for complex traffic simulation , 2015 .

[28]  Xiaogang Jin,et al.  Force-based Heterogeneous Traffic Simulation for Autonomous Vehicle Testing , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[29]  Alexandre M. Bayen,et al.  Evaluation of traffic data obtained via GPS-enabled mobile phones: The Mobile Century field experiment , 2009 .

[30]  Serge P. Hoogendoorn,et al.  Genealogy of traffic flow models , 2015, EURO J. Transp. Logist..

[31]  S. S. Ravi,et al.  Compression of trajectory data: a comprehensive evaluation and new approach , 2014, GeoInformatica.

[32]  Antonella Ferrara,et al.  Microscopic and Mesoscopic Traffic Models , 2018 .

[33]  Vladlen Koltun,et al.  Playing for Data: Ground Truth from Computer Games , 2016, ECCV.

[34]  Eric Galin,et al.  Procedural Generation of Roads , 2010, Comput. Graph. Forum.

[35]  Salissou Moutari,et al.  A Hybrid Lagrangian Model Based on the Aw--Rascle Traffic Flow Model , 2007, SIAM J. Appl. Math..

[36]  William L Eisele,et al.  TRAVEL TIME DATA COLLECTION HANDBOOK , 1998 .

[37]  Oliver Brock,et al.  Planning Long Dynamically-Feasible Maneuvers for Autonomous Vehicles , 2009 .

[38]  R. Jiang,et al.  Full velocity difference model for a car-following theory. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[39]  M. Kendall,et al.  ON THE METHOD OF PAIRED COMPARISONS , 1940 .

[40]  Eugene Zhang,et al.  Interactive procedural street modeling , 2008, ACM Trans. Graph..

[41]  Hema Swetha Koppula,et al.  Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[42]  Serge P. Hoogendoorn,et al.  Platoon-Based Multiclass Modeling of Multilane Traffic Flow , 2001 .

[43]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[44]  Ming C. Lin,et al.  Estimating urban traffic states using iterative refinement and Wardrop equilibria , 2018, IET Intelligent Transport Systems.

[45]  A. Schadschneider,et al.  Empirical test for cellular automaton models of traffic flow. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[46]  D. Manocha,et al.  AADS: Augmented autonomous driving simulation using data-driven algorithms , 2019, Science Robotics.

[47]  Norman I. Badler,et al.  Virtual Crowds: Methods, Simulation, and Control , 2008, Virtual Crowds: Methods, Simulation, and Control.

[48]  I. Lubashevsky,et al.  Probabilistic description of traffic breakdowns. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[49]  Dinesh Manocha,et al.  AutonoVi-Sim: Autonomous Vehicle Simulation Platform with Weather, Sensing, and Traffic Control , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[50]  Takashi Nagatani,et al.  Jam formation in traffic flow on a highway with some slowdown sections , 2007 .

[51]  Xiaobo Yu,et al.  Template-based generation of road networks for virtual city modeling , 2002, VRST '02.

[52]  Johann Marius Zöllner,et al.  Learning how to drive in a real world simulation with deep Q-Networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[53]  Hai-Jun Huang,et al.  Stability of the car-following model on two lanes. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[54]  Nakayama,et al.  Dynamical model of traffic congestion and numerical simulation. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[55]  Silvio Savarese,et al.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Gennady L. Andrienko,et al.  Exploratory analysis of spatial and temporal data - a systematic approach , 2005 .

[57]  Matthew Johnson-Roberson,et al.  Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks? , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[58]  Sebastian Thrun,et al.  Apprenticeship learning for motion planning with application to parking lot navigation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[59]  William E. Schiesser,et al.  Linear and nonlinear waves , 2009, Scholarpedia.

[60]  Anthony G. Brown,et al.  TRAFFIC DATA COLLECTION AND ANONYMOUS VEHICLE DETECTION USING WIRELESS SENSOR NETWORKS , 2012 .

[61]  Jordi Casas,et al.  Dynamic Network Simulation with AIMSUN , 2005 .

[62]  Alois Knoll,et al.  Deep neural networks for Markovian interactive scene prediction in highway scenarios , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[63]  Mark H. Overmars,et al.  Kinodynamic motion planning on roadmaps in dynamic environments , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[64]  Pradeep Dubey,et al.  Interactive hybrid simulation of large-scale traffic , 2011, SIGGRAPH Talks.

[65]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[66]  Dirk Helbing,et al.  MASTER: macroscopic traffic simulation based on a gas-kinetic, non-local traffic model , 2001 .

[67]  Ming C. Lin,et al.  ADAPS: Autonomous Driving Via Principled Simulations , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[68]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[69]  Wolfram Burgard,et al.  Learning driving styles for autonomous vehicles from demonstration , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[70]  Constantinos Antoniou,et al.  Traffic and mobility data collection for real-time applications , 2010, 13th International IEEE Conference on Intelligent Transportation Systems.

[71]  L C Davis,et al.  Multilane simulations of traffic phases. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[72]  Serge P. Hoogendoorn,et al.  Generic gas-kinetic traffic systems modeling with applications to vehicular traffic flow , 2001 .

[73]  D. Helbing,et al.  DERIVATION, PROPERTIES, AND SIMULATION OF A GAS-KINETIC-BASED, NONLOCAL TRAFFIC MODEL , 1999, cond-mat/9901240.

[74]  Fabrice Neyret,et al.  Real‐Time Rendering and Editing of Vector‐based Terrains , 2008, Comput. Graph. Forum.

[75]  Zhigang Deng,et al.  A Deep Learning-Based Framework for Intersectional Traffic Simulation and Editing , 2020, IEEE Transactions on Visualization and Computer Graphics.

[76]  David Silver,et al.  Learning Autonomous Driving Styles and Maneuvers from Expert Demonstration , 2012, ISER.

[77]  Ming C. Lin,et al.  Flow reconstruction for data-driven traffic animation , 2013, ACM Trans. Graph..

[78]  Xiaoru Yuan,et al.  Visual Exploration of Sparse Traffic Trajectory Data , 2014, IEEE Transactions on Visualization and Computer Graphics.

[79]  Srinivas Peeta,et al.  A Compressive Sensing Approach for Connected Vehicle Data Capture and Recovery and its Impact on Travel Time Estimation , 2018, 1806.10046.

[80]  Zhigang Deng,et al.  Vehicle–pedestrian interaction for mixed traffic simulation , 2015, Comput. Animat. Virtual Worlds.

[81]  Jean-Patrick Lebacque,et al.  Riemann Problem Resolution and Godunov Scheme for the Aw-Rascle-Zhang Model , 2009, Transp. Sci..

[82]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Hannes Hartenstein,et al.  A tutorial survey on vehicular ad hoc networks , 2008, IEEE Communications Magazine.

[84]  Ming C. Lin,et al.  Transforming GIS Data into Functional Road Models for Large-Scale Traffic Simulation , 2012, IEEE Transactions on Visualization and Computer Graphics.

[85]  Michael Schreckenberg,et al.  A cellular automaton model for freeway traffic , 1992 .

[86]  Zhigang Deng,et al.  AA‐FVDM: An accident‐avoidance full velocity difference model for animating realistic street‐level traffic in rural scenes , 2014, Comput. Animat. Virtual Worlds.

[87]  Kate Saenko,et al.  Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[88]  Markos Papageorgiou,et al.  Macroscopic traffic flow modeling with adaptive cruise control: Development and numerical solution , 2015, Comput. Math. Appl..

[89]  Philip H. S. Torr,et al.  DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[90]  Magnus Wrenninge,et al.  Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing , 2018, ArXiv.

[91]  Xiaogang Jin,et al.  Detailed traffic animation for urban road networks , 2012, Graph. Model..

[92]  H. M. Zhang A NON-EQUILIBRIUM TRAFFIC MODEL DEVOID OF GAS-LIKE BEHAVIOR , 2002 .

[93]  Mykel J. Kochenderfer,et al.  Imitating driver behavior with generative adversarial networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[94]  Dinesh Manocha,et al.  Virtualized Traffic: Reconstructing Traffic Flows from Discrete Spatio-Temporal Data , 2009, VR.

[95]  Ihor Lubashevsky,et al.  Probabilistic Description of Traffic Flow , 2001 .

[96]  Lawrence D. Jackel,et al.  Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car , 2017, ArXiv.

[97]  Eric P. Xing,et al.  CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving , 2018, ECCV.

[98]  P. I. Richards Shock Waves on the Highway , 1956 .

[99]  Narciso García,et al.  Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[100]  Vladlen Koltun,et al.  Playing for Benchmarks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[101]  Haris N. Koutsopoulos,et al.  Statistical Validation of Traffic Simulation Models , 2004 .

[102]  Zhigang Deng,et al.  A personality model for animating heterogeneous traffic behaviors , 2014, Comput. Animat. Virtual Worlds.

[103]  Christophe Hurter,et al.  Visualization, Selection, and Analysis of Traffic Flows , 2016, IEEE Transactions on Visualization and Computer Graphics.

[104]  Dizan Vasquez,et al.  A survey on motion prediction and risk assessment for intelligent vehicles , 2014, ROBOMECH Journal.

[105]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[106]  Alessio Del Bue,et al.  MX-LSTM: Mixing Tracklets and Vislets to Jointly Forecast Trajectories and Head Poses , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[107]  Luc Van Gool,et al.  End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners , 2018, ECCV.

[108]  Julien Perret,et al.  StreetGen : In base city scale procedural generation of streets: road network, road surface and street objects , 2018, ArXiv.

[109]  Germán Ros,et al.  CARLA: An Open Urban Driving Simulator , 2017, CoRL.

[110]  Paul Newman,et al.  1 year, 1000 km: The Oxford RobotCar dataset , 2017, Int. J. Robotics Res..

[111]  P. Moran On the method of paired comparisons. , 1947, Biometrika.

[112]  Serge P. Hoogendoorn,et al.  State-of-the-art of vehicular traffic flow modelling , 2001 .

[113]  Magnus Wrenninge,et al.  Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications , 2017, ArXiv.

[114]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[115]  Trevor Darrell,et al.  Textual Explanations for Self-Driving Vehicles , 2018, ECCV.

[116]  Harold J Payne,et al.  MODELS OF FREEWAY TRAFFIC AND CONTROL. , 1971 .

[117]  Hua Wang,et al.  An efficient lane model for complex traffic simulation , 2015, Comput. Animat. Virtual Worlds.

[118]  S. Hoogendoorn,et al.  Continuum modeling of multiclass traffic flow , 2000 .

[119]  Dirk Helbing,et al.  Microsimulations of Freeway Traffic Including Control Measures , 2002, cond-mat/0210096.

[120]  E. Williams Experimental Designs Balanced for the Estimation of Residual Effects of Treatments , 1949 .

[121]  M J Lighthill,et al.  On kinematic waves II. A theory of traffic flow on long crowded roads , 1955, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[122]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[123]  Zhigang Deng,et al.  Dictionary-based Fidelity Measure for Virtual Traffic , 2020, IEEE Transactions on Visualization and Computer Graphics.

[124]  R. LeVeque Finite Volume Methods for Hyperbolic Problems: Characteristics and Riemann Problems for Linear Hyperbolic Equations , 2002 .

[125]  Silvio Savarese,et al.  Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[126]  Serge P. Hoogendoorn,et al.  A review on travel behaviour modelling in dynamic traffic simulation models for evacuations , 2012 .

[127]  Harilaos N. Koutsopoulos,et al.  A microscopic traffic simulator for evaluation of dynamic traffic management systems , 1996 .

[128]  LUISA FERMO,et al.  A Fully-Discrete-State Kinetic Theory Approach to Modeling Vehicular Traffic , 2013, SIAM J. Appl. Math..

[129]  Guillaume Leduc,et al.  Road Traffic Data: Collection Methods and Applications , 2008 .

[130]  Xiaogang Jin,et al.  Video-based personalized traffic learning , 2013, Graph. Model..

[131]  Javier Alonso-Mora,et al.  Planning and Decision-Making for Autonomous Vehicles , 2018, Annu. Rev. Control. Robotics Auton. Syst..

[132]  Eric P. Xing,et al.  Real-to-Virtual Domain Unification for End-to-End Autonomous Driving , 2018, ECCV.

[133]  Hua Wang,et al.  Shadow traffic: A unified model for abnormal traffic behavior simulation , 2018, Comput. Graph..

[134]  Cewu Lu,et al.  Virtual to Real Reinforcement Learning for Autonomous Driving , 2017, BMVC.

[135]  L. A. Pipes An Operational Analysis of Traffic Dynamics , 1953 .

[136]  Jean Oh,et al.  Social Attention: Modeling Attention in Human Crowds , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[137]  Takashi Nagatani,et al.  Transition and saturation of traffic flow controlled by traffic lights , 2003 .

[138]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[139]  Zhigang Deng,et al.  Realistic Data-Driven Traffic Flow Animation Using Texture Synthesis , 2018, IEEE Transactions on Visualization and Computer Graphics.

[140]  Dirk Helbing,et al.  General Lane-Changing Model MOBIL for Car-Following Models , 2007 .

[141]  Vidya N. Murali,et al.  DeepLanes: End-To-End Lane Position Estimation Using Deep Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[142]  Shenghua Gao,et al.  Encoding Crowd Interaction with Deep Neural Network for Pedestrian Trajectory Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[143]  Sylvain Piechowiak,et al.  A behavioral multi-agent model for road traffic simulation , 2008, Eng. Appl. Artif. Intell..

[144]  Geoffrey J. Gordon,et al.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , 2010, AISTATS.

[145]  Ming C. Lin,et al.  Citywide Estimation of Traffic Dynamics via Sparse GPS Traces , 2017, IEEE Intelligent Transportation Systems Magazine.

[146]  M. Rascle An improved macroscopic model of traffic flow: Derivation and links with the Lighthill-Whitham model , 2002 .

[147]  I. Prigogine,et al.  A Boltzmann-Like Approach for Traffic Flow , 1960 .

[148]  JinXiaogang,et al.  Vehicle-pedestrian interaction for mixed traffic simulation , 2015 .

[149]  Michael Weinmann,et al.  StreetGAN: towards road network synthesis with generative adversarial networks , 2017 .

[150]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[151]  Pascal Müller,et al.  Procedural modeling of cities , 2001, SIGGRAPH.

[152]  Werner Huber,et al.  Experience, Results and Lessons Learned from Automated Driving on Germany's Highways , 2015, IEEE Intelligent Transportation Systems Magazine.

[153]  Cewu Lu,et al.  LiDAR-Video Driving Dataset: Learning Driving Policies Effectively , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[154]  Xiaoru Yuan,et al.  Visual Traffic Jam Analysis Based on Trajectory Data , 2013, IEEE Transactions on Visualization and Computer Graphics.

[155]  Silvio Savarese,et al.  Social LSTM: Human Trajectory Prediction in Crowded Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[156]  Rahul Narain,et al.  Aggregate dynamics for dense crowd simulation , 2009, SIGGRAPH 2009.

[157]  Christos Dimitrakakis,et al.  TORCS, The Open Racing Car Simulator , 2005 .