Using a Rotating 3D LiDAR on a Mobile Robot for Estimation of Person’s Body Angle and Gender

We studied the use of a rotating multi-layer 3D Light Detection And Ranging (LiDAR) sensor (specifically the Velodyne HDL-32E) mounted on a social robot for the estimation of features of people around the robot. While LiDARs are often used for robot self-localization and people tracking, we were interested in the possibility of using them to estimate the people’s features (states or attributes), which are important in human–robot interaction. In particular, we tested the estimation of the person’s body orientation and their gender. As collecting data in the real world and labeling them is laborious and time consuming, we also looked into other ways for obtaining data for training the estimators: using simulations, or using LiDAR data collected in the lab. We trained convolutional neural network-based estimators and tested their performance on actual LiDAR measurements of people in a public space. The results show that with a rotating 3D LiDAR a usable estimate of the body angle can indeed be achieved (mean absolute error 33.5°), and that using simulated data for training the estimators is effective. For estimating gender, the results are satisfactory (accuracy above 80%) when the person is close enough; however, simulated data do not work well and training needs to be done on actual people measurements.

[1]  Matteo Munaro,et al.  Fast RGB-D people tracking for service robots , 2014, Auton. Robots.

[2]  Simon Lucey,et al.  Argoverse: 3D Tracking and Forecasting With Rich Maps , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Manfred Tscheligi,et al.  To Err Is Robot: How Humans Assess and Act toward an Erroneous Social Robot , 2017, Front. Robot. AI.

[4]  Omar Y. Al-Jarrah,et al.  A Survey on 3D Object Detection Methods for Autonomous Driving Applications , 2019, IEEE Transactions on Intelligent Transportation Systems.

[5]  Diego Alonso,et al.  A Machine Learning Approach to Pedestrian Detection for Autonomous Vehicles Using High-Definition 3D Range Data , 2016, Sensors.

[6]  Wolfram Burgard,et al.  Deep 3D perception of people and their mobility aids , 2019, Robotics Auton. Syst..

[7]  Takayuki Kanda,et al.  Person Tracking in Large Public Spaces Using 3-D Range Sensors , 2013, IEEE Transactions on Human-Machine Systems.

[8]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[9]  Jaehoon Jung,et al.  Object Recognition, Segmentation, and Classification of Mobile Laser Scanning Point Clouds: A State of the Art Review , 2019, Sensors.

[10]  You Li,et al.  Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems , 2020, IEEE Signal Processing Magazine.

[11]  Kai Oliver Arras,et al.  People detection in RGB-D data , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[13]  Ji Wan,et al.  Multi-view 3D Object Detection Network for Autonomous Driving , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Horst-Michael Gross,et al.  Deep orientation: Fast and Robust Upper Body orientation Estimation for Mobile Robotic Applications , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[15]  Jianqiang Wang,et al.  Density Enhancement-Based Long-Range Pedestrian Detection Using 3-D Range Data , 2016, IEEE Transactions on Intelligent Transportation Systems.

[16]  Kai Oliver Arras,et al.  People tracking in RGB-D data with on-line boosted target models , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Ji Zhang,et al.  LOAM: Lidar Odometry and Mapping in Real-time , 2014, Robotics: Science and Systems.

[18]  Takayuki Kanda,et al.  Understanding a public environment via continuous robot observations , 2020, Robotics Auton. Syst..

[19]  Joachim Hertzberg,et al.  6D SLAM—3D mapping outdoor environments , 2007, J. Field Robotics.

[20]  Dragomir Anguelov,et al.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Shinpei Kato,et al.  LIBRE: The Multiple 3D LiDAR Dataset , 2020, 2020 IEEE Intelligent Vehicles Symposium (IV).

[22]  Wei Zhan,et al.  Fusing Bird’s Eye View LIDAR Point Cloud and Front View Camera Image for 3D Object Detection , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[23]  Takayuki Kanda,et al.  Facilitating Software Development for Mobile Social Robots by Simulating Interactions Between a Robot and Pedestrians , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[24]  Georg Schitter,et al.  MEMS-based lidar for autonomous driving , 2018, Elektrotech. Informationstechnik.

[25]  Junxuan Zhao,et al.  Detection and tracking of pedestrians and vehicles using roadside LiDAR sensors , 2019, Transportation Research Part C: Emerging Technologies.

[26]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[27]  Farzin Amzajerdian,et al.  Imaging Flash Lidar for Autonomous Safe Landing and Spacecraft Proximity Operation , 2016 .

[28]  Sebastian Thrun,et al.  Towards 3D object recognition via classification of arbitrary object tracks , 2011, 2011 IEEE International Conference on Robotics and Automation.

[29]  Cordelia Schmid,et al.  Learning from Synthetic Humans , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Wolfram Burgard,et al.  3D Human Pose Estimation in RGBD Images for Robotic Task Learning , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Christoph Stiller,et al.  Velodyne SLAM , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[32]  Tetsuo Ono,et al.  Stop Doing it! Approaching Strategy for a Robot to Admonish Pedestrians , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[33]  Jun Miura,et al.  Pedestrian Recognition Using High-definition LIDAR , 2011 .

[34]  Kai Oliver Arras,et al.  Real-time full-body human gender recognition in (RGB)-D data , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[35]  Paul Newman,et al.  What could move? Finding cars, pedestrians and bicyclists in 3D laser data , 2012, 2012 IEEE International Conference on Robotics and Automation.

[36]  Yi-Ting Chen,et al.  The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[37]  Tom Duckett,et al.  Online learning for 3D LiDAR-based human detection: experimental analysis of point cloud clustering and classification methods , 2019, Autonomous Robots.

[38]  Dietrich Paulus,et al.  Confidence-based pedestrian tracking in unstructured environments using 3D laser distance measurements , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Kai Oliver Arras,et al.  Tracking people in 3D using a bottom-up top-down detector , 2011, 2011 IEEE International Conference on Robotics and Automation.

[40]  Didier Stricker,et al.  DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth , 2018, 2018 International Conference on 3D Vision (3DV).

[41]  Kai Oliver Arras,et al.  Real-time full-body human attribute classification in RGB-D using a tessellation boosting approach , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[42]  日向 俊二.  Kinect for Windowsアプリを作ろう , 2012 .

[43]  Hang-Bong Kang,et al.  Object Detection and Classification by Decision-Level Fusion for Intelligent Vehicle Systems , 2017, Sensors.

[44]  Fei Liu,et al.  Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Federico Toschi,et al.  Pedestrian orientation dynamics from high-fidelity measurements , 2020, Scientific Reports.