Reference Systems for Environmental Perception: Requirements, Validation and Metric-based Evaluation

The performance of advanced driver assistance systems (ADAS) is strongly dependent on the quality of the environmental perception sensors and algorithms used. This quality needs therefore to be assessed in order to allow an exhaustive evaluation of the ADAS perception system. For the quantitative assessment of the perception sensors and algorithms, a reference system, that can provide ground-truth information on the environment of a vehicle with higher quality than the ADAS sensors, is required. This paper defines and discusses requirements that a system has to satisfy in order to be used as reference system. In addition, a typical example of reference systems for ADAS perception is evaluated in terms of satisfaction of these requirements and validation with help of other external systems. Furthermore, this paper describes a metric-based evaluation of object-based perception systems with help of a this reference system.