Evaluation of Maritime Event Detection Against Missing Data

Detecting and preventing maritime events like collisions or unusual behaviour of vessels are of high importance for maritime safety and security. As the trust of human operators in automated maritime event detection and prediction depends on the quality of the corresponding algorithms, the evaluation methodology becomes a driving force for the future development of maritime event detection and forecasting methods. The main contribution of this article consists in the development of an evaluation methodology and its application to a selected set of maritime event detectors. The approach links a reference dataset, controlled data variations, maritime event detection algorithms with internal parameters, and performance criteria. Among pre-established possible input data variations applied to a reference Automatic Identification System (AIS) dataset, the article focuses on the evaluation of detection accuracy of maritime event detectors implemented with the Event Calculus logical language against variable amounts of missing data, as a frequently observable type of AIS data degradation. Twelve maritime event pattern detectors are evaluated and most of them are found to vary very little in performance while only one detector shows an unexpected strong performance drop giving insights into how to improve the detection method. Results are provided on a real AIS data enriched with specific simulated events.