Evaluation Metrics of Object Detection for Quantitative System-Level Analysis of Autonomous Systems

Title	Evaluation Metrics of Object Detection for Quantitative System-Level Analysis of Autonomous Systems
Authors	Apurva Badithela, Tichakorn Wongpiromsarn and and Richard M. Murray
Source	To appear, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Abstract	This paper proposes two new metrics to evalu- ate learned object detection models for quantitative system-level analysis via probabilistic model-checking. In particular, proposition-labeled and distance-parametrized confusion matrices are defined for evaluating object detection, and these metrics are leveraged to compute the probability of the closed- loop system satisfying its system-level formal specifications. Instead of using object class labels, the proposition-labeled confusion matrix uses atomic propositions relevant to the high- level planning strategy. Furthermore, unlike the traditional confusion matrix, the distance-parametrized confusion matrix accounts for variations in detection performance with respect to the distance between the ego and the object. We empirically show that these evaluation metrics chosen with the context of i) system-level specifications and ii) the planning module lead to a less conservative analysis in comparison to canonical metrics that do not take these into account. We demonstrate this framework on a discrete-state car-pedestrian example by computing the satisfaction probabilities for safety requirements formalized in Linear Temporal Logic (LTL).
Type	Conference paper
URL	https://www.cds.caltech.edu/~murray/preprints/bwm23-iros s.pdf
DOI
Tag	bwm23-iros
ID	2023b
Funding	AFOSR T&E
Flags