# Evaluation Metrics of Object Detection for Quantitative System-Level Analysis of Autonomous Systems

From Murray Wiki

Jump to navigationJump to search
Title | Evaluation Metrics of Object Detection for Quantitative System-Level Analysis of Autonomous Systems |
---|---|

Authors | Apurva Badithela, Tichakorn Wongpiromsarn and and Richard M. Murray |

Source | To appear, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) |

Abstract | This paper proposes two new metrics to evalu- ate learned object detection models for quantitative system-level analysis via probabilistic model-checking. In particular, proposition-labeled and distance-parametrized confusion matrices are defined for evaluating object detection, and these metrics are leveraged to compute the probability of the closed- loop system satisfying its system-level formal specifications. Instead of using object class labels, the proposition-labeled confusion matrix uses atomic propositions relevant to the high- level planning strategy. Furthermore, unlike the traditional confusion matrix, the distance-parametrized confusion matrix accounts for variations in detection performance with respect to the distance between the ego and the object. We empirically show that these evaluation metrics chosen with the context of i) system-level specifications and ii) the planning module lead to a less conservative analysis in comparison to canonical metrics that do not take these into account. We demonstrate this framework on a discrete-state car-pedestrian example by computing the satisfaction probabilities for safety requirements formalized in Linear Temporal Logic (LTL). |

Type | Conference paper |

URL | https://www.cds.caltech.edu/~murray/preprints/bwm23-iros s.pdf |

DOI | |

Tag | bwm23-iros |

ID | 2023b |

Funding | AFOSR T&E |

Flags |