SURF 2024: Task-Relevant Metrics for Perception

2024 SURF Task-Relevant Metrics for Perception

• Mentor: Richard Murray
• Co-mentor: Apurva Badithela

Project Description

Caption: System-level requirements are easier to formalize than requirements on perception tasks.

Problem

In this SURF, we will explore the interface between perception and planning more carefully. Misclassification or misdetection in a single frame is unlikely to trigger a different decision from the planner. Therefore, we need to incorporate a notion of tracking objects across multiple frames to make system-level evaluations less conservative. This will require identifying new metrics beyond confusion matrices to capture detection performance across multiple frames. While there exist metrics to evaluate tracking, these metrics are not informed by the system-level task [1].

Goals for this SURF include:

• Proposing new metrics for tracking or other perception tasks, and rigorously connecting these metrics to system-level evaluations of safety.
• Evaluating state-of-the-art perception models on the nuScenes dataset with respect to tracking metrics derived from system-level specifications
• Time permitting, to validate theoretical results on a hardware platform such as Duckietown.

Desired:

• Experience programming in Python, ROS, OpenCV.
• Coursework in control, robotics, computer vision.
• Interest in theoretical research, robotics, and working with hardware, and industry datasets such as nuScenes.

References:

[1]