Efficient local validation of partially ordered models via Baysian directed sampling

Title	Efficient local validation of partially ordered models via Baysian directed sampling
Authors	Kellan Moorse and Richard Murray
Source	Submitted, 2024 American Control Conference (ACC)
Abstract	We consider the problem of estimating the subset of test conditions under which a simplified model—or set of simplified models—accurately approximates the behavior of a true system. We approach the problem by proposing a compact set of possible test conditions, and an unknown but samplable continous validity function over that set that quantifies the accuracy of the model under each possible condition. We propose a novel Bayes estimator that optimally directs function sampling to greedily minimize the expected posterior misclassification rate of the valid set, which we call minimum posterior misclassification sampling (GP-MPM), and we show that the the method can be extended to approximate the valid sets of a partially ordered set of models, with sample complexity growing sublinearly with the number of models. In testing against a safety-focused model, we show that the algorithm’s estimated valid set approaches the true valid set much more quickly than undirected sampling, even with small sample sizes.
Type	Conference submission
URL	https://www.cds.caltech.edu/~murray/preprints/mm24-acc s.pdf
DOI
Tag	mm24-acc
ID	2023i
Funding	AFOSR T&E2
Flags	NCS