The Taub Faculty of Computer Science Events and Talks

In this seminar, I will discuss two of my papers (published in ICLR 2023) on uncertainty estimation and class-out-of-distribution detection. We present a novel framework to benchmark the ability of image classifiers to detect class-out-of-distribution (C-OOD) instances (i.e., instances whose true labels do not appear in the training distribution) at various levels of detection difficulty. We apply this technique to ImageNet, and benchmark 500+ pretrained, publicly available, ImageNet-1k classifiers. We then evaluate these classifiers both for their C-OOD detection performance (second paper) and for their uncertainty estimation performance (first paper), i.e., their ranking, calibration, and selective classification performance. This results in a large-scale study that reveals many factors (such as architecture types and training regimes) previously unknown to contribute to performance.