Abstract:

Self-supervised metric learning has been a successful approach for learning a distance from an unlabeled dataset. The resulting distance is broadly useful for improving various distance-based downstream tasks, even when no information from downstream tasks is utilized in the metric learning stage. To gain insights into this approach, we develop a statistical framework to theoretically study how self-supervised metric learning can benefit downstream tasks in the context of multi-view data. Under this framework, we show that the target distance of metric learning satisfies several desired properties for the downstream tasks. On the other hand, our investigation suggests the target distance can be further improved by moderating each direction’s weights. In addition, our analysis precisely characterizes the improvement by self-supervised metric learning on four commonly used downstream tasks: sample identification, two-sample testing, k-means clustering, and k-nearest neighbor classification. When the distance is estimated from an unlabeled dataset, we establish the upper bound on distance estimation’s accuracy and the number of samples sufficient for downstream task improvement.

Abstract:

From both a statistical and practical perspective, the estimation of prediction uncertainty is a critical aspect of deep learning (DL) models. While standard deep learning models do not provide uncertainty estimates for their predictions, much recent research has focused on obtaining such estimates. Despite this, little attention has been given to the quality of estimated uncertainty from these methods. We will discuss the challenges of implementing the traditional metrics for assessing estimated uncertainty quality, interval coverage and width, for deep learning problems. Prediction interval coverage and width metrics will be given for several uncertainty-enabled deep learning models on (1) a simple regression problem and (2) a binary classification problem. We will also discuss both current and future research directions related to assessing uncertainty quality in DL models.