论文标题

主观注释的视觉识别任务的贝叶斯评估框架

A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks

论文作者

Prijatelj, Derek S., McCurrie, Mel, Scheirer, Walter J.

论文摘要

自动视觉识别的一个有趣的发展是任务的出现,在这些任务中,无法将客观标签分配给图像,但仍可为收集反映人类对其判断的注释而仍然可行。这些任务的基于机器学习的预测因素依赖于监督培训,该培训模拟了注释者的行为,即,普通人对图像的判断是什么?这类工作的一个关键开放问题,尤其是对于与人类行为不一致可能导致道德失误的应用,如何评估训练有素的预测因子的认知不确定性,即来自预测指标模型带来的不确定性。我们提出了一个贝叶斯框架,用于评估该制度中的黑匣子预测因子,对预测指标的内部结构不可知。该框架指定了如何通过近似条件分布并为预测及其绩效指标产生可靠的间隔,从而估计来自人类标签的认知不确定性。该框架成功应用于使用主观人类判断的四个图像分类任务:面部美容评估,社会属性分配,明显的年龄估计和模棱两可的场景标签。

An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels to images, yet still feasible to collect annotations that reflect human judgements about them. Machine learning-based predictors for these tasks rely on supervised training that models the behavior of the annotators, i.e., what would the average person's judgement be for an image? A key open question for this type of work, especially for applications where inconsistency with human behavior can lead to ethical lapses, is how to evaluate the epistemic uncertainty of trained predictors, i.e., the uncertainty that comes from the predictor's model. We propose a Bayesian framework for evaluating black box predictors in this regime, agnostic to the predictor's internal structure. The framework specifies how to estimate the epistemic uncertainty that comes from the predictor with respect to human labels by approximating a conditional distribution and producing a credible interval for the predictions and their measures of performance. The framework is successfully applied to four image classification tasks that use subjective human judgements: facial beauty assessment, social attribute assignment, apparent age estimation, and ambiguous scene labeling.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源