论文标题
压缩：通过压缩表示形式进行自我监督的学习
CompRess: Self-Supervised Learning by Compressing Representations
论文作者
论文摘要
自我监督的学习旨在通过未标记的数据来学习良好的表示。最近的作品表明，较大的模型比较小的模型从自我监督的学习中受益更多。结果，对于大型模型，监督和自我监督学习之间的差距已大大减少。在这项工作中，我们没有为自我监督学习设计新的伪任务，而是开发了一种模型压缩方法来压缩已经学习过的，深厚的自我监督模型（教师）为较小的模型（学生）（学生）。我们训练学生模型，以模仿教师嵌入空间中数据点之间的相对相似性。对于Alexnet，我们的方法胜过所有以前的方法，包括Imagenet线性评估的完全监督模型（59.0％，而56.5％）和最近的邻居评估（50.7％，而41.4％）。据我们所知，这是自我监督的Alexnet首次在Imagenet分类上优于监督的Alexnet。我们的代码可在此处提供：https：//github.com/umbcvision/compress
Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between supervised and self-supervised learning has been greatly reduced for larger models. In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student). We train the student model so that it mimics the relative similarity between the data points in the teacher's embedding space. For AlexNet, our method outperforms all previous methods including the fully supervised model on ImageNet linear evaluation (59.0% compared to 56.5%) and on nearest neighbor evaluation (50.7% compared to 41.4%). To the best of our knowledge, this is the first time a self-supervised AlexNet has outperformed supervised one on ImageNet classification. Our code is available here: https://github.com/UMBCvision/CompRess
