论文标题
使用人均推理有效的语义视频细分
Efficient Semantic Video Segmentation with Per-frame Inference
论文作者
论文摘要
对于语义细分，大多数现有的实时深层模型独立训练了每个框架，可能会为视频序列产生不一致的结果。高级方法考虑了视频序列中的相关性，例如，使用光流传播到相邻帧的结果，或使用其他帧提取帧表示，这可能导致结果不准确或延迟不平衡。在这项工作中，我们在推理过程中以人均方式处理有效的语义视频细分。与以前的人均模型不同，我们明确将框架之间的时间一致性视为训练过程中的额外约束，并将时间一致性嵌入到分割网络中。因此，在推论过程中，我们可以独立处理每个框架而没有延迟，并提高时间一致性，而没有额外的计算成本和后处理。我们采用紧凑型模型进行实时执行。为了缩小紧凑型模型和大型模型之间的性能差距，设计了新的知识蒸馏方法。我们的结果优于以前的基于密钥帧的方法，在流行基准的准确性和推理速度（包括CityScapes和Camvid）之间取决于更好的权衡。与相应的基线相比，时间一致性也得到了提高，这些基线与每个框架进行了独立训练。代码可在以下网址找到：https：//tinyurl.com/segment-video
For semantic segmentation, most existing real-time deep models trained with each frame independently may produce inconsistent results for a video sequence. Advanced methods take into considerations the correlations in the video sequence, e.g., by propagating the results to the neighboring frames using optical flow, or extracting the frame representations with other frames, which may lead to inaccurate results or unbalanced latency. In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. Different from previous per-frame models, we explicitly consider the temporal consistency among frames as extra constraints during the training process and embed the temporal consistency into the segmentation network. Therefore, in the inference process, we can process each frame independently with no latency, and improve the temporal consistency with no extra computational cost and post-processing. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed. Our results outperform previous keyframe based methods with a better trade-off between the accuracy and the inference speed on popular benchmarks, including the Cityscapes and Camvid. The temporal consistency is also improved compared with corresponding baselines which are trained with each frame independently. Code is available at: https://tinyurl.com/segment-video
