论文标题
正交SVD协方差条件和潜在的分解
Orthogonal SVD Covariance Conditioning and Latent Disentanglement
论文作者
论文摘要
将SVD元层插入神经网络中很容易使协方差不良，这可能会损害训练稳定性和概括能力中的模型。在本文中，我们系统地研究了如何通过对前SVD层的正交性来改善协方差调节。首先研究了重量的现有正交治疗。但是，这些技术可以改善条件，但会损害性能。为了避免这种副作用，我们提出了最近的正交梯度（NOG）和最佳学习率（OLR）。我们方法的有效性在两个应用程序中得到了验证：反相关的批处理归一化（BN）和全局协方差池（GCP）。关于视觉识别的广泛实验表明，我们的方法可以同时改善协方差调节和泛化。与正交重量的组合可以进一步提高性能。此外，我们表明我们的正交技术可以通过一系列在各种基准上进行一系列实验来使生成模型有益于更好的潜在分解。代码可在：\ href {https://github.com/kingjamessong/orthoimprovecond} {https://github.com/kingjamessong/orthoimprovecond}中获得。
Inserting an SVD meta-layer into neural networks is prone to make the covariance ill-conditioned, which could harm the model in the training stability and generalization abilities. In this paper, we systematically study how to improve the covariance conditioning by enforcing orthogonality to the Pre-SVD layer. Existing orthogonal treatments on the weights are first investigated. However, these techniques can improve the conditioning but would hurt the performance. To avoid such a side effect, we propose the Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR). The effectiveness of our methods is validated in two applications: decorrelated Batch Normalization (BN) and Global Covariance Pooling (GCP). Extensive experiments on visual recognition demonstrate that our methods can simultaneously improve covariance conditioning and generalization. The combinations with orthogonal weight can further boost the performance. Moreover, we show that our orthogonality techniques can benefit generative models for better latent disentanglement through a series of experiments on various benchmarks. Code is available at: \href{https://github.com/KingJamesSong/OrthoImproveCond}{https://github.com/KingJamesSong/OrthoImproveCond}.
