论文标题
用于机器学习的简单有效的张量计算
A Simple and Efficient Tensor Calculus for Machine Learning
论文作者
论文摘要
张量表达式的计算衍生物，也称为张量计算，是机器学习中的一项基本任务。一个关键问题是评估表达式表示这些表达式的表达式及其衍生物的效率。最近，引入了一种用于计算张量表达式的高阶衍生物（如雅各布人或黑森）的算法，该算法比以前的最新方法快几个数量级。不幸的是，该方法基于RICCI符号，因此不能将使用更简单的爱因斯坦符号的深度学习中的自动分化框架纳入自动分化框架中。这留下了两个选项，要么在这些框架中更改基本张量表示，要么基于爱因斯坦符号开发了一种新的，可证明的算法。显然，第一个选择是不切实际的。因此，我们追求第二种选择。在这里，我们表明，使用RICCI符号对于有效的张量计算并不是必需的，并为更简单的爱因斯坦符号开发了同样有效的方法。事实证明，转向爱因斯坦符号可以进一步改进，从而提高效率。 本文描述的方法已在在线工具www.matrixcalculus.org中实现，用于计算矩阵和张量表达式的衍生物。 本文的扩展摘要似乎是“简单有效的张量演算”，AAAI 2020。
Computing derivatives of tensor expressions, also known as tensor calculus, is a fundamental task in machine learning. A key concern is the efficiency of evaluating the expressions and their derivatives that hinges on the representation of these expressions. Recently, an algorithm for computing higher order derivatives of tensor expressions like Jacobians or Hessians has been introduced that is a few orders of magnitude faster than previous state-of-the-art approaches. Unfortunately, the approach is based on Ricci notation and hence cannot be incorporated into automatic differentiation frameworks from deep learning like TensorFlow, PyTorch, autograd, or JAX that use the simpler Einstein notation. This leaves two options, to either change the underlying tensor representation in these frameworks or to develop a new, provably correct algorithm based on Einstein notation. Obviously, the first option is impractical. Hence, we pursue the second option. Here, we show that using Ricci notation is not necessary for an efficient tensor calculus and develop an equally efficient method for the simpler Einstein notation. It turns out that turning to Einstein notation enables further improvements that lead to even better efficiency. The methods that are described in this paper have been implemented in the online tool www.MatrixCalculus.org for computing derivatives of matrix and tensor expressions. An extended abstract of this paper appeared as "A Simple and Efficient Tensor Calculus", AAAI 2020.
