Symbolic tensor differentiation for applications in machine learning

Abstract

Automated methods for computing derivatives of cost functions are essential to many modern applications of machine learning. Reverse-mode automatic differentiation provides relatively cheap means for it but generated code often requires a lot of memory and is hardly amenable to later optimizations. Symbolic differentiation, on the other hand, generates much more flexible code, yet applying it to multidimensional tensors is a poorly studied topic. In this paper presents a method for symbolic tensor differentiation based on extended Einstein indexing notation, which allows to overcome many limitation of both - automatic and classic symbolic differentiation, and generate efficient code for CPL and GPU.