Developing a Seq2Seq neural network using visual attention to transform mathematical expressions from images to LaTeX

Abstract

The presented neural network with the seq2seq architecture and Attention mechanism successfully solves the im2latex problem, which is confirmed by the results of measuring metrics. Generated captions for images with equations are quite accurate and, in most cases, coincide with the real ones. Such a solution can be used in mathematical programs to automatically translate images into LaTeX and further solve and analyze the resulting equations or expressions. The scope of use can be expanded by training the network on images with handwritten equations. A similar technology is used in the PhotoMath application, however, it has low accuracy and a small set of supported mathematical symbols. The described solution is devoid of both problems.