Font Size:

Learning Unitary Operators with Help From u(n)

Last modified: 2017-02-13

#### Abstract

A major challenge in the training of recurrent neural networks is the so-called vanishing or exploding gradient problem. The use of a norm-preserving transition operator can address this issue, but parametrization is challenging. In this work we focus on unitary operators and describe a parametrization using the Lie algebra u(

*n*) associated with the Lie group*U*(*n*) of*n*×*n*unitary matrices. The exponential map provides a correspondence between these spaces, and allows us to define a unitary matrix using*n*^{2}real coefficients relative to a basis of the Lie algebra. The parametrization is closed under additive updates of these coefficients, and thus provides a simple space in which to do gradient descent. We demonstrate the effectiveness of this parametrization on the problem of learning arbitrary unitary operators, comparing to several baselines and outperforming a recently-proposed lower-dimensional parametrization. We additionally use our parametrization to generalize a recently-proposed unitary recurrent neural network to arbitrary unitary matrices, using it to solve standard long-memory tasks.#### Keywords

recurrent neural network; lie algebra; lie group; deep learning

Full Text:
PDF