SpHMC: Spectral Hamiltonian Monte Carlo

  • Haoyi Xiong Baidu Inc. & National Engineering Laboratory for Deep Learning Technology and Applications
  • Kafeng Wang Chinese Academy of Sciences & University of Chinese Academy of Sciences
  • Jiang Bian University of Central Florida
  • Zhanxing Zhu Peking University & Beijing Institute of Big Data Research
  • Cheng-Zhong Xu Chinese Academy of Sciences
  • Zhishan Guo University of Central Florida
  • Jun Huan Baidu Inc. & National Engineering Laboratory for Deep Learning Technology and Applications

Abstract

Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) methods have been widely used to sample from certain probability distributions, incorporating (kernel) density derivatives and/or given datasets. Instead of exploring new samples from kernel spaces, this piece of work proposed a novel SGHMC sampler, namely Spectral Hamiltonian Monte Carlo (SpHMC), that produces the high dimensional sparse representations of given datasets through sparse sensing and SGHMC. Inspired by compressed sensing, we assume all given samples are low-dimensional measurements of certain high-dimensional sparse vectors, while a continuous probability distribution exists in such high-dimensional space. Specifically, given a dictionary for sparse coding, SpHMC first derives a novel likelihood evaluator of the probability distribution from the loss function of LASSO, then samples from the high-dimensional distribution using stochastic Langevin dynamics with derivatives of the logarithm likelihood and Metropolis–Hastings sampling. In addition, new samples in low-dimensional measuring spaces can be regenerated using the sampled high-dimensional vectors and the dictionary. Extensive experiments have been conducted to evaluate the proposed algorithm using real-world datasets. The performance comparisons on three real-world applications demonstrate the superior performance of SpHMC beyond baseline methods.

Published
2019-07-17
Section
AAAI Technical Track: Machine Learning