On the Learning of Energy-Based Models Using Noise Contrastive Estimation

Abstract

Energy-Based Models (EBMs) are a family of unsupervised machine learning models that associate each point in the input space with an energy value in which low energy indicates a high likelihood. Specifically, an EBM can be viewed as an unnormalized probabilistic model, which upon normalization, gives rise to the probability density function of data. The main difficulty in learning EBMs lies in the computation of the normalization constant, or the partition function, a task known to be intractable in general. Several approaches have been proposed to avoid or overcome this difficulty, including Maximum Likelihood Estimation (MLE) with Markov chain Monte Carlo (MCMC), Score Matching (SM), Noise Contrastive Estimation (NCE), and so on.

This thesis studies the learning of EBMs using NCE. Briefly, in NCE, the EBM learning problem is converted to learning a binary classifier, which aims to distinguish the real data from fake data drawn from a noise distribution. This process allows the learning of the energy function in the EBM to bypass a direct estimation of the partition function and a certain theoretical guarantee is available under some assumptions in some asymptotic limit.

Despite the nice theoretical properties of NCE, in this work, we show that learning EBMs using NCE entails significant practical limitations. Specifically, there appears a tension between the quality of the learned model and the computational efficiency, due to which we must sacrifice one to achieve the other. We establish these limitations via empirical studies as well as a theoretical analysis based on a simple “Gaussian data learning problem”. Our analysis inspires a revised NCE scheme, Adaptive Noise Contrastive Estimation (ANCE), to overcome these limitations. Empirically, we show that ANCE achieves a better quality-efficiency trade-off than the standard NCE in some regimes.