Using inherent structures to design lean 2-layer RBMs

Abstract

Understanding the representational power of Restricted Boltzmann Machines (RBMs) with mul-tiple layers is an ill-understood problem and ' is an area of active research. Motivated from the approach of Inherent Structure formalism (Stillinger & Weber, 1982), extensively used in analysing Spin Glasses, we propose a novel measure called Inherent Structure Capacity (ISC), which characterizes the representation capacity of a fixed architecture RBM by the expected number of modes of distributions emanating from the RBM with parameters drawn from a prior distribution. Though ISC is intractable, we show that for a single layer RBM architecture ISC approaches a finite constant as number ; of hidden units are increased and to further improve the ISC, one needs to add a second layer. Furthermore, we introduce Lean RBMs, which are multi-layer RBMs where each layer can have1 at-most 0(n) units with the number of visible units being n. We show that for every single layer RBM with ft(n2+r),r > 0, hidden units there exists a two-layered lean RBM with 0(n2) parameters with the same ISC, establishing that 2 layer RBMs can achieve the same representational power as single-layer RBMs but using far fewer number of parameters. To the best of our knowledge, this is the first result which quantitatively establishes the need for layering.