LOCOCODE - Low-Complexity Coding and Decoding

Authors: Sepp Hochreiter (TUM) and Juergen Schmidhuber (IDSIA).

Abstract: Low-complexity coding and decoding (LOCOCODE) is a novel approach to sensory coding and unsupervised learning. Unlike previous methods it explicitly takes into account the information-theoretic complexity of the code generator: lococodes (1) convey information about the input data and (2) can be computed and decoded by low-complexity mappings. We implement LOCOCODE by training autoassociators with Flat Minimum Search, a recent, general method for discovering low-complexity neural nets. Experiments show: unlike codes obtained with standard autoenco- ders, lococodes are based on feature detectors, never unstructured, usually sparse, sometimes factorial or local (depending on the data). Although LOCOCODE's objective function does not contain an explicit term enforcing sparse or factorial codes, it extracts optimal codes for difficult versions of the "bars" benchmark problem. Unlike, e.g., independent component analysis (ICA) it does not need to know the num- ber of independent data sources. It produces familiar, biologically plausible feature detectors when applied to real world images. As a preprocessor for a vowel recognition benchmark problem it sets the stage for excellent classification performance.

Based on:

• S. Hochreiter and J. Schmidhuber. Flat Minima. Neural Computation, 9(1):1-42, 1997, (201 K).

Recent conference publications on LOCOCODE:

• S. Hochreiter and J. Schmidhuber. Unsupervised coding with LOCOCODE. In W. Gerstner, A. Germond, M. Hasler, J.-D. Nicoud, eds., Proceedings of the International Conference on Artificial Neural Networks, Lausanne, Switzerland, Springer, 655-660, 1997.
• S. Hochreiter and J. Schmidhuber. Low-complexity coding and decoding. In Proc. of Theoretical Aspects of Neural Computation (TANC 97), Springer, 1997.