Ó³»­´«Ã½

Jon Bloom, Cotton Seed
Hail Team, Ó³»­´«Ã½
From Morse theory to geometric ensembling via the topology of PCA

Abstract: We'll start with a visual introduction to Morse theory, which relates the topology (shape) of a manifold (space) to the behavior of smooth, real-valued functions on that manifold. We'll then apply this relationship in both directions. First, we’ll consider the function on the space of k-planes in R^m given by squared distance to a fixed point cloud, leading to a visceral understanding of the gradient dynamics of PCA as learned by a linear autoencoder. Second, we’ll consider the loss function of a deep neural network. We’ll explain how the Morse homology of Euclidean space forces geometric relationships between critical points, establishing a theoretical foundation for fast geometric ensembling that in turn suggests new algorithms. Paper and visualization .

Aleksandrina Goeva, Daniel Kunin
Macosko Lab, Ó³»­´«Ã½; Stanford
Regularized linear autoencoders, probabilistic PCA, and backpropagation in the brain

Abstract: Autoencoders are a deep learning model for representation learning. When trained to minimize the Euclidean distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. Here we prove that L2-regularized LAEs learn the principal directions as the left singular vectors of the decoder, providing an extremely simple and scalable algorithm for rank-k SVD. More generally, we consider LAEs with (i) no regularization, (ii) regularization of the composition of the encoder and decoder, and (iii) regularization of the encoder and decoder separately. We relate the minimum of (iii) to the MAP estimate of probabilistic PCA and show that for all critical points the encoder and decoder are transposes. Building on the topological intuition of the primer, we smoothly parameterize the critical manifolds for all three losses via a novel unified framework and illustrate these results empirically. Overall, this work clarifies the relationship between autoencoders and Bayesian models and between regularization and orthogonality. Most excitingly, it suggests a simple, biologically-plausible, and testable resolution of the "weight symmetry problem," namely a local mechanism by which maximizing information flow and minimizing energy expenditure gives rise to backpropagation as the optimization algorithm underlying efficient neural coding. Paper and visualization .