Abstract: A central phenomenon in modern machine learning is feature learning: rather than operating on a fixed representation of the data, successful models learn representations adapted to the prediction task. We study a tractable model of this phenomenon through a compositional variant of kernel ridge regression, where the kernel is applied after a learnable linear transformation. When the response depends on the input only through a low-dimensional predictive subspace, we show that optimizing the population objective automatically eliminates directions orthogonal to this subspace and, in certain regimes, exactly recovers it. Surprisingly, the same exact low-dimensional structure persists at finite sample sizes with high probability, even without explicitly penalizing the linear transformation to be low-dimensional.

Bio: Feng Ruan is currently an assistant professor in Department of Statistics and Data Science at Northwestern University. His research focuses on the foundations of feature learning and stochastic and nonsmooth optimization, and is supported by an NSF CAREER Award.