Learning dynamics in linear recurrent neural networks

Alexandra Proca, Clémentine Dominé, Murray Shanahan, Pedro Mediano

International Conference on Machine Learning 2025 · Oral

This talk, presented by Alexandra Proca at ICML 2025, delves into the intricate mechanisms of learning within **linear recurrent neural networks (RNNs)**. The research addresses a critical gap in the understanding of how RNNs, vital for tasks involving temporal dependencies in both machine learning and neuroscience, acquire their functional structures during training. While much prior work has focused on analyzing the properties of trained RNNs, this presentation shifts the focus to the dynamic learning process itself, exploring how network parameters evolve when exposed to temporally structured data.

AI review

Proca et al. extend the Saxe-style deep linear network program to the recurrent setting, deriving analytically tractable gradient flow equations for linear RNNs and using them to characterize recency bias, extrapolation failure, stability conditions, computational phase transitions, and a bias toward rich feature learning via NTK analysis. This is a genuine theoretical contribution — not just an empirical characterization — and it fills a real gap: the deep linear network framework has been extraordinarily productive for feedforward architectures, and its extension to recurrent computation…