Why Deep Networks Don’t Need to Memorize Everything — Matthieu Wyart

Introduction and Physics Lens

• Matthieu Wyart’s Background and Approach:
  • Full professor at John Hopkins University and EPFL, specializing in statistical physics, focusing on collective phenomena (e.g., freezing, stock markets, complex systems with “rough energy landscapes” like sand).
  • His fascination with AlphaGo led him to machine learning, where he views ML training as a complex system.
  • He observed analogies between sand’s “jamming transition” and the “double descent” phenomenon in ML loss landscapes, where under-parameterized systems have rough landscapes and over-parameterized systems exhibit smoother, flow-like behavior.
  • Wyart’s core interest is understanding why deep networks discover abstractions, predicting latents versus tokens, and answering these questions through a unified physics framework, particularly by analyzing the “landscape of data.”

• Universality of Physics Concepts:
  • Wyart argues that physics concepts like phase transitions, coarse-graining, and universality are applicable to ML, even though LLMs lack a material substrate.
  • He draws analogies (e.g., Huygens’ wave theory of light) to illustrate how scientific progress often involves applying universal concepts across diverse phenomena.
  • Constraints in ML are viewed as the algorithm’s objective (e.g., parameters collectively fitting data), which creates a

• Tags:
• Self-Supervised Learning
• Generative Models
• Feature
• Scaling Laws
• Machine Learning
• Deep Learning
• Diffusion Models
• JEPA
• Sample Efficiency
• Abstractions
• Statistical Physics
• Hidden Hierarchies
• Curse of Dimensionality
• Latent Space Prediction
• Token Prediction
• Chomsky
• Context-Free Grammars
• Rough Loss Landscapes
• Double Descent