Tuesday / CS Seminar

May 26 / 1:00 PM

Hybrid / Mudd 3514

Speaker

Amil Dravid, UC Berkeley

Talk Title

Deep Learning as a Natural Science: From Neurons to Scaling Laws

Abstract

"As neural networks grow larger, they increasingly resemble complex systems with systematic behavior rather than isolated engineered artifacts. This is reflected in neural scaling laws, which show that performance follows a precise power law as a function of model size, data, and compute. Yet, we still lack a principled scientific understanding of how these models organize information internally.

In this talk, I will describe an approach to studying neural networks as scientific objects, drawing on ideas from neuroscience and statistical physics. I will focus on neurons that respond to similar inputs across independently trained models of the same modality -- for example, neurons that fire for similar words across language models or similar objects and textures across vision models. We find that the population of shared neurons grows as a power law with model size, but represents a shrinking fraction of the overall population. We account for this trend with a simple theoretical model, which also predicts a polarization effect: as models grow, shared neurons become increasingly selective for specific concepts, such as code or visual textures, while the remaining neurons become less selective and encode multiple features. We then test this prediction in large-scale language and vision models. I will conclude with a case study showing how neuron selectivity can guide targeted data selection to improve training. Together, these results suggest that scaling laws are not restricted to external model behavior; they also describe regularities in how neural networks are organized internally."

Biography

Amil Dravid is a third-year PhD student in computer science at UC Berkeley, advised by Alyosha Efros. His research develops tools for understanding the internal structure of deep learning systems, with the goal of informing the design of more effective algorithms and models. He previously earned his BS in Computer Science from Northwestern University. His research is supported by the U.S. Department of Energy Computational Science Graduate Fellowship.

Research Areas: Deep learning, computer vision, natural language processing

---

Zoom Link

Panopto Link