IDEAL Theory of Deep Learning Special Quarter Seminar

Title: The Blind Men and the Elephant: The Mysteries of Deep Learning

Speaker: Babak Hassibi, Mose and Lillian S. Bohn Professor of Electrical Engineering and Computing and Mathematical Sciences, California Institute of Technology

Time: Thursday October 1, 2020, 11:30-12:30

Venue: See https://www.ideal.northwestern.edu/special-quarters/fall-2020/ for streaming link.

Abstract: Deep learning has demonstrably enjoyed a great deal of recent practical success and is arguably the main driver behind the resurgent interest in machine learning and AI. Despite its tremendous empirical achievements, we are far from a theoretical understanding of deep networks. In this talk, we will argue that the success of deep learning is not only due to the special deep architecture of the models, but also due to the behavior of the stochastic descent methods used, which play a key role in reaching "good" solutions that generalize well to unseen data. We will connect learning algorithms such as stochastic gradient descent (SGD) and stochastic mirror descent (SMD) to work in H-infinity control in the 1990's, and thereby explain the convergence and implicit-regularization behavior of the algorithms when we are highly over-parametrized (what is now being called the "interpolating regime"). This gives us insight into why deep networks exhibit such powerful generalization abilities, a phenomenon now being referred to as "the blessing of dimensionality".

Biography: Babak Hassibi was born in Tehran, Iran, in 1967. He received the B.S. in electrical engineering degree from the University of Tehran, Tehran, Iran, in 1989, and the M.S. and Ph.D. degrees in electrical engineering from Stanford University, Stanford, CA, USA, in 1993 and 1996, respectively. Since January 2001, he has been with the California Institute of Technology, where he is currently the Mose and Lilian S. Bohn Professor of electrical engineering. From 2008 to 2015, he was Executive Officer of electrical engineering, as well as Associate Director of information science and technology. From November 1998 to December 2000, he was a member of the Technical Staff with the Mathematical Sciences Research Center, Bell Laboratories, Murray Hill, NJ, USA. He has also held short-term appointments with the Ricoh California Research Center, the Indian Institute of Science, and Linkoping University, Sweden. His research interests include communications and information theory, control and network science, and signal processing and machine learning. He is the co-author of the books (both with A.H. Sayed and T. Kailath) Indefinite Quadratic Estimation and Control: A Unified Approach to $H^2$ and $H^\infty$ Theories (SIAM, 1999) and Linear Estimation (Prentice-Hall, 2000). He is a recipient of the Alborz Foundation Fellowship, 1999 O. Hugo Schuck Best Paper Award of the American Automatic Control Council (with H. Hindi and S. P. Boyd), 2002 National Science Foundation Career Award, 2002 Okawa Foundation Research Grant for Information and Telecommunications, 2003 David and Lucille Packard Fellowship for Science and Engineering, 2003 Presidential Early Career Award for Scientists and Engineers (PECASE), and 2009 Al-Marai Award for Innovative Research in Communications, and was a participant in the 2004 National Academy of Engineering "Frontiers in Engineering" program.