EVENT DETAILS
Raghu Pasupathy
Purdue University
Title: Adaptive Sequential SAA as a Universal Paradigm for Stochastic Optimization
Abstract: The Stochastic Approximation (SA) recursion, also known as Stochastic Gradient Descent (SGD), is the ``workhorse" recursion for solving stochastic optimization problems arising in machine learning and elsewhere. For smooth, strongly convex objectives, it is well-known that SA iterates achieve the information theoretic Cramer-Rao (CR) lower bound when the step size is chosen correctly. Such correct choice, however, depends on unknown curvature constants; moreover, the CR bound as a measure of performance ignores transient evolution and is thus not fully reflective of finite-time performance. This explains why SGD is rarely implemented in its native form. Instead, heuristic variations, e.g., ADAM, have come to be widely preferred (see, for example, TensorFlow). In this talk, I will argue for an alternative to SGD and its variants. Specifically, I will re-introduce a decades-old idea and rigorize it in a way that allows well-established deterministic optimization methods, e.g., line search and trust-region with L-BFGS, to be used ``as is." The power of the framework is that it naturally retains the structure that is often inherent to sample-paths, and is expressly equipped to exploit deterministic optimization methods. I will demonstrate that the proposed framework retains the CR lower bound to within a factor in the convex smooth case, and an analogous theorem for the smooth non-convex case. An extension to the non-smooth convex context using the proximal method as a solver will be evident. I will also introduce two solvers, RA-LLBFGS and RA-TLBFGS, as specific variations of the framework.
Biography: Raghu Pasupathy has been Associate Professor in the Department of Statistics at Purdue University since 2014. His recent work has focused on the theoretical and computational aspects of stochastic optimization, statistical modeling, and Monte Carlo. His teaching interests include probability and stochastic processes, statistics (especially modeling), and optimization. Raghu Pasupathy currently serves as the President of the INFORMS SImulation Society.
TIME Tuesday November 5, 2019 at 11:00 AM - 12:00 PM
LOCATION M228, Technological Institute map it
ADD TO CALENDAR&group= echo $value['group_name']; ?>&location= echo htmlentities($value['location']); ?>&pipurl= echo $value['ppurl']; ?>" class="button_outlook_export">
CONTACT Agnes Kaminski a-kaminski@northwestern.edu
CALENDAR Department of Industrial Engineering and Management Sciences