Title: Non-stationary A/B Tests

Abstract: A/B tests, also known as online randomized controlled experiments, have been used at scale by data-driven enterprises to guide decisions and test innovative ideas. Non-stationarities, such as the time-of-day effect, the day-of-week effect, and longer-term trends, can often arise nonparametrically in core business metrics. We discuss in this presentation the impact of non-stationarities on A/B tests. We argue that inadequately addressing non-stationarities can lead to wrong decisions and missed opportunities. We discuss some scenarios where the challenges brought by non-stationarities may be appropriately addressed.

Bio: Zeyu Zheng is an assistant professor at UC Berkeley, Department of Industrial Engineering and Operations Research since 2018. He received a PhD in Management Science and Engineering (2018), PhD minor in Statistics (2018), and MS in Economics (2016) from Stanford University, and BS in Mathematics (2012) from Peking University. He serves as an Associate Editor for Operations Research, Probability in the Engineering and Informational Sciences. His research group currently works on simulation, experiment design and inference, generative AI, stochastic optimization and non-stationary modeling.