Running online experiments can be slow and expensive. Picking a few worth running experiments from a large set of candidates relies heavily on subjective choices and intuition. In some situations, we sequentially iterate through numerous experimental candidates and wait for weeks to get results back. In others, a carefully tuned model with AUC improvement disappoints with a negative experiment result.
We developed an offline experimentation framework, which allows us to select optimal candidates in a matter of hours instead of days required for online experiments. It can forecast the output of key metrics offline through a replay approach on past data. This framework enables us to quickly iterate product design and provide better user experience.
The talk will cover the methodology, empirical results and use cases of the framework.
Maxine Qian(Pinterest)
Maxine Qian is a Senior Data Scientist at Pinterest, leading the offline experimentation efforts in the Experimentation and Metrics Science team. She also worked on online experimentation quality and ads measurement solutions. Her work focuses on developing new methodologies and improving velocity and quality in experimentation.
Dec 03, 10:00 AM PST 
