Running online experiments can be slow and expensive. Picking a few worth running experiments from a large set of candidates relies heavily on subjective choices and intuition. In some situations, we sequentially iterate through numerous experimental candidates and wait for weeks to get results back. In others, a carefully tuned model with AUC improvement disappoints with a negative experiment result.

We developed an offline experimentation framework, which allows us to select optimal candidates in a matter of hours instead of days required for online experiments. It can forecast the output of key metrics offline through a replay approach on past data. This framework enables us to quickly iterate product design and provide better user experience.

The talk will cover the methodology, empirical results and use cases of the framework.