Basic experimentation systems typically start with implementing t-tests or z-tests. But full experimentation systems comprise design, randomization and traffic allocation, targeting, logging, data processing, and statistical analysis.

In this talk we will explore two topics: First we will review some pitfalls in randomization and traffic allocation that can introduce bias in your results, and how to avoid them in your experimentation system. Then we will show how a good traffic allocation algorithm can make implementing multi-armed bandits much easier. This talk will be aimed at a general audience with basic knowledge of experimentation.