In this talk, We will share the work and experiences towards alleviating the management overhead of large-scale Kafka clusters using Cruise Control at LinkedIn. Cruise Control is a system that aims to provide effortless management of large-scale Kafka clusters. To achieve this goal, Cruise Control (1) provides real-time monitoring for Kafka clusters, (2) supports admin operations for cluster maintenance, (3) enables automatic anomaly detection with self-healing.
The talk will consist of two parts: The first part will provide an overview of Cruise Control, including the operational challenges that it solves, its high-level architecture and main functionality it provides out-of-box, some evaluation results from real-world scenarios. The second part will go through a demo to show how we can effortlessly deploy a Cruise control and use it to manage a real Kafka cluster.