Big Data Platform at Lyft and Workday

Jul 14, 12:00PM PDT(07:00PM GMT).
  • Free 183 Attendees
This event is hosted by San Francisco Big Analytics meetup group

Agenda (US Pacific Time):
12:00 - 12:05 pm: Introduction
12:05 - 12:40 pm: Tech Talk 1 by Lyft
12:40 - 01:15 pm Tech Talk 2 by Workday
1:30 pm - event closed

Talk #1: Can Apache Kafka Handle a Lyft Ride?
What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with. In this talk, we will be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We will need to take care of your consumers as well to ensure that they dont process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters. We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.

Talk #2: Optimize Cloud Services through Analyzing Operational Data on A Big Data Platform
To provide a high customer satisfaction, Workday leverages the operational data (both structured and unstructured data) to optimize services. At Workday, a centralized big data platform is used for descriptive, diagnostic and predictive analytics for all personas. This talk will introduce the challenges that the platform needs to address. Afterwards, we will present the tools used for various analysis tasks. Moreover, an in-house data pipeline platform will be presented, which is used for automation.


Andrey Falko
Staff Software Engineer at Lyft, where he has been for more than a year. He is currently focused on building and scaling reliable PubSub systems for Lyfts Data Platform. Prior to Lyft, Andrey worked at Salesforce for nine years where he researched Kafka and Pulsar performance and reliability.
Lei Gao
Senior machine learning engineer at Workday, who is leading a team to build a data science analytics platform. .
The event ended.
Watch Recording
*Recordings hosted on Youtube, click the link will open the Youtube page.