How LinkedIn Navigates Streams Infrastructure Using Cruise Control
How LinkedIn Navigates Streams Infrastructure Using Cruise Control

Abstract: 

Kafka has been widely adopted as the data streaming platform backbone of the leading companies. However, this rising adoption introduced new challenges. In particular, the growing cluster sizes, increasing volume and diversity of user traffic, and the age of network and server components led to a management overhead. Getting near-optimal performance from such an infrastructure service, maintaining its availability in the face of cascading failures, and achieving these objectives with minimal overhead are critical, but non-trivial tasks. Hence, human intervention alone tends to be insufficient in providing both reactive and proactive mitigation measures.

In this talk, we will share our work and experiences towards alleviating the management overhead of large-scale Kafka clusters using Cruise Control at LinkedIn. The talk will consist of three parts: The first part will provide an overview of Cruise Control, including the operational challenges that it solves, its high-level architecture, and some evaluation results from real-world scenarios. The second part will go through a tutorial to demonstrate how we can manage a real Kafka cluster using Cruise Control. Finally, we will also have time for a Q&A session at the end of the discussion.

Bio: 

Adem Efe Gencer is a Senior Software Engineer at LinkedIn. He received his PhD in Computer Science from Cornell University in 2017. His PhD research has focused on improving the scalability of blockchain technologies. The protocols introduced in his research were adopted by many companies, and are actively being developed into other systems. His papers received over 700 citations and a best student paper award in Middleware Conference. Efe actively acts as a reviewer for top-tier journals and conferences.

Efe develops Apache Kafka and the ecosystem around it, and supports their operation at LinkedIn. In particular, he works on the design, development, and maintenance of Cruise Control — a system for alleviating the management overhead of large-scale Kafka clusters at LinkedIn.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google