Quine: A Streaming Graph for Event-Driven Data Pipelines

Abstract: 

This talk will introduce Quine: a brand new open source "steaming graph interpreter" meant as a new fundamental infrastructure component to address major challenges in data engineering and simplify enterprise data pipelines.

Quine fits in between the world of databases and stream processing systems. As data streams in from Kafka, Kinesis, etc., Quine builds it into a graph. Then using "standing queries"—queries that live inside the graph and efficiently propagate—it finds matches to complex patterns in the graph and streams the results out right away. Quine maintains a stateful representation of all data streamed through (like a database) so that complex results are built from the combination of new streaming data and potentially very old data—all without having to manage any time windows. Since the graph is fully versioned, you can always query for what the data used to be, at any historical moment. Quine is meant to be a complete package of everything that lives between two Kafka topics: high-volume events stream in, and highly-meaningful interpreted results stream out.

In this talk, we will explain how Quine works under the hood, discuss some of the interesting and brain-bending challenges we had to confront in order to create it, and show some use cases to illustrate why it's important for modern data pipelines. Quine implements a property-graph data model on top of an asynchronous graph computational model. It's like Pregel with Actors. Each node is capable of performing arbitrary computation, so we can bake in some powerful capabilities deep in the graph; and then package it up for easy use into user-contributed "recipes" available in the Github repo. Quine is free and open to all, available at https://quine.io and actively supported by thatDot and the community.

Session Outline
This talk will introduce Quine: a new open source "steaming graph interpreter" meant as a new fundamental infrastructure component to address major challenges in data engineering and simplify enterprise event-driven data pipelines.

Bio: 

Ryan Wright is the creator of Quine, and has been leading software teams focused on data infrastructure and data science for two decades. He has served as principal engineer, director of engineering, principal investigator on DARPA-funded research programs, and is currently the founder and CEO of thatDot—the company supporting Quine. Ryan particularly enjoys taking the philosophical ends of computer science—usually problems related to language, meaning, and data—and making them more practical.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google