Within the Modern Data Stack, there are a variety of tools that move and transform data as it goes from source to destination. 1'īefore we dive into the details of what these fields mean in OpenLineage, let's take a look at some of the questions we may wish to answer with lineage data.
LINEAGE W BEST CLASS SERIES
What this ends up looking in practice is a series of JSON-formatted events, like so: [ , The Github repo comes with two clients in both Java and Python but the standard is simple enough that writing a client in any language should be fairly straightforward. The standard is formalized as a JsonSchema which makes it easy to work with and validate. The standard defines some key concepts, such as a Job, Run, and Datasets while allowing for flexibility for each provider to add additional metadata using facets. The goal is to have a unified schema for describing metadata and data lineage across tools to make data lineage collection and analysis easier. It is supported with contributions from major projects such as pandas, Spark, dbt, Airflow, and Great Expectations. OpenLineage is an open standard for metadata and lineage collection. Even more mundane tasks like identifying which teams are consuming data from a field that is about to be deprecated for regulatory reasons can be an impossible task without a good understanding of data lineage.
LINEAGE W BEST CLASS CODE
This immediately becomes apparent when a bug is found somewhere downstream and analysts need to search backwards through code across tools to identify why something broke. The problem is that there are a multitude of systems and understanding the graph of the movement of data across systems is much more difficult. Many data tools already have some concept of data lineage built in, whether it's Airflow's DAGs or dbt's graph of models, the lineage of data within a system is well understood. What is Data Lineage?ĭata Lineage describes the flow of data to and from various systems that ingest, transform and load it. OpenLineage is an emerging standard that helps to bridge the gap across these various tools when it comes to metadata. However, while the modern data stack is highly interoperable, there is still a lack of cohesion when it comes to metadata and lineage data. We are also seeing tooling expose valuable metadata that can help trace data lineage, dependencies, and pipeline health. We've seen companies like Monte Carlo Data and Datakin emerge to help address some of these issues, with a focus on increasing data observability. With the ever-expanding ecosystem around data analytics, we've started to see an increase in interest around metadata and data lineage but it's not always clear what data lineage is and why it is useful.
0 kommentar(er)
