Lineage V3 Overview

Applies from version 2021.4

Lineage V3, or the lineage service, is a microservice operating inside the Alation server. It is responsible for the creation, storage, and retrieval of lineage data into the Catalog.

Lineage V3 is required for the Manual Lineage Curation feature to work.

The Alation server creates lineage data from multiple sources, such as metadata extraction (MDE), query log ingestion (QLI), Compose query history, and public APIs. With Lineage V3 enabled, lineage events generated from these sources are sent to the Lineage V3 service via Event Bus. In the lineage service:

  • the lineage write service consumes lineage events from the Event Bus and stores this lineage data into the lineage database;

  • the lineage read service retrieves the stored lineage data and powers the lineage diagrams in the Alation user interface.

../../_images/lineageV3_01.png

Lineage V3 is disabled by default. It can be enabled by migrating the lineage data from Lineage V2 to Lineage V3. Migration from Lineage V2 to Lineage V3 is required if an existing Alation instance already has lineage data. On new installations of Alation where no lineage data exists yet, Lineage V3 can be enabled using alation_conf.

Lineage V2 and Lineage V3

Lineage V2 and Lineage V3 are both frameworks for processing lineage data and representing it in the Alation Catalog. V2 and V3 use different application architectures. V2 is the older framework that was introduced in Alation version VR6 (5.10.x) and made default in version 2021.2. Lineage V3 is available from version 2021.4.

Both V2 and V3 support lineage diagrams, Dataflow objects, Lineage Impact Analysis reports, and Lineage V2 API. A big differentiator is the ability to create lineage data manually that is only available with Lineage V3.

Lineage V3 aims at addressing the three main challenges of the Lineage V2 framework:

  • Scalability: Lineage V3 is capable of performing ingestion of millions of lineage events and is expected to be much more performant.

  • Flexibility: Lineage V3 can potentially be extended to new object types and serves as a foundation for manual creation of lineage.

  • Cloud Readiness: Lineage V3 has potential to be leveraged as a component in the containerized application architecture, which is the basis for Alation cloud deployments.