Databricks Unity Catalog OCF Connector: Overview

The OCF Connector for Databricks Unity Catalog was developed by Alation and is available out of the box. The connector is compatible with release 2022.4.x or later releases.

Create a ticket with Alation Support about receiving the Databricks Unity Catalog OCF connector package.

The connector should be used to catalog Databricks Unity Catalog metadata in Alation. It supports both cluster and SQL endpoints for metadata extraction and catalogs the three-level namespace (Catalog, Schema, Table) and objects from multiple workspaces using a single data source connection.

The connector supports Databricks on AWS and Databricks Azure.

Team

  • Databricks account administrator:

    • Performs metastore configuration (not required if already done)
      • Creates a Metastore

      • Assigns the metastore to a workspace that will be used for metadata extraction

    • Creates a service account for Alation

    • Creates a compute resource for metadata extraction
    • Provides the JDBC URI to access metadata.

  • Alation Server Admin:

    • Installs the connector

    • Creates and configures a Databricks Unity Catalog OCF data source in Alation.

Scope

The table below shows which metadata objects are extracted by this connector and which operations are supported.

Feature

Scope

Availability

Authentication

Token-based

Authentication with a personal access token

Yes

Metadata Extraction (MDE)

Default MDE

Extraction of metadata based on default extraction queries in the connector code

Yes

Custom query-based MDE

Extraction of metadata based on extraction queries provided by a user

No

Extracted metadata objects

Data Source

Data source object in Alation that is parent to extracted metadata

Yes

Schemas

List of schemas

Yes

Tables

List of tables

Yes

Columns

List of columns

Yes

Column data types

Column data types

Yes

Views

List of views

Yes

Source comments

Source comments

Yes

Primary keys

Primary key information for extracted tables

No

Foreign keys

Foreign key information for extracted tables

No

Functions

Extraction of function metadata

No

Sampling and Profiling

Yes

Query Log Ingestion (QLI)

No

Lineage

No*

Compose

Yes

Data upload

Yes

* As QLI is not supported, the lineage data auto-generated from ingesting query history is not available. However, lineage can be created manually or added using the API. Lineage is also available from DDL queries run in Compose.