Databricks Unity Catalog OCF Connector: Overview

The OCF Connector for Databricks Unity Catalog was developed by Alation and is available out of the box. The connector is compatible with release 2022.4.x or later releases.

Create a ticket with Alation Support about receiving the Databricks Unity Catalog OCF connector package.

The connector should be used to catalog Databricks Unity Catalog metadata in Alation. It supports both cluster and SQL endpoints for metadata extraction and catalogs the three-level namespace (Catalog, Schema, Table) and objects from multiple workspaces using a single data source connection.

The connector supports Databricks on AWS and Databricks Azure.

Team

• Databricks account administrator:

  • Performs metastore configuration (not required if already done)

    • Creates a Metastore

    • Assigns the metastore to a workspace that will be used for metadata extraction

  • Creates a service account for Alation

  • Creates a compute resource for metadata extraction

    • Creates a compute resource (cluster or SQL warehouse) with single user or user isolation as described in Create a cluster or SQL warehouse

  • Provides the JDBC URI to access metadata.

• Alation Server Admin:

  • Installs the connector

  • Creates and configures a Databricks Unity Catalog OCF data source in Alation.

Scope

The table below shows which metadata objects are extracted by this connector and which operations are supported.

Feature	Scope	Availability
Authentication
Token-based	Authentication with a personal access token	Yes
Metadata Extraction (MDE)
Default MDE	Extraction of metadata based on default extraction queries in the connector code	Yes
Custom query-based MDE	Extraction of metadata based on extraction queries provided by a user	No
Extracted metadata objects
Data Source	Data source object in Alation that is parent to extracted metadata	Yes
Schemas	List of schemas	Yes
Tables	List of tables	Yes
Columns	List of columns	Yes
Column data types	Column data types	Yes
Views	List of views	Yes
Source comments	Source comments	Yes
Primary keys	Primary key information for extracted tables	No
Foreign keys	Foreign key information for extracted tables	No
Functions	Extraction of function metadata	No
Sampling and Profiling		Yes
Query Log Ingestion (QLI)		No
Lineage		No*
Compose		Yes
Data upload		Yes

* As QLI is not supported, the lineage data auto-generated from ingesting query history is not available. However, lineage can be created manually or added using the API. Lineage is also available from DDL queries run in Compose.