Databricks Unity Catalog OCF Connector: Overview¶
The OCF Connector for Databricks Unity Catalog was developed by Alation and is available out of the box. The connector is compatible with release 2022.4.x or later releases.
Create a ticket with Alation Support about receiving the Databricks Unity Catalog OCF connector package.
The connector should be used to catalog Databricks Unity Catalog metadata in Alation. It supports both cluster and SQL endpoints for metadata extraction and catalogs the three-level namespace (Catalog, Schema, Table) and objects from multiple workspaces using a single data source connection.
The connector supports Databricks on AWS and Databricks Azure.
Team¶
Databricks account administrator:
- Performs metastore configuration (not required if already done)
Assigns the metastore to a workspace that will be used for metadata extraction
Creates a service account for Alation
- Creates a compute resource for metadata extraction
Creates a compute resource (cluster or SQL warehouse) with single user or user isolation as described in Create a cluster or SQL warehouse
Provides the JDBC URI to access metadata.
Alation Server Admin:
Installs the connector
Creates and configures a Databricks Unity Catalog OCF data source in Alation.
Scope¶
The table below shows which metadata objects are extracted by this connector and which operations are supported.
Feature |
Scope |
Availability |
---|---|---|
Authentication |
||
Token-based |
Authentication with a personal access token |
Yes |
Metadata Extraction (MDE) |
||
Default MDE |
Extraction of metadata based on default extraction queries in the connector code |
Yes |
Custom query-based MDE |
Extraction of metadata based on extraction queries provided by a user |
No |
Extracted metadata objects |
||
Data Source |
Data source object in Alation that is parent to extracted metadata |
Yes |
Schemas |
List of schemas |
Yes |
Tables |
List of tables |
Yes |
Columns |
List of columns |
Yes |
Column data types |
Column data types |
Yes |
Views |
List of views |
Yes |
Source comments |
Source comments |
Yes |
Primary keys |
Primary key information for extracted tables |
No |
Foreign keys |
Foreign key information for extracted tables |
No |
Functions |
Extraction of function metadata |
No |
Sampling and Profiling |
Yes |
|
Query Log Ingestion (QLI) |
No |
|
Lineage |
No* |
|
Compose |
Yes |
|
Data upload |
Yes |
* As QLI is not supported, the lineage data auto-generated from ingesting query history is not available. However, lineage can be created manually or added using the API. Lineage is also available from DDL queries run in Compose.