Troubleshooting

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

Issues with Metadata Extraction

Problem: MDE fails with error “iterating schema children”

The issue may occur with older connector versions that use the legacy Spark driver. This driver sometimes cannot keep up the connection long enough for MDE to complete.

Solution

This issue was resolved starting with connector version 2.0.2.6259, where the driver was changed to the Databricks JDBC driver. See more in Build the JDBC URI.

Problem: Fetch Schemas fails with error “unable to instantiate HiveMetastoreClient”

The issue may occur when Hive metastore is not instantiated. Hive Metastore is the default Databricks connection catalog.

Solution

Include the ConnCatalog and ConnSchema parameters into the connection JDBC URI:
  • ConnCatalog - Specifies the catalog which contains the metadata schema.

  • ConnSchema - Specifies the schema inside the catalog in which metadata is stored.

Example:

ConnCatalog=en_dlake_cat;ConnSchema=data_governance;

Issues with Lineage Extraction

Problem: After upgrading the connector to version 1.0.3, MDE fails at ingestion

Connector logs will show the error “system.access.table_lineage table does not exist”.

Solution

Direct lineage extraction from system tables is a beta feature. The lineage system tables are not enabled by default and need to be enabled by your Databricks admin. See more in Lineage Extraction (Beta).

Problem: No lineage is generated after MDE

Solution

Verify that lineage data is available in the system lineage tables in Databricks.

Issues with QLI

Problem: After running QLI, some queries don’t appear in the History section of the Queries tab

Explanation

Some SQL statements are currently not supported by the Databricks SQL parser that is used to parse SQL statements during QLI, for example:

  • Partially qualified names in SQL statements are not parsed. Object names in query history must be fully qualified to be ingested.

  • The USE statements are not parsed.

  • Queries containing CTEs are not parsed.

  • Queries containing comments (COMMENT, COMMENT ON) are not parsed

This is a limitation with the current GSP and a known issue.

Problem: After QLI, parts of a query appear as separate queries in the catalog

Explanation

To support proper query parsing, accurate execution times count, and lineage resolution, batched queries are split into individual query statements before ingestion.

Problem: Queries that appeared in Preview don’t appear in the catalog after QLI

Explanation

The Preview is not directly tied to queries that will appear in the catalog. It returns 100 statements which may be a subset of all imported queries. For queries to appear in the catalog after QLI:

  • Queries must contain fully qualified object names with three-level namespace (catalog.schema.table).

  • The objects referenced in queries must have been cataloged with MDE.

  • Queries must not duplicate the queries that were ingested previously. Duplicated queries are not resolved.