AWS Glue OCF Connector: Overview

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

The OCF connector for AWS Glue was developed by Alation and is available as a Zip file that can be uploaded and installed in the Alation application.

To download the AWS Glue OCF connector package, go to the Alation Connector Hub available from the Customer Portal. Go to Customer Portal > Connectors > Alation Connector Hub. Only Alation users with access to the Customer Portal can access the Alation Connector Hub. If you don’t have access to the Customer Portal, contact Alation Support.

This connector should be used to catalog AWS Glue as a data source on Alation on-prem and Cloud Service instances. It uses the AWS Glue API to connect to your AWS Glue instance and extract metadata. The connector is compiled with all necessary libraries to make API calls to AWS. No separate code or drivers are required. After the metadata is extracted, it is represented in the data catalog as a hierarchy of catalog pages under the parent data source. Alation users can leverage the full catalog functionality to search for and find the metadata, curate the corresponding catalog pages, enrich the catalog with documentation about the data source, and exchange information with other catalog users.

Team

You may need the assistance of your AWS admin to configure this connector.

  • AWS administrator:

    • Creates a service account for Alation

    • Creates the required policy

    • Provides the connection information.

  • Alation administrator:

    • Installs the connector

    • Creates and configures an AWS Glue data source in the catalog.

Scope

The table below describes which metadata objects are extracted by the connector and which catalog functionality is supported.

Feature

Scope

Availability

Authentication

AWS IAM user

Authentication with an IAM user access key and secret

Yes

AWS IAM role

Authentication with an STS token for an AWS IAM role

Yes*

* Authentication with the AWS IAM role is supported from connector version 1.0.4.

Metadata Extraction (MDE)

Default MDE

Extraction of metadata based on default extraction queries in the connector code

Yes

Custom query-based MDE

Extraction of metadata based on extraction queries provided by a user

No

Popularity

Indicator of the popularity (intensity of use) of a data object, such as a table or a column

No

Extracted metadata objects

Data Source

Data source object in Alation that is parent to extracted metadata

Yes

Schemas

List of AWS Glue databases

Yes

Tables

List of AWS Glue tables

Yes

Columns

List of columns

Yes

Column data types

Column data types

Yes

Views

List of views

Yes

Source comments

Source comments

Yes

Primary keys

Primary key information for extracted tables

No

Foreign keys

Foreign key information for extracted tables

No

Sampling and Profiling

No

Query Log Ingestion (QLI)

No

Lineage

No

Compose

No

Object Mapping

AWS Glue Object

Alation Object

AWS Glue database

Schema

AWS Glue table

Table

AWS Glue column

Column