AWS Glue OCF Connector: Install and Configure

Authentication

Authentication from Alation to AWS Glue requires a user account. Create an AWS IAM user account for Alation and save the values of the access key, access key secret, AWS region, and user ARN.

You can choose to authenticate directly with the IAM user account (basic authentication) or to use STS authentication through an AWS role (STS authentication).

Using Basic Authentication

Grant the user account you created for Alation the required permissions by creating and attaching these policies:

  • Policy for the Glue service

    • "glue:GetDatabase"

    • "glue:GetDatabases"

    • "glue:GetTable"

    • "glue:GetTableVersions"

    • "glue:GetTables"

    • "glue:GetConnection"

    • "glue:GetConnections"

    • "glue:GetJob"

    • "glue:GetJobs"

  • Policy for the S3 service

    • "s3:ListBucket"

    • "s3:ListAllMyBuckets"

    • "s3:GetBucketAcl"

Using STS Authentication

You can choose to authenticate with an STS token through an AWS IAM role. To set up STS authentication:

  1. In the AWS IAM service, create IAM policies for the S3 and Glue services as described in Using Basic Authentication.

  2. Create an IAM role selecting the Type of Trusted Entity to be AWS Service and Use Case to be EC2. To this role, attach the policies that grant the required permissions. This role will be assumed by the service account when performing MDE from your AWS Glue data source.

  3. Edit the Trust Relationship of this role and add the ARN of the IAM user account that you created for Alation as Principal.

    ../../../_images/OCF_AWSGlue_InstallConfig_STS.png

  4. Save the ARN of this role. It is required for the next configuration steps in Alation.

  5. See Using Region-Specific Endpoint next.

Using Region-Specific Endpoint

From connector version 1.0.4, you can use the region-specific STS endpoint or the global endpoint.

Using the global STS endpoint does not require any specific configuration.

To use the region-specific endpoint, make sure it is active under your AWS account. To check this:

  1. Under AWS IAM, go to Access Management > Account settings.

  2. Under the Security Token Service (STS) section, in the Endpoints table, make sure that the STS endpoint for your AWS region is active or activate it.

Configuration in Alation

STEP 1: Install the Connector

Alation On-Prem

Important

Installation of OCF connectors requires Alation Connector Manager to be installed as a prerequisite.

To install an OCF connector:

  1. If this has not been done on your instance, install the Alation Connector Manager: Install Alation Connector Manager.

  2. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  3. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

Alation Cloud Service

Note

On Alation Service Cloud instances, Alation Connector Manager is available by default.

  1. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  2. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

STEP 2: Create and Configure a New Data Source

In Alation, add a new data source:

  1. Log in to Alation as a Server Admin.

  2. Expand the Apps menu on the right of the main toolbar and select Sources.

  3. On the Sources page, click +Add on the top right of the page and in the list that opens, click Data Source. This will open the Add a Data Source wizard.

  4. On the first screen of the wizard, specify a name for your data source, assign additional Data Source Admins, if necessary, and click the Continue Setup button on the bottom. The Add a Data Source screen will open.

  5. On the Add a Data Source screen, the only field you should populate is Database Type. From the Database Type dropdown, select the connector name. After that you will be navigated to the Settings page of your new data source.

The name of this connector is AWS Glue OCF Connector.

Access

On the Access tab, set the data source visibility using these options:

  • Public Data Source—The data source will be visible to all users of the catalog.

  • Private Data Source—The data source will be visible to the users allowed access to the data source by Data Source Admins.

You can add new Data Source Admin users in the Data Source Admins section.

General Settings

Application Settings

Skip this section as it’s not applicable to AWS Glue data sources.

Connector Settings

Configure AWS Connection

Populate the data source connection information and save the values by clicking Save in this section.

Parameter

Description

AWS Region

Specify your AWS region.

Basic Authentication

Select this radio button to configure authentication with the IAM user.

STS Authentication

Select this radio button to configure STS authentication through an IAM role.

Basic Authentication

Specify the authentication details in this section if you have selected the Basic Authentication radio button under Configure AWS Connection and click Save to save the information you entered.

Parameter

Description

AWS Access Key ID

Specify the access key ID of the service account.

AWS Access Key Secret

Specify the access key secret of the service account.

STS Authentication

Specify the authentication details in this section if you have selected the STS Authentication radio button under Configure AWS Connection and click Save to save the information you entered.

Parameter

Description

Region-Specific Endpoint

Select this checkbox if you prefer to use the STS endpoint specific to your AWS region. The regional endpoint has to be active under your AWS account.

Leave this checkbox clear to use the global endpoint.

STS: AWS Access Key ID

Specify the access key ID of the service account.

STS: AWS Access Key Secret

Specify the access key secret of the service account.

Role ARN

Specify the ARN of the role you created following the steps in STS Authentication.

STS Duration

Specify the STS duration value, in seconds. The default value is 3600 seconds.

Logging Configuration

Select the logging level for the connector logs and save the values by clicking Save in this section. The available log levels are based on the Log4j framework.

Parameter

Description

Log level

Select the log level to generate logs. The available options are INFO, DEBUG, WARN, TRACE, ERROR, FATAL, ALL.

Obfuscate Literals

Skip this section as it’s not applicable to AWS Glue data sources.

Test Connection

Under Test Connection, click Test to validate network connectivity.

Metadata Extraction

You can configure metadata extraction (MDE) for an OCF data source on the Metadata Extraction tab of the Settings page. For AWS Glue data sources, Alation supports full and selective default MDE. Custom query-based MDE is not supported.

Refer to Configure Metadata Extraction for OCF Data Sources for information about the available configuration options.

Sampling and Profiling

Not supported.

Query Log Ingestion

Not supported.

Compose

Not supported.