AWS Glue OCF Connector: Install and Configure

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

Authentication

The OCF connector for AWS Glue supports several authentication methods:

Basic Authentication

Basic authentication requires an AWS IAM user and the access key ID and secret access key for this user.

To use basic authentication, create an AWS IAM user account for Alation and save the values of the access key ID, secret access key, AWS region, and user ARN.

Grant the IAM user the required permissions (see Permissions for IAM User Account below).

Permissions for IAM User Account

Grant the user account you created for Alation the required permissions by creating and attaching these policies:

  • Policy for the Glue service

    • "glue:GetDatabase"

    • "glue:GetDatabases"

    • "glue:GetTable"

    • "glue:GetTableVersions"

    • "glue:GetTables"

    • "glue:GetConnection"

    • "glue:GetConnections"

    • "glue:GetJob"

    • "glue:GetJobs"

  • Policy for the S3 service

    • "s3:ListBucket"

    • "s3:ListAllMyBuckets"

    • "s3:GetBucketAcl"

STS Authentication with an AWS IAM User

STS Authentication with an IAM user requires an IAM user account and an AWS role.

To use this type of authentication:

  1. Create an IAM user and assign it the required permissions as it’s described in Basic Authentication.

  2. Perform the configuration described in Create an IAM Role for STS Authentication below.

Create an IAM Role for STS Authentication

To set up STS authentication with an IAM user:

  1. In the AWS IAM service, create IAM policies for the S3 and Glue services as described in Basic Authentication.

  2. Create an IAM role selecting the Type of Trusted Entity to be AWS Service and Use Case to be EC2. To this role, attach the policies that grant the required permissions. This role will be assumed by the service account when performing MDE from your AWS Glue data source.

  3. Edit the Trust Relationship of this role and add the ARN of the IAM user account that you created for Alation as Principal.

    ../../../_images/OCF_AWSGlue_InstallConfig_STS.png
  4. Save the ARN of this role. It is required for the next configuration steps in Alation.

  5. See Using Region-Specific Endpoint next.

Using Region-Specific Endpoint

From connector version 1.0.4, you can use the region-specific STS endpoint or the global endpoint.

Using the global STS endpoint does not require any specific configuration.

To use the region-specific endpoint, make sure it is active under your AWS account. To check this:

  1. Under AWS IAM, go to Access Management > Account settings.

  2. Under the Security Token Service (STS) section, in the Endpoints table, make sure that the STS endpoint for your AWS region is active or activate it.

STS Authentication with an AWS IAM Role

STS authentication with an AWS IAM role does not require an IAM user. This authentication method uses an instance profile that assumes a role allowing access to Amazon resources. This authentication method works for authenticating across AWS accounts.

Note

This authentication method is available with connector version 1.1.1.6409 or newer.

To configure STS authentication with an AWS IAM role, use the steps in Configure Authentication via AWS STS and an IAM Role. To provide access to the data source via an IAM role, use the permissions information in Permissions for IAM User Account.

Configuration in Alation

STEP 1: Install the Connector

Alation On-Prem

Important

Installation of OCF connectors requires Alation Connector Manager to be installed as a prerequisite.

To install an OCF connector:

  1. If this has not been done on your instance, install the Alation Connector Manager: Install Alation Connector Manager.

  2. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  3. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

Alation Cloud Service

Note

On Alation Service Cloud instances, Alation Connector Manager is available by default.

  1. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  2. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

STEP 2: Create and Configure a New Data Source

In Alation, add a new data source:

  1. Log in to Alation as a Server Admin.

  2. Expand the Apps menu on the right of the main toolbar and select Sources.

  3. On the Sources page, click +Add on the top right of the page and in the list that opens, click Data Source. This will open the Add a Data Source wizard.

  4. On the first screen of the wizard, specify a name for your data source, assign additional Data Source Admins, if necessary, and click the Continue Setup button on the bottom. The Add a Data Source screen will open.

  5. On the Add a Data Source screen, the only field you should populate is Database Type. From the Database Type dropdown, select the connector name. After that you will be navigated to the Settings page of your new data source.

    Note

    Agent-based connectors will have the Agent name appended to the connector name.

The name of this connector is AWS Glue OCF Connector.

Access

On the Access tab, set the data source visibility using these options:

  • Public Data Source—The data source will be visible to all users of the catalog.

  • Private Data Source—The data source will be visible to the users allowed access to the data source by Data Source Admins.

You can add new Data Source Admin users in the Data Source Admins section.

General Settings

Note

This section describes configuring settings for credentials and connection information stored in the Alation database. If your organization has configured Azure KeyVault or AWS Secrets Manager to hold such information, the user interface for the General Settings page will change to include the following icons to the right of most options:

../../../_images/VaultOrDB.png

By default, the database icon is selected, as shown. In the vault case, instead of the actual credential information, you enter the ID of the secret. See Configure Secrets for OCF Connector Settings for details.

Application Settings

Skip this section as it’s not applicable to AWS Glue data sources.

Connector Settings

Under the Connector Settings section of the General Settings tab, populate the data source connection information and save the values by clicking Save.

Parameter

Description

AWS Region

Specify your AWS region of the account under which your AWS Glue resource is located.

Basic Authentication

Default. Leave this radio button selected if you are going to configure basic authentication.

STS Authentication

Select this radio button to configure STS authentication with an IAM user.

Important

If you are going to Configure STS Authentication with an AWS IAM Role, disregard the Basic Authentication and STS Authentication radio buttons. They will not apply. Only provide a valid JDBC URI.

Configure Basic Authentication

If you selected the Basic Authentication radio button, specify the information in the Basic Authentication section of General Settings. Save the values by clicking Save.

Refer to Basic Authentication for more information about this authentication method.

Parameter

Description

AWS Access Key ID

Specify the access key ID of the service account.

AWS Access Key Secret

Specify the access key secret of the service account.

Configure STS Authentication

If you selected the STS Authentication radio button, specify the information in the STS Authentication section of General Settings. Save the values by clicking Save.

Refer to STS Authentication with an IAM User for more information about this authentication method.

Parameter

Description

Region-Specific Endpoint

Select this checkbox if you prefer to use the STS endpoint specific to your AWS region. The regional endpoint has to be active under your AWS account.

Leave this checkbox clear to use the global endpoint.

STS: AWS Access Key ID

Specify the access key ID of the service account.

STS: AWS Access Key Secret

Specify the access key secret of the service account.

Role ARN

Specify the ARN of the role you created following the steps in Create an IAM Role for STS Authentication.

STS Duration

Specify the STS duration value, in seconds. The default value is 3600 seconds.

Configure STS Authentication with an AWS IAM Role

To use STS authentication with an AWS IAM role, specify the information in the IAM Role Authentication section of General Settings. Save the values by clicking Save.

Parameter

Description

Auth Type

Select AWS IAM.

Authentication Profile

Select the authentication profile you created in Admin Settings.

Role ARN

Provide the ARN of the role that gives access to the Amazon resource.

External ID

Provide the External ID you added to the role that gives access to the Amazon resource.

STS Duration

Provide the STS token duration in seconds. This value must be less than or equal to the Maximum session duration of the IAM role that provides access to the Amazon resource(s).

Logging Configuration

Select the logging level for the connector logs and save the values by clicking Save in this section. The available log levels are based on the Log4j framework.

Parameter

Description

Log level

Select the log level to generate logs. The available options are INFO, DEBUG, WARN, TRACE, ERROR, FATAL, ALL.

Obfuscate Literals

Skip this section as it’s not applicable to AWS Glue data sources.

Test Connection

Under Test Connection, click Test to validate network connectivity.

Metadata Extraction

You can configure metadata extraction (MDE) for an OCF data source on the Metadata Extraction tab of the Settings page. For AWS Glue data sources, Alation supports full and selective default MDE. Custom query-based MDE is not supported.

Refer to Configure Metadata Extraction for OCF Data Sources for information about the available configuration options.

Sampling and Profiling

Not supported.

Query Log Ingestion

Not supported.

Compose

Not supported.