DBT Add-On OCF Connector: Install and Configure

Prerequisites

Generate a Manifest File

Before you can configure the DBT add-on connector, make sure that the DBT admin generates a manifest.json file using the dbt run or dbt docs commands. For more details on the manifest file, refer to Manifest. Place the manifest.json file in a location supported by Alation — AWS S3 or GitHub.

Note

There is 90 minutes latency time to update views ACCOUNT_USAGE views. So the new descriptions created will be extracted during metadata extraction after the latency time. Refer to Account Usage — Snowflake Documentation for information about the latency time for each item.

Create an AWS S3 User

Note

This section applies if you store the manifest.json files on AWS S3.

Create a user in AWS IAM with read-only access to the folder(s) that store the manifest file to set up a connection in Alation and perform the metadata extraction (MDE). Make sure to save the values of the access key and secret key.

Generate a Personal Access Token on GitHub

Note

This section is applicable, if the manifest.json files are stored on GitHub.

For the steps to create a token, refer to Create a Personal Access Token in GitHub documentation.

URI

AWS S3 URI

Format
<hostname>/buckets/<bucket-name>?region=<region_name>

Example:

https://s3.console.aws.amazon.com/s3/buckets/Project?region=us-east-1

GitHub URI

To get the GitHub URI:

  1. Go to the repository that stores the manifest.json file and click the manifest.json file.

  2. Click the Raw button.

../../../_images/DBTOCF_01.png
  1. Copy the part of the URL that inlcudes the repository name as shown below. The copied URL can be used as GitHub URI in Alation.

    Example:

    https://raw.githubusercontent.com/Project1984-2019/repo/
    

Installation

Install the DBT Add-On OCF connector

Alation On-Prem

Important

Installation of OCF connectors requires Alation Connector Manager to be installed as a prerequisite.

To install an OCF connector:

  1. If this has not been done on your instance, install the Alation Connector Manager: Install Alation Connector Manager.

  2. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  3. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

Alation Cloud Service

Connecting to Cloud Databases

Note

On Alation Cloud Service instances, Alation Connector Manager is available by default.

To install an OCF connector:

  1. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  2. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

Connecting to On-Prem Databases

Important

To connect to an on-prem database from an Alation Cloud Service instance, you may need to use Alation Agent.

  1. Ensure that the Alation Agent functionality is enabled on your Alation instance. Create a Support ticket with Alation for an Alation representative to enable the Alation Agent feature on your instance and to receive the Alation Agent installer.

  2. Install Alation Agent using the information in Alation Agent.

  3. Install the OCF connector on Alation Agent.

Note

Install the DBT add-on connector on the same Alation Agent on which the primary connector is installed.

Configuration

Enable the DBT Connector

To use the DBT add-on connector:

  1. Go to Admin Settings > Server Admin > Feature Configuration.

  2. Enable the Enable DBT Connector for OCF RDBMS sources (Paid Feature) flag. By default, this flag is disabled.

Enabling this feature will add the DBT connector configuration fields to the Settings page of the supported primary connectors (Snowflake, Redshift, PostgreSQL).

General Settings

After you have enabled the feature flag of the DBT connector, you can configure extraction from DBT on the General Settings tab of the primary connector:

  1. In Add-On Connector Settings, enable the Configure DBT OCF connector toggle. Once you enable the toggle, the DBT configuration settings fields will appear.

  2. Choose the DBT Configuration Settings (DBT Cloud or DBT Core) and click Save.

  3. If you selected DBT Cloud, specify the DBT configuration settings for DBT Cloud.

    Parameter

    Description

    Cloud URL

    Specify the DBT Cloud URL.

    Project(s) name

    Specify the projects name separated by comma.

    Example: project1, project2

    Access token

    Specify the DBT Cloud access token of the account to download the manifest.json file.

  4. If you selected DBT core, Specify the DBT configuration settings for DBT Core.

    Parameter

    Description

    Data source

    Select the primary data source from the dropdown list. The available datasource options are:

    • Snowflake

    • Redshift

    • PostgreSQL

    Repository type

    Select a repository type from the dropdown list. Supported repository types are:

    • GitHub

    • Amazon S3

    GitHub

    GitHub URL

    Specify the GitHub URI in the required format.

    Manifest file(s) location

    Specify the manifest.json file locations separated by comma. Example: project1, project 2

    Access Token

    Specify the GitHub personal access token.

    AWS S3

    Region

    Specify the AWS S3 region. Example: us-east-1

    Bucket Name

    Specify the AWS S3 bucket name.

    Manifest file(s) location

    Specify the manifest.json file locations separated by comma. Example: project1, project 2

    Access Key

    Provide the AWS access key ID of the IAM user. Make sure that the IAM user has access to the destination bucket.

    Secret key

    Specify the secret key.

  1. Click Save.

    ../../../_images/DBTOCF_02.png

Metadata Extraction

The metadata extraction from DBT will run as a child job of the primary connector’s MDE. To make sure that the extraction from DBT is successful:

  1. In the Extract Job Status section, click View Details.

    ../../../_images/DBTOCF_03.png
  2. The successful extraction message will be displayed as shown below.

    ../../../_images/DBTOCF_04.png

The descriptions extracted from DBT will be cataloged as Source Comments as shown below.

../../../_images/DBTOCF_05.png

Lineage

Note

Lineage is available from DBT Add-On OCF connector version 2.0.0 onwards and Alation version 2022.4.3 and later.

The Lineage information will be generated from the Jinja code extracted into the Dataflow content.

../../../_images/DBTOCF_06.png

Troubleshooting

Refer to Troubleshooting.