DBT Add-On OCF Connector: Install and Configure¶
Prerequisites¶
Generate a Manifest File¶
Before you can configure the DBT add-on connector, make sure that the DBT admin generates a manifest.json file using the dbt run or dbt docs commands. For more details on the manifest file, refer to Manifest. Place the manifest.json file in a location supported by Alation — AWS S3 or GitHub.
Note
There is 90 minutes latency time to update views ACCOUNT_USAGE views. So the new descriptions created will be extracted during metadata extraction after the latency time. Refer to Account Usage — Snowflake Documentation for information about the latency time for each item.
Create an AWS S3 User¶
Note
This section applies if you store the manifest.json files on AWS S3.
Create a user in AWS IAM with read-only access to the folder(s) that store the manifest file to set up a connection in Alation and perform the metadata extraction (MDE). Make sure to save the values of the access key and secret key.
Generate a Personal Access Token on GitHub¶
Note
This section is applicable, if the manifest.json files are stored on GitHub.
For the steps to create a token, refer to Create a Personal Access Token in GitHub documentation.
URI¶
AWS S3 URI¶
Format¶
<hostname>/buckets/<bucket-name>?region=<region_name>
Example:
https://s3.console.aws.amazon.com/s3/buckets/Project?region=us-east-1
GitHub URI¶
To get the GitHub URI:
Go to the repository that stores the manifest.json file and click the manifest.json file.
Click the Raw button.
Copy the part of the URL that inlcudes the repository name as shown below. The copied URL can be used as GitHub URI in Alation.
Example:
https://raw.githubusercontent.com/Project1984-2019/repo/
Installation¶
Install the DBT Add-On OCF connector¶
Alation On-Prem¶
Important
Installation of OCF connectors requires Alation Connector Manager to be installed as a prerequisite.
To install an OCF connector:
If this has not been done on your instance, install the Alation Connector Manager: Install Alation Connector Manager.
Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.
Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.
Alation Cloud Service¶
Connecting to Cloud Databases¶
Note
On Alation Cloud Service instances, Alation Connector Manager is available by default.
To install an OCF connector:
Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.
Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.
Connecting to On-Prem Databases¶
Important
To connect to an on-prem database from an Alation Cloud Service instance, you may need to use Alation Agent.
Ensure that the Alation Agent functionality is enabled on your Alation instance. Create a Support ticket with Alation for an Alation representative to enable the Alation Agent feature on your instance and to receive the Alation Agent installer.
Install Alation Agent using the information in Alation Agent.
Install the OCF connector on Alation Agent.
Note
Install the DBT add-on connector on the same Alation Agent on which the primary connector is installed.
Configuration¶
Enable the DBT Connector¶
To use the DBT add-on connector:
Go to Admin Settings > Server Admin > Feature Configuration.
Enable the Enable DBT Connector for OCF RDBMS sources (Paid Feature) flag. By default, this flag is disabled.
Enabling this feature will add the DBT connector configuration fields to the Settings page of the supported primary connectors (Snowflake, Redshift, PostgreSQL).
General Settings¶
After you have enabled the feature flag of the DBT connector, you can configure extraction from DBT on the General Settings tab of the primary connector:
In Add-On Connector Settings, enable the Configure DBT OCF connector toggle. Once you enable the toggle, the DBT configuration settings fields will appear.
Choose the DBT Configuration Settings (DBT Cloud or DBT Core) and click Save.
If you selected DBT Cloud, specify the DBT configuration settings for DBT Cloud.
Parameter
Description
Cloud URL
Specify the DBT Cloud URL.
Project(s) name
Specify the projects name separated by comma.
Example: project1, project2
Access token
Specify the DBT Cloud access token of the account to download the manifest.json file.
If you selected DBT core, Specify the DBT configuration settings for DBT Core.
Parameter
Description
Data source
Select the primary data source from the dropdown list. The available datasource options are:
Snowflake
Redshift
PostgreSQL
Repository type
Select a repository type from the dropdown list. Supported repository types are:
GitHub
Amazon S3
GitHub
GitHub URL
Specify the GitHub URI in the required format.
Manifest file(s) location
Specify the manifest.json file locations separated by comma. Example: project1, project 2
Access Token
Specify the GitHub personal access token.
AWS S3
Region
Specify the AWS S3 region. Example: us-east-1
Bucket Name
Specify the AWS S3 bucket name.
Manifest file(s) location
Specify the manifest.json file locations separated by comma. Example: project1, project 2
Access Key
Provide the AWS access key ID of the IAM user. Make sure that the IAM user has access to the destination bucket.
Secret key
Specify the secret key.
Click Save.
Metadata Extraction¶
The metadata extraction from DBT will run as a child job of the primary connector’s MDE. To make sure that the extraction from DBT is successful:
In the Extract Job Status section, click View Details.
The successful extraction message will be displayed as shown below.
The descriptions extracted from DBT will be cataloged as Source Comments as shown below.
Lineage¶
Note
Lineage is available from DBT Add-On OCF connector version 2.0.0 onwards and Alation version 2022.4.3 and later.
The Lineage information will be generated from the Jinja code extracted into the Dataflow content.
Troubleshooting¶
Refer to Troubleshooting.