AWS DynamoDB¶

Supported as Custom DB from version 2020.3

Scope of Support¶

Supported as Custom DB data source type with a CData DynamoDB Driver 2020. Please contact your AWS account manager to find out if this driver is included into your AWS solution.
Basic authentication with the AWS Access Key and Secret Key
Both local and cloud deployments are supported
SSL is supported using the URL parameter in the URI; no SSL certificate is required to be added to Alation.
MDE
- Regular (automatic)
Sampling and Profiling
- Regular (automatic)
Compose
- Alation supports SSO connection to DynamoDB for end-users: Compose SSO for AWS Data Sources
  
  Note
  
  Compose SSO authentication does not apply to the service account used for MDE, automatic Profiling, and QLI.

Limitations¶

The DynamoDB URI is not accepted by Alation in the Add Data Source wizard, and a workaround has to be used to move through the wizard screens.
The Connection Status on the Settings > General Settings page of the data source displays an error and the “No connection” state, which can be disregarded.
Table Properties (Table Type, Owner, Partitions, Bucket keys, etc) are not captured by the Catalog after MDE
QLI is not supported. Only the query data from Compose is ingested. Lineage and Popularity data are based on Compose queries only.
“Explain” in Compose is not supported.
If new tables are added to DynamoDB after the data source is added to Alation, these new tables are not immediately accessible for querying from Compose. They become available only after MDE is rerun in the Catalog.
Role-Based Authentication is not supported

AWS Billing¶

Billable by Amazon on your AWS account:

The queries Alation runs on the connected Dynamo DB instance during MDE and Profiling
The queries users run in Alation Compose

Required Information¶

JDBC driver used to connect to the database: CData DynamoDB Driver 2020
JDBC URI for the DynamoDB data source. See Construct the URI.
An IAM user with the required set of permissions and the Access Key and Secret Key to authenticate on the DynamoDB.
Alation server-side access to place the custom driver on the Alation system.

Construct the URI¶

The JDBC URI must be entered without the “JDBC” tag at the beginning. The parameters to be included:

Access Key=<your_access_key>

Secret Key=<your_secret_key>

SupportsCatalogsInTableDefinitions=True

SupportsSchemasInTableDefinitions=True

AWSRegion=<your_AWS_region> - may be required for MDE to work with your DynamoDB instance

AuthScheme=AwsIamRoles - may be required for MDE to work with your DynamoDB instance

RTK - Use the RTK parameter only if you have the RTK provided by Alation.

Minimal set of required parameters¶

amazondynamodb:URL=<DynamoDB_URL>;Access Key=<access_key>;Secret Key=<secret_key>;SupportsCatalogsInTableDefinitions=True;SupportsSchemasInTableDefinitions=True;RTK=<RTK_Code>

Example for a cloud instance:

amazondynamodb:URL=https://dynamodb.us-east-1.amazonaws.com;Access Key=AKIATTRQEDGQE2OHM7RVFD;Secret Key=StokWo0dn79vPXBg7GxPu1X0SqUwx23f3d0J3HiX;SupportsCatalogsInTableDefinitions=True;SupportsSchemasInTableDefinitions=True;RTK=444752465641535552425641454E545042424D333236323900000000000000000000000000000000414C4154494F4E5800005559475655474E4E464242370000

Example for a local instance:

amazondynamodb:URL=http://10.13.47.194:8000;Access Key=AKIATTRQEDGQE2OHM72D;Secret Key=StokWo0dn79vPXBg7GxPu1X0SqUwxKbN5a0J3Hax;SupportsCatalogsInTableDefinitions=True;SupportsSchemasInTableDefinitions=True;RTK=444752465641535552425641454E545042424D333236323900000000000000000000000000000000414C4154494F4E5800005559475655474E4E464242370000

Additional parameters¶

If MDE does not work with the minimal set of required parameters in the URI, you may need to additionally include the following parameters:

AWSRegion=<your_AWS_region>

AuthScheme=AwsIamRoles

amazondynamodb:URL=https://<dynamodb.eu-east-1.amazonaws.com>;Access Key=<access_key>;Secret Key=<secret_key>;SupportsCatalogsInTableDefinitions=True;SupportsSchemasInTableDefinitions=True;AuthScheme=AwsIamRoles;AWSRegion=<your_AWS_region>;RTK=<RTK_Code>

If you have included the Tables parameter into the URI and the table names inlucde dashes, use the following format to list the tables: Tables=[table-1-name],[table-2-name].

amazondynamodb:URL=https://<dynamodb.eu-east-1.amazonaws.com>;Access Key=<access_key>;Secret Key=<secret_key>;SupportsCatalogsInTableDefinitions=True;SupportsSchemasInTableDefinitions=True;AuthScheme=AwsIamRoles;AWSRegion=<your_AWS_region>;Tables=[table-1-name],[table-2-name];RTK=<RTK_Code>

For the other URI parameters that support specific cases, please refer to the CData Driver documentation: Connection.

Authentication¶

Use an Existing IAM User Account¶

You can use an existing DynamoDB account as the Service Account for Alation.

Note that the billing for queries run by Alation during MDE and Profiling and the billing for queries run by this account in AWS DynamoDB UI and in Compose will be combined;
The existing account must be assigned the managed policies required by Alation.

Create New IAM User Account¶

Create a new IAM user account that can operate the DynamoDB

Important

Take note of the AWS Access Key and Secret Key. If you lose the Secret Key you will have to create another account.

Permissions for Metadata Extraction¶

The IAM user needs to be able to access the tables and metadata to complete the extraction process.

Permissions for Profiling/Sampling¶

AWS DynamoDB operation permissions are required for IAM user.

Add the CData Driver to Alation¶

Refer to Add the CData Driver into Alation Instance.

Steps in Alation¶

To add a DynamoDB data source to the Catalog,

On the Sources page, add a new data source to Alation. Provide a title and proceed to the Add Data Source wizard.
On the Add a Data Source screen of the wizard, specify:
- Database Type: select Custom DB
- JDBC URI: enter any valid JDBC URI - but NOT the URI of the DynamoDB
  
  Important
  
  The JDBC URI for Dynamo DB is not accepted by the Add Data Source wizard even if it is a valid URI. To move through the wizard, please enter any valid JDBC URI, for example: http://10.13.23.33:8080
  
  This allows you to move through the wizard.
  
  You will add the JDBC URI for Dynamo DB on the Settings > General Settings page after completing the steps of the wizard.
- Select Driver: select the CData driver for DynamoDB that you have added to Alation:
cdata.jdbc.amazondynamodb.AmazonDynamoDBDriver.cdata.jdbc.amazondynamodb
- Privacy: set Public or Private
Click Save and Continue. You will get an error message for the JDBC URI you entered.

Click Continue with Errors in order to move to the next screen.
On the next screen - Set Up a Service Account - select Yes, and in the Username field, type User (or any other string). Then click Save and Continue. As the Access Key and the Secret Key will need to be included into the URI, there is no need to provide this information on this screen.
On the next screen - Configure Data Source - click Skip This Step.

After this step, you are navigated to the Settings page of the new data source.

Configure the Dynamo DB Data Source¶

Complete the configuration on the Settings page and perform MDE and Profiling.

Access¶

Verify Privacy settings. Add Data Source Admins.

General Settings¶

Under Network Connection, edit the JDBC URI: click the Edit icon next to the URI and in a dialog that opens, provide the correct URI for DynamoDB. See Construct the URI.

Important

After you provide the correct URI, both the Network Connection status and the Service Account status will still be in red. This is a known limitation. Disregard this error and proceed to Metadata Extraction.

Metadata Extraction¶

Configure and perform MDE.

Sampling and Profiling¶

See Sampling and Profiling.

Troubleshooting¶

Logs to collect/review:

For logs related to MDE: taskserver.log, taskserver_err.log
For logs related to Compose: connector.log, connector_err.log
For any other errors: alation-error.log, alation-debug.log