OCF Connector for Impala on CDP: Install and Configure

Ports

  • Open outbound TCP port 21050 to the Impala server

  • Open outbound TCP port 9083 to the metastore

  • Open outbound TCP port 9870 to the HDFS server

  • Open outbound TCP port 9871 to the HDFS server if the cluster uses TLS.

Hive Configuration

Ensure that in the Hive service configuration, the property metastore.storage.schema.reader.impl is set to rg.apache.hadoop.hive.metastore.SerDeStorageSchemaReader.

To check the value or set it:

  1. Open the Cloudera Manager homepage.

  2. Click on Hive.

  3. Click the Configuration tab.

  4. On the Configuration tab, search for Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml and click the plus icon on Hive Metastore Server Default Group.

    ../../../_images/OCF_CDP_Impala_HiveConfig.png

  5. In the Name field, enter the name metastore.storage.schema.reader.impl.

  6. In the Value field, enter the value org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader.

  7. Click Save Changes.

    ../../../_images/OCF_CDP_Impala_HiveConfigSave.png

  8. A restart icon will pop up after a few seconds. Click on it to restart the service.

  9. In the window that opens, click Restart Stale Service.

  10. Select Re-deploy client configuration and then click Restart Now.

  11. Wait for the service to restart and then click on the Continue button that will appear after the restart.

Extraction of Complex Data Types

The Impala on CDP OCF connector supports extraction of complex data types, such as map, array, and struct. To enable their representation in the Alation user interface as a tree structure, on your Alation instance, set the alation_conf parameter alation.feature_flags.enable_generic_nosql_support to True. Additionally, you can use the parameter alation.feature_flags.docstore_tree_table_depth to define the depth of the display (default is three levels).

For details about using alation_conf, refer to Using alation_conf.

Important

After changing values of these parameters, restart Alation Supervisor from the Alation shell: alation_supervisor restart all.

Service Account

Prior to adding your Impala data source to the catalog, create a service account for Alation.

The service account requires these permissions:

  • SELECT permission for all schemas which you plan to extract into the catalog

  • READ permission for the HDFS location that stores Impala query logs

  • READ, WRITE, and EXECUTE permissions for the Impala external table

  • Access to the metastore.

In case your Impala data source is kerberized, we recommend creating the service account in the same realm as the realm used by Impala users.

Kerberos Configuration File

In case your Impala data source is kerberized, prepare the krb5.conf file and, if applicable, the keytab file for the service account. They will need to be uploaded to Alation.

Connection over SSL

If your connection from Alation will go over SSL, prepare the corresponding SSL certificate. Alation expects the file cm-auto-global_cacerts.pem that can usually be found in the directory /var/lib/cloudera-scm-agent/agent-cert. Consult your CDP admin as to locating and downloading the certificate. It will need to be uploaded to Alation in the settings of the Impala data source.

Configuration in Alation

STEP 1: Install the Connector

Alation On-Prem

Important

Installation of OCF connectors requires Alation Connector Manager to be installed as a prerequisite.

To install an OCF connector:

  1. If this has not been done on your instance, install the Alation Connector Manager: Install Alation Connector Manager.

  2. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  3. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

Alation Cloud Service

Connecting to Cloud Databases

Note

On Alation Cloud Service instances, Alation Connector Manager is available by default.

To install an OCF connector:

  1. Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.

  2. Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.

Connecting to On-Prem Databases

Important

To connect to an on-prem database from an Alation Cloud Service instance, you may need to use Alation Agent.

  1. Ensure that the Alation Agent functionality is enabled on your Alation instance. Create a Support ticket with Alation for an Alation representative to enable the Alation Agent feature on your instance and to receive the Alation Agent installer.

  2. Install Alation Agent using the information in Alation Agent.

  3. Install the OCF connector on Alation Agent.

STEP 2: Create and Configure a New Data Source

In Alation, add a new data source:

  1. Log in to Alation as a Server Admin.

  2. Expand the Apps menu on the right of the main toolbar and select Sources.

  3. On the Sources page, click +Add on the top right of the page and in the list that opens, click Data Source. This will open the Add a Data Source wizard.

  4. On the first screen of the wizard, specify a name for your data source, assign additional Data Source Admins, if necessary, and click the Continue Setup button on the bottom. The Add a Data Source screen will open.

  5. On the Add a Data Source screen, the only field you should populate is Database Type. From the Database Type dropdown, select the connector name. After that you will be navigated to the Settings page of your new data source.

The connector name is Impala CDP OCF Connector.

JDBC URI

Impala supports different types of authentication methods. Use the JDBC URI format that corresponds to your authentication method.

Basic Authentication

Format

impala://<HOSTNAME>:21050/default/;auth=noSasl

Example

impala://ip-10-13-28-190.alation-test.com:21050/default/;auth=noSasl

Kerberized Impala

Format

impala://<HOSTNAME>:21050/default;principal=impala/<sevice_principal>

Example

impala://ip-10-13-6-82.alation-test.com:21050/default/;principal=impala/ip-10-13-6-82.alation-test.com@ALATION-TEST.COM

Connection over SSL

Format

impala://<hostname>:<port>/;auth=noSasl;ssl=true

Examples

If you are using the default truststore, add the parameter ssl=true after the hostname and port.

  • SSL

    impala://ip-10-13-22-251.alation-test.com:21050/default/;auth=noSasl;ssl=true

  • Kerberos + SSL

    impala://ip-10-13-6-82.alation-test.com:21050/default/;ssl=true;principal=impala/ip-10-13-6-82.alation-test.com@ALATION-TEST.COM

Metastore URI Format

thrift://ip-10-13-22-251.alation-test.com:9083

Metastore Principal

hive/ip-10-13-6-82.alation-test.com@ALATION-TEST.COM

Access

On the Access tab, set the data source visibility using these options:

  • Public Data Source—The data source will be visible to all users of the catalog.

  • Private Data Source—The data source will be visible to the users allowed access to the data source by Data Source Admins.

You can add new Data Source Admin users in the Data Source Admins section.

General Settings

Perform the configuration on the General Settings tab.

Application Settings

Specify Application Settings if applicable. Click Save to save the changes after providing the information.

Parameter

Description

BI Connection Info

This parameter is used to generate lineage between a BI source and this data source. Specify the host and the port used by BI connections that pull data from this data source.

Use the following format: host:port

You can provide multiple values as a comma-separated list:

10.13.71.216:1541,sever.com:1542

The host and port details can also be found under Properties > Database connections on the BI Datasource page > Connections tab.

Disable Automatic Lineage Generation

Select this checkbox to disable automatic lineage generation from QLI, MDE, and Compose queries. By default, automatic lineage generation is enabled.

Connector Settings

Data Source Connection

Populate the data source connection information and save the values by clicking Save in this section.

Parameter

Description

JDBC URI

Specify the JDBC URI in the required format. See JDBC URI.

Metastore URI

Specify the metastore URI. See Metastore URI Format.

Username

Specify the service account username.

Password

Specify the service account password.

Enable SSL

Enable SSL authentication by selecting the Enable SSL checkbox.

If the Enable SSL checkbox is enabled, upload the SSL certificate using the upload link below.

Truststore password

Specify the password for the SSL certificate.

Note: The password will be deleted if the data source connection is deleted.

Enable Kerberos Authentication

Enable or disable Kerberos authentication by selecting or clearing the Enable Kerberos Authentication checkbox.

If the Kerberos authentication checkbox is enabled, upload the krb5.conf file using the upload link under the checkbox.

Use keytab

To enable Kerberos with Keytab authentication, select the Use Keytab checkbox and upload the keytab file for the service account using the upload link under the checkbox.

Metastore Principal

Specify the metastore principal.

Logging Configuration

Select the logging level for the connector logs and save the values by clicking Save in this section. The available log levels are based on the Log4j framework.

Parameter

Description

Log level

Select the log level to generate logs. The available options are INFO, DEBUG, WARN, TRACE, ERROR, FATAL, ALL.

You can view the connector logs in Admin Settings > Manage Connectors > Impala CDP OCF Connector.

Obfuscate Literals

Obfuscate Literals—Enable this toggle to hide actual values in the query statements that are ingested during query log ingestion or executed in Compose. This toggle is disabled by default.

Test Connection

After specifying the connector settings, under Test Connection, click Test to validate network connectivity.

Metadata Extraction

The OCF Connector for Impala on CDP supports default extraction—This type of MDE is based on default Impala API calls that are built in the connector code.

Custom query-based MDE is not supported.

Note

Complex data types are extracted. See Extraction of Complex Data Types for information on how to enable their representation in the Alation user interface.

You can configure metadata extraction (MDE) for an OCF data source on the Metadata Extraction tab of the Settings page. Refer to Configure Metadata Extraction for OCF Data Sources for information about the available configuration options.

Compose

For details about configuring the Compose tab of the Settings, refer to Configure Compose for OCF Data Sources.

Note

Compose is not available for on-prem databases connected to the Alation Cloud Service instances through Alation Agent.

Sampling and Profiling

For Impala on CDP, Alation supports a number of ways to retrieve data samples and column profiles. For details, see Configure Sampling and Profiling for OCF Data Sources.

Query Log Ingestion

On the Query Log Ingestion tab, you can select the QLI options for your data source and schedule the QLI job if necessary.

For Impala, Alation supports QLI from query log files stored on HDFS or Amazon S3.

Prerequisites for QLI

In order to run Query Log Ingestion for an Impala data source, you will need to ensure that query logs are available on HDFS or Amazon S3.

Enable Audit for Impala

Enable the audit log for Impala. For specific details refer to documentation for your CDP version. Follow these generic guidelines:

  • Decide how many queries will be represented in each log file. Limiting the size manages disk space by archiving older logs and reduces the amount of text to process when analyzing activity for a particular period. Follow Impala documentation when configuring this aspect of Impala auditing.

  • Periodically pull audit logs from Impala coordinator nodes to HDFS. By default, the audit logs are located in the directory /var/log/impalad/audit/ inside Impala coordinator nodes. An admin can also configure a custom place for audit logs. In each environment, the location of the audit logs depends on the current audit log configuration.

  • Create a directory in HDFS or Amazon S3 for storing all Impala audit logs you want ingested into Alation.

  • The audit log files from each Impala coordinator node should stay in an HDFS subdirectory of the audit log directory. We recommend naming the subdirectory using the ID of the Impala coordinator node, for example, the IP address, or some other unique identifier.

  • You can automate the process by writing a script to pull audit logs from Impala coordinator nodes to HDFS and run it as a cron job every day.

Place Query Log Files on HDFS

Alation supports query log extraction from files located on HDFS and accessible through WebHDFS. To set up QLI:

  1. Make sure there are Impala audit event log files in the audit directories on Impala coordinator nodes that you want to track in Alation. The file names use the format: impala_audit_event_log_1.0-*.

    Note

    The default audit directory is /var/log/impalad/audit. However, your environment may be using a custom audit log directory. Look for the audit files in the audit directory that has been configured as the audit log directory in your CDP environment. To get the audit directory, select the Impala cluster in the CDP Manager, click on Configuration and search for Impala Daemon Audit Log Directory.

  2. After confirming the files exist, create a directory in the /user/history/done directory on HDFS, for example:

    sudo -u hdfs hadoop fs -mkdir -p /user/history/done/log_1

    The name of the directory can be anything you want. We named it log_1 in this example.

  3. After creating the log storage directory on HDFS, recursively modify the permission of the parent directory /done and its sub-directories with the following command:

    sudo -u hdfs hadoop fs -chmod -R 777 /user/history/done

  4. Copy the impala_audit_event_log_1.0-* files into the log_1 directory, for example:

    hadoop fs -put impala_audit_event_log_1.0-* /user/history/done/log_1

As an alternative to the above steps that manually copy log files to a directory, you can use A.8 Impala QLI Script as an example of how to periodically copy the log file to the HDFS.

Place Query Logs on Amazon S3

Alation supports query log extraction from files located in an Amazon S3 bucket. You will need to provide an AWS access key and secret for an account that has Read permissions for this bucket when configuring QLI in Alation.

To set up QLI from Amazon S3:

  1. Make sure there are Impala audit event log files in the audit directories on Impala coordinator nodes that you want to track in Alation. The file names use the format: impala_audit_event_log_1.0-*.

    Note

    The default audit directory is /var/log/impalad/audit. However, your environment may be using a custom audit log directory. Look for the audit files in the audit directory that has been configured as the audit log directory in your CDP environment. To get the audit directory, select the Impala cluster in the CDP Manager, click on Configuration and search for Impala Daemon Audit Log Directory.

  2. Copy log file to the home directory.

    cp impala_audit_event_log_1.0-* /home/<your_user>/

  3. Download the log file to your local machine.

  4. Upload the file to the S3 bucket that you can allow Alation to access. Grant read access to the service account.

Configure QLI in Alation

You can configure QLI on the Query Log Ingestion tab of the data source settings page.

  1. Open the Query Log Ingestion tab of the settings page.

  2. Under Configure Connection Type, select either WebHDFS or Amazon S3, depending on where you made the logs available.

  3. If the logs are on HDFS, provide the following information and click Save:

    Parameter

    Description

    Logs Directory

    /user/history/done

    WebHDFS Server

    IP address or the name of the CDP server

    WebHDFS Port

    Specify the HDFS port number for your environment. The default is 9870. If you are using a different port, clear the Use Default checkbox under the HDFS Port field.

    Use Default

    Leave selected if you are using the default port. Clear this checkbox if you are using a port number other than the default.

    WebHDFS User

    hdfs

  4. If the logs are on Amazon S3, provide the following information and click Save:

    Parameter

    Description

    Impala Log Path

    Your Amazon S3 bucket name

    Impala Log File Name Prefix

    Log file name prefix

    Number of Log files

    Number of log files to ingest

    AWS Access Key ID

    Access key ID to access the bucket

    AWS Access Key Secret

    Access key secret to access the bucket

    AWS Region

    Your AWS region

Perform QLI

You can either perform QLI manually on demand or enable automated QLI:

  1. To perform manual QLI, under the Automated and Manual Query Log Ingestion section of the Query Log Ingestion tab, ensure that the Enable Automated Query Log Ingestion toggle is disabled.

  2. Click Preview to get a sample of the query history data to be ingested.

  3. Click the Import button to perform QLI on demand.

    Note

    Metadata extraction must be completed first before running QLI.

  4. To schedule QLI, enable the Enable Automated Query Log Ingestion toggle.

  5. Set a schedule under Automated Query Log Ingestion Time by specifying values in the week, day and time fields. The next QLI job will run on the schedule you have specified.

    Note

    The hourly schedule for automated QLI is not supported.