Overview

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

The OCF connector for Google BigQuery was developed by Alation and available for download from Alation Connector Hub (requires you to login to the Alation Customer Portal). For instructions to install or manage a connector, see see Manage Connectors. The connector is compiled together with the required database driver, so no additional effort is needed to procure and install the driver.

Use the connector catalog Google BigQuery as a data source on Alation customer-managed (on-premise) and Cloud Service instances. It extracts projects, datasets, tables, and columns and catalogs them as Schema, Table or View, and Columns, respectively.

After metadata is extracted, it is represented in the data catalog as a hierarchy of catalog pages under a parent data source. Alation users can leverage the full catalog functionality to search and find the metadata, curate the corresponding catalog pages, create documentation about the data source, exchange information about it, and follow data transformations using lineage diagrams.

Team

  • Alation administrator:

    • Ensures that Alation Connector Manager is available or installs it.

    • Installs the OCF connector.

    • Creates and configures a Google BigQuery data source in the data catalog.

    • Performs initial extraction and prepares the data source for Alation users.

  • Google Cloud Platform administrator:

    • Assists in configuring authentication for extraction and Compose.

    • Provides the JDBC URI.

Scope

The table below describes which metadata objects are extracted by the connector and which catalog functionality is supported.

Feature

Scope

Availability

Authentication

Authentication with a service account

Authentication with a service account

Yes

Authentication with OAuth and a user account

Authentication with a user account and an OAuth client

Yes

Metadata extraction (MDE)

Default MDE

Extraction of metadata from Google BigQuery projects. This connector uses Google BigQuery API to retrieve metadata into the Alation catalog

Yes

Custom query-based MDE

Extraction of metadata based on extraction queries provided by a user

No

Extracted metadata objects

Data Source

Data source object in Alation that is parent to extracted metadata

Yes

Schemas (Datasets)

List of schemas

Yes

Tables

List of tables

Yes

Columns

List of columns

Yes

Column data types

Column data types

Yes

Views

List of views

Extraction of materialized views is supported

Yes

Source comments

Source comments are extracted for dataset, table, view, | Yes materialized view, and column objects |

Primary keys

Primary key information for extracted tables

No

Foreign keys

Foreign key information for extracted tables

No

Functions

Function metadata

No

Function definitions

Function definition metadata

No

Sampling and profiling

Table sampling

Retrieval of data samples from extracted tables

Yes

Column sampling

Retrieval of data samples from extracted columns

Yes

Deep column profiling

Profiling of columns with the calculation of value distribution stats

Yes

Dynamic profiling

Ability for individual users to connect with their own database accounts to retrieve table and column samples and profiles

Yes

Custom query-based table sampling

Ability to use custom queries for sampling specific tables

Yes

Custom query-based column sampling

Ability to use custom queries for profiling specific columns

Yes

Query log ingestion (QLI)

API-based QLI

Connector performs extraction using API requests.

Yes

Lineage

Automatic lineage generation

Auto-calculation of lineage based on query history ingested from QLI, MDE, and Compose queries

Yes

Column-level lineage

Calculation of lineage data at the column level.

Requires the column-level lineage parser add-on for Google BigQuery

Yes

Compose

Customer-managed (on-prem) Alation instances

Compose on on-prem Alation instances

Yes

Alation Cloud Service instances

Depending on your network configuration, you may be using Alation Agent to connect to your data source.

Compose with Agent is supported from connector version 1.1.1.4664.

Yes

OAuth

Authentication in Compose using the OAuth protocol

Yes