Tableau OCF Connector: Migration from Tableau Native Connector to OCF

You can migrate your existing Tableau BI source that is cataloged using the built-in (native) connector for Tableau to the Tableau OCF connector. The Tableau OCF connector extracts metadata using the Tableau REST API and Metadata API and improves the overall quality and consistency of the extracted metadata.

The migration of a native Tableau BI source to OCF consists of two stages and involves running two migration scripts. The first stage will migrate the Tableau BI source and most of its child objects except three object types — BI Report Columns, BI Datasource Columns, and BI Datasources. These three object types are soft-deleted but not migrated by the first migration script. The second stage and the second script will migrate the logical metadata for these three remaining object types.

Perform the steps below to migrate your Tableau BI source to OCF.

Step 1: Migrate Your Native Tableau BI Source to Tableau OCF Connector

Perform the Tableau BI source migration using the instructions in Migration from Native to OCF Connector.

Once this migration is complete, the Tableau BI source and most of its physical metadata (child BI objects under this BI source) and the corresponding logical metadata (descriptions, custom field values, @-mentions, and references) will be migrated to OCF. However, the BI object types of BI Report Columns, BI Datasource Columns, and BI Datasources (unpublished or embedded Tableau datasources) require performing additional steps for the curated information to be migrated. Follow the next steps to migrate the BI Report Column, BI Datasource Column, and BI Datasource objects under your Tableau BI source.

Step 2: Perform Metadata Extraction

Re-extract all objects from the migrated Tableau BI source using selective or full extraction, depending on the extraction configuration you had before migrating the source.

To perform extraction:

  1. On the Alation host, enter the Alation shell:

sudo /etc/init.d/alation shell

  1. Using alation_conf, make sure that the following flags are set to True. These flags should be True by default. This step ensures that your instance uses the default values and that they were not previously changed.

    alation_conf alation.bi_metadata.enable_soft_delete
alation_conf alation.ocf.mde.gbm.soft_delete.enabled

You can leave the Alation shell session open for now as further instructions require performing more actions in the shell.

  1. In the Alation user interface, open the Settings page of the migrated Tableau BI source.

  2. Make sure that the Remove Projects that are not captured by the list above checkbox under Settings > Extraction Settings > Selective Extraction is selected.

  3. Select the projects to extract and perform extraction. This will refresh the catalog metadata using the Tableau OCF connector. As the pages for the child BI Report Column, BI Datasource Column, and BI Datasource objects were soft-deleted and not migrated, Alation will create new pages for these objects. These pages will have new object IDs in the catalog and no curated information.

Next, you need to match the new pages with the corresponding soft-deleted pages and migrate the logical metadata from the old to the new pages.

Step 3: Match the Old and the New Pages

As was mentioned above, the following objects are soft-deleted but not migrated during the Tableau BI source migration process:

  • BI Report Column

  • BI Datasource Column

  • BI Datasource (only unpublished or embedded datasources)

The soft-deleted pages still exist in the internal server database and can be used to migrate the curated content (descriptions, custom field values, @-mentions, and references) to the corresponding new pages created by extraction.

Perform the following steps to match the soft-deleted pages with the corresponding new pages:

  1. In the Alation shell, navigate to the directory /opt/alation/django/.

  2. Alation provides a special script to migrate the soft-deleted object types. Run the command given below from /opt/alation/django/ to view the available script execution options.

    python -m rosemeta.one_off_scripts.copy_tableau_logical_data_from_deleted_objects_to_new_objects --help

    Arguments

    Description

    -s SOURCE_ID

    Allows to test-run the script and outputs the potential changes. Returns the list of soft-deleted objects for the migrated Tableau BI source that cannot be matched to the corresponding new objects.

    –confirm

    Executes the script and applies the changes to the database.

    –path MATCHES_FILE_PATH

    Provides the path to the .txt file that contains the old ID and new ID matches for the bi_report_column, bi_datasource_column, or bi_datasource child objects.
    ../../../_images/TableauOCF_08.png

  3. Next, you will need to work on the mapping between the soft-deleted objects and the corresponding new objects. You will need to create a separate .txt file that maps the old and the new IDs for each of the object types — BI Report Columns, BI Datasource Columns, and BI Datasources.

    Run the command given below substituting the placeholder value <bi_server_id> with the ID of your migrated Tableau source. It will test-run the script and return the list of all changes that will be potentially performed without applying the changes to the database yet. The output will include lists of the IDs of the soft-deleted objects that were not migrated. These IDs will be introduced by the sentence Could not find a definitive matching “<object_type>” for deleted <object_type> ids.

    python -m rosemeta.one_off_scripts.copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <bi_server_id>

    Example output:

    Could not find a definitive matching "new datasource column"  for the deleted datasource column ids: [45, 46, 47, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 255, 144, 151, 188, 219, 225, 114, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 12, 13] Skipping them for now...

  4. Run the following command to include the changes in the database for the IDs that matched the logic.

    python -m rosemeta.one_off_scripts.copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <id> --confirm

  5. Find the new IDs of the corresponding objects that were created during the extraction. We recommend tackling this task by object type. For example, start with BI Report Column objects and match all old and new objects of this type. Save the old IDs and the new IDs for each matched object as a pair of values.

    Use the following paths templates to go to each soft-deleted object page under the migrated Tableau BI source. Substitute the placeholder value <old_id> with the actual ID of the soft-deleted object that was returned in step 3 above.

    Object Type

    Path

    BI Report Column

    /bi/v2/report_column/<old_id>/

    BI Datasource Column

    /bi/v2/datasource_column/<old_id>/

    BI Datasource

    /bi/v2/datasource/<old_id>/

  6. When you open each soft-deleted page, copy the Title of the soft-deleted object.

    ../../../_images/TableauOCF_09.png

  7. Go to the page of the corresponding parent object under the migrated Tableau BI source.

  8. In the Filter field, paste the Title of the soft- deleted object.

    ../../../_images/TableauOCF_10.png

  9. Click the link in the filtered table to go to the corresponding new page.

  10. Save the new ID of the object that is displayed in the URL. We recommend saving this ID next to the ID of the corresponding soft-deleted object.

    Note

    You can ignore the object IDs that do not have any curated information.

    ../../../_images/TableauOCF_11.png

  11. After matching all soft-deleted and the corresponding new objects, create three .txt files with the following names:

    • bi_report_column.txt

    • bi_datasource_column.txt

    • bi_datasource.txt

  12. In each file, list the corresponding old IDs and the new IDs for all matched objects in each type of objects. Each pair of values should be on a separate line. The format should always be <old ID>, <new ID>.

    Example:

    ../../../_images/TableauOCF_12.png

  13. Copy the files into the /opt/alation/site/tmp/ folder inside the Alation shell.

Step 4: Migrate the Logical Metadata From Soft-Deleted Pages

  1. From the Alation shell and the directory /opt/alation/django/,run the following command for each of the files you created.

    python -m rosemeta.one_off_scripts.copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <Tableau_source_ID> --path <Path/File_Name.txt>

    Example:

    python -m rosemeta.one_off_scripts.copy_tableau_logical_data_from_deleted_objects_to_new_objects -s 4 --path /opt/alation/site/tmp/bi_datasource_column.txt

  2. Run the next command to include the changes into the database.

    python -m rosemeta.one_off_scripts.copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <id> --confirm