Update Alation to 2024.1.4

Customer Managed Applies to customer-managed instances of Alation

2024.1.4 Update Version Dependencies

Alation supports a direct update to version 2024.1.4 from the following previous versions:

• 2024.1.x

• 2023.3.x

Upgrade from Version 2023.1.x or Older

Direct update to 2024.1.4 is not supported from 2023.1.x or older versions. First, update Alation to 2023.3.x or 2024.1.3 and then perform a second update to 2024.1.4.

Use the update instructions specific to your release.

2024.1.4 Release-Specific Information

Note

The full list of new features is available in the 2024.1.4 release notes.

PostgreSQL Upgrade to Version 16.2

In version 2024.1.4, Alation’s internal database and the Alation Analytics database have been upgraded to version 16.2 to take advantage of new features and security enhancements.

Important

The database upgrade happens automatically during the update but requires additional actions. Detailed instructions are provided in this guide.

The Django Component Upgrade to Version 4.2

If you’re updating to 2024.1.4 from a version older than 2024.1.3, take note of the information below.

In version 2024.1.3, we have upgraded our software infrastructure to Django 4.2, replacing the end-of-support Django 2.x. This new version contains enhancements that improve the robustness, security, and performance of the Alation application.

Note

ACTION REQUIRED

The Django upgrade impacts the SAML authentication configuration in Microsoft Entra ID (former Azure Active Directory) and Active Directory Federation Service (AD FS) as identity providers (IdPs). The default value of the Name Format property of SAML attributes on the Alation server has been changed to unspecified from the previous value of URI. This creates a requirement to explicitly set the Name Format property to URI or Basic in the SAML application in Microsoft Entra ID, as it also defaults to Omitted (default).

Customers who use Entra ID or AD FS as IdPs must perform an additional configuration in their SAML application before upgrading to version 2024.1.3. Refer to the advisory Important Advisory - SAML authentication failure with Azure/Microsoft IDP and Microsoft ADFS after upgrade to 2024.1.3 on Alation Community for specific details.

Update Known Issues

Reset Postgres Password

If you have a password set on the internal PostgreSQL database (Rosemeta), the update will result in an error similar to the following:

WARN: unable to check pg-1: [DbConnectError] unable to connect to 'dbname='postgres' port=5432':
FATAL: password authentication failed for user "postgres"
password retrieved from file "/home/postgres/.pgpass"
ERROR: [056]: unable to find primary cluster - cannot proceed

The update process requires that the password on the internal PostgreSQL database should be cleared for the time of the update. The password can be set again after the update is completed. On how to set the Postgres password, see Set Password for Internal PostgreSQL Instances.

Use Cluster Splitting for HA Pair Update

The Alation update of an HA pair without splitting the cluster fails. We recommend upgrading HA instances using the cluster splitting upgrade method until this issue is resolved.

Update Alation to 2024.1.4

Use the steps in this section to update Alation to 2024.1.4 from versions 2023.3.x and 2024.1.x.

Note

If you are updating from release 2023.3.x and skipping release 2024.1.x, review the release information in the corresponding release notes.

• Release Notes 2024.1 (General Availability)

Follow the steps below to update:

• Step 1: Scan Postgres

• Step 2: Verify Backup Availability

• Step 3: Prepare for the Postgres Upgrade

• Step 4: Update the Alation Application

• Step 5: Update Alation Connector Manager

• Step 6: Update Alation Analytics

• Step 7: Update Extensions and Restore Custom Aggregate Functions

• Step 8: Rebuild Search Index

Step 1: Scan Postgres

We recommend using the scan_postgres action to validate that the internal Postgres database is in a healthy state before the update. For steps, see How to Scan Postgres for Corrupted Indexes.

Note

If in your instance the Postgres scan runs on a schedule, you can check the scan-postgres.log file in /opt/alation/site/logs inside the Alation shell to check the Postgres state.

Step 2: Verify Backup Availability

Ensure you have a valid, up-to-date Alation backup. We recommend taking the most recent backup possible to closely match your current data.

Step 3: Prepare for the Postgres Upgrade

As part of the update to 2024.1.4, Alation’s internal PostgreSQL database, or Postgres, will be upgraded to version 16.2. To facilitate a successful upgrade:

• Check Disk Space

• Update Extensions and Drop Custom Aggregate Functions

Note

On a High Availability (HA) pair that is updated through splitting and rebuilding the HA cluster, these additional steps must be performed on both primary and secondary instances.

Check Disk Space

Verify the available space in the /data1 and /data2 mounts to ensure it’s at least 20% of the total allocated space. Then check that the /dev mount is set to 50% of the physical memory.

To check the disk space:

1. Use SSH to connect to the Alation server.

2. Run the following command:

   df -h
   

3. In the output, look for the Mounted on values /data1 and /data2. For these values, look at the Use% column. It must be at 80% or less, which means 20% or more of space is currently available.

   Note

   To calculate available disk space, subtract the Use% value from 100%: 100% - Use% = available space. For example, if we had to calculate the available space for the example output below, /data1 has 99% available, and /data2 has 95% available. Each has more than 20% available.

   Filesystem      Size  Used Avail Use% Mounted on
   /dev/xvdd1      147G  482M  139G   1% /data1
   /dev/xvdb1       98G  4.6G   89G   5% /data2
   devtmpfs         15G  156K   15G   1% /dev
   

   If you discover that less than 20% space is available on /data1 or /data2, add more disk space before performing the update. The available space must be 20% or more.

   Note

   If you use Alation Analytics, please note that it has similar space requirement for the update to 2024.1.4. See more in Check Disk Space for Alation Analytics. If you need to add space, consider increasing disk space for both simultaneously to Alation to avoid repetitive adjustments.

4. Check that the /dev directory is set to 50% of the physical memory. If not, it’s recommended setting it to 50%.

   Note

   The following command can be used:

   sudo mount -o remount,size=<value> devtmpfs /dev
   

   For example, if the physical memory is 64GB, then to set /dev to 50%:

   sudo mount -o remount,size=32G devtmpfs /dev
   

Update Extensions and Drop Custom Aggregate Functions

To ensure compatibility with the updated database version in 2024.1.4, you’ll need to update database extensions and identify and drop any custom aggregate functions of types anyarray and anyelement in your Postgres databases. Such functions are not typically part of the standard Alation product but may exist due to previous customizations of Postgres. After the upgrade, any custom functions that were dropped must be manually restored to maintain their functionality.

If any of the user-defined functions exist during the update, it is expected to fail with the following exception in the update logs installer.log (/opt/alation/site/logs inside the chroot):

Your installation contains user-defined objects that refer to internal polymorphic functions with arguments of type “anyarray” or “anyelement”. These user-defined objects must be dropped before upgrading and restored afterwards, changing them to refer to the new corresponding functions with arguments of type “anycompatiblearray” and “anycompatible”.

Note

If you have custom aggregate functions in your Postgres instances, you may need to review and update the SQL code of the function definition when recreating them in PostgreSQL version 16.2. You will be able to retrieve the current function definitions using this guide. Carefully review the instructions and plan your steps accordingly. You can reach out to Alation Support if you need assistance recreating the functions.

Alation provides a script that helps prepare for the Postgres upgrade. The script covers both types of Postgres deployments: built-in or externalized to Amazon RDS. The script should be used to:

• Update Postgres extensions

• Check for the presence of custom aggregate functions in the internal Postgres databases (Rosemeta, Lineage, Template1).

If custom aggregate functions are found, the same script will be used to drop them.

To prepare for the Postgres upgrade:

1. Download the Zip file with the script to your local machine from Alation Community: 2024.1.4 Update Helper Script (requires a login)

2. Extract the script file pgupgrade_helper.py.

3. Copy the file to the Alation server. Initially, you can place it into the /tmp directory on the host.

4. Use SSH to connect to the Alation server.

5. Transfer the extracted pgupgrade_helper.py file to a directory accessible from the Alation chroot. For example, if you have it in the /tmp directory on the host, you could copy it to the /tmp directory /opt/alation/alation/data1/tmp/, which is /data1/tmp when accessed from inside the chroot:

   sudo cp /tmp/pgupgrade_helper.py /opt/alation/alation/data1/tmp/
   

   Note

   The same script will also be used when updating Alation Analytics. We recommend leaving the script in this directory until you have upgraded all Alation components.

6. Enter the Alation shell.

   sudo /etc/init.d/alation shell
   

7. Change the user to alation.

   sudo su alation
   

8. Navigate to the directory where you placed the pgupgrade_helper.py file, for example /data1/tmp/ (inside the chroot) if you previously moved it there.

9. Run the script with the --pre parameter. The script will update Postgres extensions and check for the presence of custom aggregate functions of types anyarray and anyelement. If any functions are dropped, their definitions will be saved on the Alation server. The script will not drop any functions yet, allowing for an analysis of what customizations your instance contains.

   python pgupgrade_helper.py --pre
   

10. Analyze the output: look for the text Aggregate functions Check [Failed] and a list of functions below it.

    • If you don’t find this line and a list of functions, the script hasn’t identified any functions to be dropped. Exit from the alation user and the Alation shell by using exit twice. Continue with the Alation update: Step 4: Update the Alation Application.

      Example output when no functions were identified:

      (env) PROD [alation@ip-10-13-31-60 ~]$ python pgupgrade_helper.py --pre
      Pre flag: True
      Post flag: False
      CPU: None
      Drop Function: False
      Connected to database postgres [Success]
      Updating Extensions [Success]
      Connected to database rosemeta [Success]
      Updating Extensions [Success]
      Connected to database template1 [Success]
      Updating Extensions [Success]
      Connected to database lineage [Success]
      Updating Extensions [Success]
      Pre Upgrade tasks completed [Success]
      

    • If you find the line Aggregate functions Check [Failed] and a list of functions below it, those functions need to be dropped. Continue to step 11 of this instruction.

      Example output where a function was identified:

      (env) PROD [alation@ip-10-13-31-60 ~]$ python pgupgrade_helper.py --pre
      Pre flag: True
      Post flag: False
      CPU: None
      Drop Function: False
      Connected to database postgres [Success]
      Updating Extensions [Success]
      Connected to database rosemeta [Success]
      Aggregate functions Check [Failed]
      - array_agg_array
      Updating Extensions [Success]
      Connected to database template1 [Success]
      Updating Extensions [Success]
      Connected to database lineage [Success]
      Updating Extensions [Success]
      Pre Upgrade tasks completed [Success]
      

11. Run the script again with the --pre and --drop-func parameters. The script will retrieve the definitions of the identified aggregate functions, record them into a file, and drop the functions.

    Note

    The --pre parameter is still required during the second run as it identifies the pre-upgrade stage of the process.

    Example output:

    (env) PROD [alation@ip-10-13-31-60 ~]$ python pgupgrade_helper.py --pre --drop-func
    Pre flag: True
    Post flag: False
    CPU: None
    Drop Function: True
    Connected to database postgres [Success]
    Updating Extensions [Success]
    Connected to database rosemeta [Success]
    Aggregate functions Check [Failed]
    - array_agg_array
    Dropped function array_agg_array [Success]
    Updating Extensions [Success]
    Connected to database template1 [Success]
    Updating Extensions [Success]
    Connected to database lineage [Success]
    Updating Extensions [Success]
    Saved all the dropped functions definition: /data1/tmp/recreate-agg-function.sql
    Pre Upgrade tasks completed [Success]
    

12. Check the contents of the recreate-agg-function.sql file. (The path to the file will be in the line Saved all the dropped functions definition:. It depends on where you have the script, as the file is generated in your working directory). The file contains the information about the databases and the function definitions of all dropped functions. For example:

    (env) PROD [alation@ip-10-13-31-60 ~]$ cat /data1/tmp/recreate-agg-function.sql
    rosemeta,create aggregate public.array_agg_array(anyarray)
    (sfunc = array_cat, stype = anyarray);
    

    Where:

    • rosemeta is the database from which a function was deleted

    • create aggregate public.array_agg_array(anyarray) (sfunc = array_cat, stype = anyarray); is the corresponding function definition.

13. Copy the file to your local machine, renaming it to recreate-agg-function_Postgres.sql. It’s crucial to restoring the functions manually after the update.

    Warning

    The file should be saved locally as it will be overwritten when you run the same script during the Alation Analytics update.

14. Exit from the alation user.

    exit
    

15. Exit from the Alation shell.

    exit
    

16. Proceed with the Alation update: Step 4: Update the Alation Application.

Step 4: Update the Alation Application

Update instructions:

• Manual Software Update for Standalone Systems

• Update on the HA pair:

  • with splitting the HA cluster: Update HA Pair with Cluster Splitting

    Note

    See Use Cluster Splitting for HA Pair Update under Update Known Issues for this release.

Step 5: Update Alation Connector Manager

This step applies if you are using Open Connector Framework (OCF) and OCF connectors.

Update Alation Connector Manager using the steps in Update Alation Connector Manager.

Step 6: Update Alation Analytics

This step applies if you are using the Alation Analytics application.

Use the steps in Update Alation Analytics Database to 16.2 and a Compatible Release to update the Alation Analytics.

Step 7: Update Extensions and Restore Custom Aggregate Functions

After updating Alation, it’s important to update the extensions and restore the custom aggregate functions if any were dropped to ensure compatibility with the new version.

• Update Extensions

• Restore Custom Aggregate Functions

Update Extensions

To update the Postgres extensions:

1. Use SSH to connect to the Alation instance.

2. We recommend running the next command using a terminal multiplexer, such as Screen. If Screen is available, start a screen session.

   screen -S alation-extensions
   

3. Enter the Alation shell.

   sudo /etc/init.d/alation shell
   

4. Change the user to alation.

   sudo su alation
   

5. Navigate to the directory where you placed the pgupgrade_helper.py file.

6. Run the script with the --post parameter. The script will update Postgres extensions and run an analysis of the internal Postgres databases on your instance.

   python pgupgrade_helper.py --post
   

   Example output:

   (env) PROD [alation@ip-10-13-31-60 ~]$ python pgupgrade_helper.py --post
   Pre flag: False
   Post flag: True
   CPU: None
   Drop Function: False
   Connected to database template1 [Success]
   Updating Extensions [Success]
   Connected to database postgres [Success]
   Updating Extensions [Success]
   Connected to database rosemeta [Success]
   Updating Extensions [Success]
   Connected to database lineage [Success]
   Updating Extensions [Success]
   vacuumdb: vacuuming database "lineage"
   vacuumdb: vacuuming database "postgres"
   vacuumdb: vacuuming database "rosemeta"
   vacuumdb: vacuuming database "template1"
   Analyze DB [Success]
   Post Upgrade tasks completed [Success]
   

7. Your next step depends on whether or not any custom aggregate functions were dropped before the update:

   • If the script previously didn’t drop any custom aggregate functions:

     • If you have updated from version 2024.1.x, the Alation update is complete. Exit from the alation user and the Alation shell by using exit twice. Test your instances post-update. If you identify any issues with the Alation application, contact Alation Support.

     • If you have updated from version 2023.3.x, proceed to Step 8: Rebuild Search Index and check if you need to update the search index.

   • If the script previously dropped any functions, restore them manually: Restore Custom Aggregate Functions.

Restore Custom Aggregate Functions

To restore the functions:

1. Review the function definitions from the recreate-agg-function_Postgres.sql file that was generated by the script in Step 3: Prepare for the Postgres Upgrade.

2. Update the SQL code as necessary to be compatible with PostgreSQL version 16.2. You may need to change anyarray to anycompatiblearray and anyelement to anycompatible or update the SQL code in other ways, depending on your specific customization.

3. On the Alation host, enter the Alation shell if you have exited if previously.

   sudo /etc/init.d/alation shell
   

4. Enter the Postgres shell.

   alation_psql
   

5. Run the following command for each function to be recreated, replacing the placeholder func-definition with the SQL code of the specific definition you are restoring.

   CREATE AGGREGATE func-definition;
   

   Example:

   CREATE AGGREGATE public.array_agg_array(anyarray) (sfunc = array_cat, stype = anyarray);
   

6. Exit alation_psql.

   \q
   

7. Your next step depends on which release you have updated from:

   • If you have updated from version 2024.1.x, your Alation update process is complete. Exit from the alation user and the Alation shell by using exit twice. Test your instances post-update. If you identify any issues with the Alation application, contact Alation Support.

   • If you have updated from version 2023.3.x, proceed to Step 8: Rebuild Search Index and check if you need to update the search index.

Step 8: Rebuild Search Index

Rebuild your search index if:

• Your search index hasn’t been updated since version 2022.4. Even if you’ve updated Alation from version 2023.3.x, the index may still be using the old search schemas.

Determine if Search Index Requires Rebuilding

To determine if your search index requires rebuilding:

1. Use SSH to connect to the Alation server.

2. Enter the Alation shell using the following command:

   sudo /etc/init.d/alation shell
   

3. Run the following command to inspect the schema of your search index:

   curl localhost:9200/live/_mappings | grep dbtype.
   

4. In the output, look for the dbtype field output:

   • If dbtype is text, then a rebuild is required.

     

   • If dbtype is keyword, then no action is needed.

     

5. If you’ve determined that an index rebuild is required, proceed to index rebuilding.

Rebuild Search Index

Rebuilding the search index aligns it with the latest search schemas of version 2023.3.5 and can be performed without downtime. For detailed steps, see How to Rebuild Search Index Without Downtime.