VMware Cloud Foundation: SDDC Manager VMware Compatibility Data Is Missing

Sun, Jul 16, 2023 VMware
Share on:

Recently I was upgrading one the VCF environments I manage from 4.5 to 5.0. After upgrading SDDC Manager was presented with the following LCM errors:

1Failed to retrieve silenced health checks. A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.
2Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles: Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api: 409 - "{\"errorCode\":\"LCM_ERROR_MGMT_DOMAIN_NOT_FOUND\",\"arguments\":[],\"message\":\"Management Domain not found\",\"referenceToken\":\"QLMSVA\"}"
3Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles: Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api: 500 - "{\"errorCode\":\"VCF_ERROR_INTERNAL_SERVER_ERROR\",\"arguments\":[],\"message\":\"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.\",\"causes\":[{\"type\":\"com.vmware.evo.sddc.lcm.model.error.LcmException\",\"message\":\"Failed to validate if SDDC Manager with version 5.0.0.0-21822418 is compatible with system.\"}],\"referenceToken\":\"76UR6K\"}"
4Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles: Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api: 500 - "{\"errorCode\":\"VCF_ERROR_INTERNAL_SERVER_ERROR\",\"arguments\":[],\"message\":\"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.\",\"causes\":[{\"type\":\"com.vmware.evo.sddc.lcm.model.error.LcmException\",\"message\":\"Failed to validate if SDDC Manager with version 5.0.0.0-21822418 is compatible with system.\"}],\"referenceToken\":\"EEVA7E\"}"
5Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles: Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api: 500 - "{\"errorCode\":\"VCF_ERROR_INTERNAL_SERVER_ERROR\",\"arguments\":[],\"message\":\"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.\",\"causes\":[{\"type\":\"com.vmware.evo.sddc.lcm.model.error.LcmException\",\"message\":\"Failed to validate if SDDC Manager with version 5.0.0.0-21822418 is compatible with system.\"}],\"referenceToken\":\"IPC75R\"}"
6Retrieving update patches bundles failed. Unable to retrieve aggregated LCM bundles: Error in getDomainBomVersion(): Encountered error requesting http://127.0.0.1/v1/system/upgradables api: 500 - "{\"errorCode\":\"VCF_ERROR_INTERNAL_SERVER_ERROR\",\"arguments\":[],\"message\":\"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.\",\"causes\":[{\"type\":\"com.vmware.evo.sddc.lcm.model.error.LcmException\",\"message\":\"Failed to validate if SDDC Manager with version 5.0.0.0-21822418 is compatible with system.\"}],\"referenceToken\":\"UM61AQ\"}"
7VMware compatibility data is missing. To download and update the compatibility data, see the documentation.

Most of the VCF/VCF on VxRail environments I manage are air gapped environments with no proxy server or internet access. After looking through logs and documentation I realized a “Compatibility Matrix” JSON file was missing from SDDC Manager. After reviewing the offline bundle documentation from VMware Docs I noticed I had skipped over the Compatibility Matrix import process. I also found a VMware KB that goes over how to disable “upgrade compatibility checks” as a workaround.

In this walkthrough I will go over how to download and upload Compatibility Data to SDDC Manager or bypass upgrade compatibility checks.

Table of Contents

Downloading and Uploading Compatibility Data to SDDC Manager

Prerequisites

You will need the following before continuing:

  • VMware Customer Connect account with an entitlement for VMware Cloud Foundation.
  • A endpoint (Linux or Windows) that has internet connectivity.
    • For this demo, I will be using a Windows workstation.
  • Java Runtime Environment Installed
    • Can be downloaded here
  • WinSCP Installed
    • Can be downloaded here
  • A process worked out with the security team in order to bring JSON outside of your environment.

Download the Bundle Transfer Utility

  1. Download the Bundle Transfer Utility on a computer with internet access.
    • Log in to VMware Customer Connect and browse to the Download VMware Cloud Foundation page.
    • In the Select Version field, select the version to which you are upgrading.
    • Click Drivers & Tools
    • Expand VMware Cloud Foundation Tools and click Go To Downloads.
    • Click Download Now for the Bundle Transfer Utility.

  1. Extract lcm-tools-prod.tar.gz on workstation connected to the internet

Downloading the Compatibility Data

Downloading Compatibility Data on the internet connected workstation

This is a structured metadata file that contains information about the VMware product versions included in the release Bill of Materials.

  1. On the workstation with internet access. Run Powershell or Command Prompt as administrator.

  2. Change directory to the “lcm-tools” you recently downloaded

  3. Change directory to the “bin” folder

  4. Type the following command: (Note it is case sensitive!)

    This part will require a MyVMware account that has entitlement for VMware Cloud Foundation.

1.\lcm-bundle-transfer-util --download --compatibilityMatrix --depotUser Username
2

You can also use the following command if you’re VCF on VxRail. This part will require a MyVMware account that has entitlement for VMware Cloud Foundation and Dell account.

1.\lcm-bundle-transfer-util --download --compatibilityMatrix --depotUser Username --pdu Username
2

Uploading the Bundle Transfer Utility to SDDC Manager

  1. Transfer “lcm-tools-prod.tar.gz” you downloaded from step 1 to the computer with access to SDDC Manager Appliance.

  2. Copy the “lcm-tools-prod.tar.gz” to SDDC Manager Appliance.

Note: I will be using WinSCP to accomplish this task and I will be copying the “lcm-tools-prod.tar.gz” to /home/vcf/ on SDDC Manager Appliance.

How to connect to SDDC Manager using WinSCP

  1. Open WinSCP
  2. Set file protocol to “SCP”
  3. Provide the Hostname or IP to your SDDC Manager (example, sddc-manager.sddc.esx.lab)
  4. Port Number should be 22
  5. Username should be “vcf”
  6. Enter the password for “vcf” and press login

  1. SSH into the SDDC Manager appliance using “vcf” user account.

  2. Enter “su” to switch to the root user.

  3. Create the lcm-tools directory

1 mkdir /opt/vmware/vcf/lcm/lcm-tools
  1. Extract the “lcm-tools-prod.tar.gz” to the new “lcm-tools” directory.
1tar -xvzf /home/vcf/lcm-tools-prod.tar.gz -C /opt/vmware/vcf/lcm/lcm-tools/

  1. After extracting “lcm-tools-prod.tar.gz” to the new “lcm-tools” directory will need to set the permissions.
1cd /opt/vmware/vcf/lcm/
2chown vcf_lcm:vcf -R lcm-tools
3chmod 750 -R lcm-tools
  1. After setting permissions on the “lcm-tools” directory, “exit” the “su” user

  1. Navigate to the /opt/vmware/vcf/lcm/lcm-tools/bin/ directory

  2. Validate that the utility version matches the target release. Run the following command to display the utility version:

./lcm-bundle-transfer-util -v

Note: You may have to answer the CEIP as “root”

Uploading and Importing Compatibility Data to SDDC Manager

Uploading the Compatibility Data to SDDC Manager

  1. Open WinSCP and connect to the SDDC Manager VM
  2. Change directory in WinSCP
  3. Make a new directory called “Compatibility” through WinSCP or Command line.

Note: Folder name is case sensitive.

  1. Copy the compatibility Data you downloaded into a directory (Ex. /home/vcf/Compatibility) on the SDDC Manager

Importing the Compatibility Data

  1. SSH into SDDC Manager VM
  2. Change directory to the following
1cd /opt/vmware/vcf/lcm/lcm-tools/bin
  1. Run the following command:
1./lcm-bundle-transfer-util --update --compatibilityMatrix --inputDirectory /home/vcf/ --sddcMgrFqdn FQDN -sddcMgrUser administrator@vsphere.local
2

  1. Wait for the tool to validate the compatibility data JSON file.

  2. You should see the following output when the process is completed.

Verify compatibility data import within the SDDC Manager client

  1. Log into SDDC Manager client using a web browser.
1https://sddc-manager.esx.lab

  1. Go to “Developer Center”

  2. Select “API Explorer” and “CompatibilityMatrix”

  1. Expand “GET” and Select “Execute”

  2. Under Response select “PageOfCompatibilityMatrix” and review the output to verify it was uploaded.

Thats it! You have successfully imported the compatibility data into SDDC manager.

Bypass SDDC Manager upgrade compatibility checks

Disable the upgrade compatibility checks

  1. SSH into SDDC Manager appliance with the vcf user
  2. su to root
  3. Edit the following file
1vi /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag
2

  1. Update the property “vcf.compatibility.controllers.compatibilityCheckEnabled” to false

  1. Save and close the file

  2. Restart Lifecycle Management by running the command

1systemctl restart lcm
2

  1. Continue with your upgrades

Re-enable the compatibility checks once the upgrades are complete to ensure guardrails are in place with respect to Bill-of-Materials compatibility for future upgrades.

Re-enable the compatibility checks

  1. SSH into SDDC Manager appliance with the vcf user
  2. su to root
  3. Edit the following file
1vi /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag
2

  1. Update the property vcf.compatibility.controllers.compatibilityCheckEnabled to true

  1. Save and close the file

  2. Restart Lifecycle Management by running the command

1systemctl restart lcm
2

Sources

Downloading VMware Cloud Foundation Upgrade Bundles

Bypass SDDC Manager upgrade compatibility checks when upgrade is unavailable due to missing compatibility data (90074)