VCF

The SDDC Manager offers two options to replace certificates for itself and the other BoM components in a VCF Environment: Microsoft CA or OpenSSL CA.

A built-in option to replace the component certificates using a VMCA Signed Certificate, from the Management WLD vCenter Server is not available. While we do have some manual steps to replace the component certificates with a VMCA signed cert for the other components, there really isn't a readily available option to do this for the SDDC Manager itself.

Additionally, if the SDDC Manager certificate is expired then, we cannot leverage the functionality of the SDDC Manager to replace any certificates (whether using the UI or the API) till its certificate is renewed manually, to bring its services back online.

To that end, I've written a script to automate this entire process. This python script is run as root on the SDDC Manager. The script performs the following operations:

• Create a local CSR and Private Key on the SDDC Manager
• Using the CSR, get a VMCA Signed Certificate from the Management WLD vCenter Server
• Apply the Certificate on the SDDC Manager
• Restart the nginx service on the SDDC Manager
• Confirm that we are seeing the applied cert on port 443

The script is available on my Github here.

A VCF Environment consists of multiple components, including but no limited to vCenter Server, ESXi, NSX, vSAN, the Aria Suite Products etc. Depending on the size of the VCF environment, and the number of workload domains present in it, the inventory of the components and their versions can get out-of-hand.

SDDC Manager, in its current state, is unable to dynamically query and update its inventory state information, particularly when it comes to tracking the versions of the various BoM components.

A rather easy way to update this inventory information is using the --performInventorySync option available in the Async Patch Tool.

With the announcement of End of Availability of Perpetual Licensing and SaaS services for VMware by Broadcom services and the release of VMware Cloud Foundation 5.1.1, VCF components can now use a single solution license instead of individual component licenses.

As mentioned in the VCF 5.1.1 release notes:

Solution Licensing: The component products of VCF can now derive their entitlements from a single solution license key (except for vSAN, which still requires a separate license). Individual component license keys continue to be supported. Use the vSphere Client to apply the solution license or the SDDC Manager UI to apply individual component licenses.

With the increasing growth and constant evolution of the VMware product stack, troubleshooting all the components and services involved is no simple feat. One of the most useful tools for getting a head start in the troubleshooting process is the VCF Diagnostic Tool (VDT).

VDT (developed and built by VMware Support) is a utility designed to run a series of comprehensive checks live on a target appliance. In its current state, VDT supports the vCenter Server and SDDC Manager appliances.

Depending on where its run, VDT analyzes the appliance and/or environment configuration , performing a series of checks to identify potential issues or inconsistencies. It presents results in a user-friendly format, categorizing findings as PASS, WARN, or FAIL, along with informative INFO messages. The goal is to offer real-time diagnostic information, enabling administrators to proactively address any issues that it catches.

VDT is exclusively read-only. While the tool runs several commands and API calls to gather and analyze data, it does not make any changes to the environment. This ensures that admins can rely on the diagnostic insights without worrying about unintended or unscheduled modifications. Detailed logs from each run of the utility are generated and stored within the appliance, and automatically collected as a part of each appliance support bundle.

A common error that VCF admins may encounter is the reject HostKey error. This indicates a mismatch or incorrect SSH Host Key entry stored in the SDDC Manager's known_host file(s). In this post, we'll walkthrough the process of addressing and resolving this issue.

Caused by: com.jcraft.jsch.JSchException: reject HostKey: 127.0.0.1
    at com.jcraft.jsch.Session.checkHost(Session.java:799)
    at com.jcraft.jsch.Session.connect(Session.java:345)
    at com.jcraft.jsch.Session.connect(Session.java:183)

SDDC Manager uses the jsch library, which is a pure Java implementation of SSH2. jsch allows you to connect to a sshd server and use port forwarding, file transfer, scp, sftp etc

The reject HostKey error indicates a discrepancy between the expected Host Key and the one presented during an SSH connection attempt. This can occur due to various reasons, such as system restores, upgrades, manual key changes, manual re-installs etc, leading to authentication failures. These authentication failures can affect operations and workflows throughout the VCF environment.

VMware just announced VMSA-2023-0023 with a maximum CVSSv3 base score of 9.8 - which basically implies DANGER! Here is everything you need to know about this for your VCF Environments.

VMSA-2023-0023 includes two CVEs: CVE-2023-34048, CVE-2023-34056 - The severity of these issues has been evaluated to be in the Critical severity range with a maximum CVSSv3 base score of 9.8.

This VMSA affects all versions of vCenter Server, and is only limited to vCenter, not ESXi.

Given the close integration of the SDDC Manager with all the components comprising a VCF Environment, making changes to components can be a bit of a challenge. In this article, lets talk about one of the more simpler changes - Renaming components.

As a rule, making changes to any object or component that is a part of a VCF Environment should only be done through the SDDC Manager. Making changes directly on the component itself is UNSUPPORTED - as SDDC Manager will not have visibility over this change, thereby causing a discrepancy in the inventory information.

While the changes should only be made through the SDDC Manager, there is no mechanism or workflow in place that restricts one from making the changes directly via the component itself. However, as mentioned above, the change still is unsupported. If any changes are made which cause a deviation from the inventory information that the SDDC Manager holds, this can (and will) cause issues with any number of Day-2 operations and workflows (such as adding hosts and WLDs, updates/upgrades, expanding clusters)

In terms of actually renaming components in a VCF Environment, as of VCF 4.5, here are the following components that support renaming:

Since the release of VCF 4.4.0.0, there has been a lot of chatter about how we can decouple or disassociate the vRealize suite from VCF and SDDC Manager, or how it is completely externally managed.
This is quite incorrect and stems from a misunderstanding of how the vRealize Suite is linked to VCF.

In this post, I will talk about how the vRealize Suite is linked to VCF and SDDC manager, and what has changed in VCF 4.4 and above.

During upgrades of VCF components from the SDDC Manager, we often run into situations where a component upgrade bundle does not show up for a particular VCF version upgrade. This can also present itself as skipping a component through the upgrade process. In this article, I will explain what compatibility sets are, how they work, and how they affect the VCF component upgrade process.

What are Compatibility Sets?

Compatibility sets are set of version entries for the 3 primary core VCF products - namely vCenter, ESXi and NSX(T/V), which are marked as compatible with one another. Lets break it down further.

Each entry in the compatibility sets contains one version each for the vCenter, ESXi and NSX component.

For example, from the screenshot above, the final entry indicates that vCenter version 7.0.3.00300, ESXi version 7.0.3, and NSX-T version 3.1.3.7.4 are basically marked as compatible to work with each other in a VCF environment.

The VerionAlias.yml file on a SDDC Manager is referenced quite a bit across several VMware KBs, mostly in the context of "change this value here" or "add this value there" while troubleshooting VCF upgrades. There really aren't any articles online describing what this file really means or how it works. In this post, I will describe the various elements that are in a VersionAlias.yml file, and how to interpret the information available here.

Lets start with the different versions of each VCF component that we are dealing with: