The SDDC Manager offers two options to replace certificates for itself and the other BoM components in a VCF Environment: Microsoft CA or OpenSSL CA.

A built-in option to replace the component certificates using a VMCA Signed Certificate, from the Management WLD vCenter Server is not available. While we do have some manual steps to replace the component certificates with a VMCA signed cert for the other components, there really isn't a readily available option to do this for the SDDC Manager itself.

Additionally, if the SDDC Manager certificate is expired then, we cannot leverage the functionality of the SDDC Manager to replace any certificates (whether using the UI or the API) till its certificate is renewed manually, to bring its services back online.

To that end, I've written a script to automate this entire process. This python script is run as root on the SDDC Manager. The script performs the following operations:

• Create a local CSR and Private Key on the SDDC Manager
• Using the CSR, get a VMCA Signed Certificate from the Management WLD vCenter Server
• Apply the Certificate on the SDDC Manager
• Restart the nginx service on the SDDC Manager
• Confirm that we are seeing the applied cert on port 443

The script is available on my Github here.

A VCF Environment consists of multiple components, including but no limited to vCenter Server, ESXi, NSX, vSAN, the Aria Suite Products etc. Depending on the size of the VCF environment, and the number of workload domains present in it, the inventory of the components and their versions can get out-of-hand.

SDDC Manager, in its current state, is unable to dynamically query and update its inventory state information, particularly when it comes to tracking the versions of the various BoM components.

A rather easy way to update this inventory information is using the --performInventorySync option available in the Async Patch Tool.

With the announcement of End of Availability of Perpetual Licensing and SaaS services for VMware by Broadcom services and the release of VMware Cloud Foundation 5.1.1, VCF components can now use a single solution license instead of individual component licenses.

As mentioned in the VCF 5.1.1 release notes:

Solution Licensing: The component products of VCF can now derive their entitlements from a single solution license key (except for vSAN, which still requires a separate license). Individual component license keys continue to be supported. Use the vSphere Client to apply the solution license or the SDDC Manager UI to apply individual component licenses.

With the increasing growth and constant evolution of the VMware product stack, troubleshooting all the components and services involved is no simple feat. One of the most useful tools for getting a head start in the troubleshooting process is the VCF Diagnostic Tool (VDT).

VDT (developed and built by VMware Support) is a utility designed to run a series of comprehensive checks live on a target appliance. In its current state, VDT supports the vCenter Server and SDDC Manager appliances.

Depending on where its run, VDT analyzes the appliance and/or environment configuration , performing a series of checks to identify potential issues or inconsistencies. It presents results in a user-friendly format, categorizing findings as PASS, WARN, or FAIL, along with informative INFO messages. The goal is to offer real-time diagnostic information, enabling administrators to proactively address any issues that it catches.

VDT is exclusively read-only. While the tool runs several commands and API calls to gather and analyze data, it does not make any changes to the environment. This ensures that admins can rely on the diagnostic insights without worrying about unintended or unscheduled modifications. Detailed logs from each run of the utility are generated and stored within the appliance, and automatically collected as a part of each appliance support bundle.

A common error that VCF admins may encounter is the reject HostKey error. This indicates a mismatch or incorrect SSH Host Key entry stored in the SDDC Manager's known_host file(s). In this post, we'll walkthrough the process of addressing and resolving this issue.

Caused by: com.jcraft.jsch.JSchException: reject HostKey: 127.0.0.1
    at com.jcraft.jsch.Session.checkHost(Session.java:799)
    at com.jcraft.jsch.Session.connect(Session.java:345)
    at com.jcraft.jsch.Session.connect(Session.java:183)

SDDC Manager uses the jsch library, which is a pure Java implementation of SSH2. jsch allows you to connect to a sshd server and use port forwarding, file transfer, scp, sftp etc

The reject HostKey error indicates a discrepancy between the expected Host Key and the one presented during an SSH connection attempt. This can occur due to various reasons, such as system restores, upgrades, manual key changes, manual re-installs etc, leading to authentication failures. These authentication failures can affect operations and workflows throughout the VCF environment.

VMware just announced VMSA-2023-0023 with a maximum CVSSv3 base score of 9.8 - which basically implies DANGER! Here is everything you need to know about this for your VCF Environments.

VMSA-2023-0023 includes two CVEs: CVE-2023-34048, CVE-2023-34056 - The severity of these issues has been evaluated to be in the Critical severity range with a maximum CVSSv3 base score of 9.8.

This VMSA affects all versions of vCenter Server, and is only limited to vCenter, not ESXi.

vCenter does not have an in-built mechanism to export and import Global Permissions.
In this post, I will describe a script that I have written - globalPermissionManager.py - that performs this export/import operations for Global Permissions.

What is globalPermissionManager.py ?

The script is available on my GitHub here.

The script is used to export ALL Global Permissions from a vCenter Server Appliance to an object file, and then re-import them as needed. The primary use case for this, that I use it for is : Cross Domain Repoints to a new SSO.

William Lam has a PowerCLI script which contains two functions New-GlobalPermission and Remove-GlobalPermission, if we need to interact with global permissions using PowerCLI : https://williamlam.com/2017/03/automating-vsphere-global-permissions-with-powercli.html

However, in my experience, having a python script that just handles everything for you and requires no input for individual users or groups, is a lot more convenient. The script uses the same underlying API as the PowerCLI script created by William Lam.

ESXiArgs seems to be all the rage right now on the interwebz, with what (in my opinion) started as a reddit post reporting attacks, and quickly spread like wild fire to everyone with a vSphere environment talking about it. What's interesting though, is that the most common vector used to exploit this vulnerability was patched out in late 2021. Sooooo, why has this picked up in early 2023 ?

In this blog post, I am going to briefly discuss what seems to be going on with environments getting hit by ESXiArgs, why this should have never happened in the first place, and finally, prevention and remediation for ESXiArgs.

Lets start with What is ESXiArgs ?

ESXiArgs is the fancy name given to a "new" set of ransomware attacks targeting unpatched and unprotected instances of the ESXi hypervisor. Key points here being "unpatched" ESXi hosts, and scenarios where attackers have direct access to the ESXi management interfaces (for example ESXi management directly exposed to the internet, or a machine that is exposed to the internet that also has direct access to ESXi)

If you are running a VMware environment with multiple vCenters in Enhanced Linked Mode, then chances are, you have inevitably taken snapshots of vCenters and caused a replication issue due to the vmdir DBs being out of sync. In previous posts, I have talked about how this replication works, and how to not break this replication.

In this post, I will explain the quickest way to repair and recover from a broken replication state and bring all the vCenters back in sync.

VMware GSS has multiple internal scripts and KBs to repair the vmdir DB replication in place. However, there is a publicly accessible utility that also lets us repair the replication. Its called cross-domain repoint.

https://blogs.vmware.com/vsphere/2019/10/repointing-vcenter-server-to-another-sso-domain.html

Given the close integration of the SDDC Manager with all the components comprising a VCF Environment, making changes to components can be a bit of a challenge. In this article, lets talk about one of the more simpler changes - Renaming components.

As a rule, making changes to any object or component that is a part of a VCF Environment should only be done through the SDDC Manager. Making changes directly on the component itself is UNSUPPORTED - as SDDC Manager will not have visibility over this change, thereby causing a discrepancy in the inventory information.

While the changes should only be made through the SDDC Manager, there is no mechanism or workflow in place that restricts one from making the changes directly via the component itself. However, as mentioned above, the change still is unsupported. If any changes are made which cause a deviation from the inventory information that the SDDC Manager holds, this can (and will) cause issues with any number of Day-2 operations and workflows (such as adding hosts and WLDs, updates/upgrades, expanding clusters)

In terms of actually renaming components in a VCF Environment, as of VCF 4.5, here are the following components that support renaming: