VMware

A recent post from VMware talked about a new malware for ESXi, based on information published by Mandiant - which is a cyber-security firm (a subsidiary of Google). You can read the VMware KB here.

The full posts from Mandiant can be found here:
https://www.mandiant.com/resources/blog/esxi-hypervisors-malware-persistence
https://www.mandiant.com/resources/blog/esxi-hypervisors-detection-hardening

In this post, I am going to discuss the key aspects of this issue.

What are Mandiant's findings?

Mandiant found malware on ESXi hosts that was basically installed using unsigned VIBs. The unsigned VIBs contain backdoors which then compromise the ESXi host. At that point, anything on the host can be considered as compromised - commands can be sent for execution on Guest VMs, files can be transferred between ESXi and the Guest VMs etc.

Since the release of VCF 4.4.0.0, there has been a lot of chatter about how we can decouple or disassociate the vRealize suite from VCF and SDDC Manager, or how it is completely externally managed.
This is quite incorrect and stems from a misunderstanding of how the vRealize Suite is linked to VCF.

In this post, I will talk about how the vRealize Suite is linked to VCF and SDDC manager, and what has changed in VCF 4.4 and above.

In February of 2022, I attempted and passed the exam for the VCAP-DCV Design 2022. This was my second attempt at a VCAP Design exam, with the first one being the VCAP-DCV 6.5 Design. In this blog post, I will talk about my preparation for the exam, what I focused on, what my experience was like, and some exam tips.

The VMware Certified Advanced Professional - Data Center Virtualization - Design 2022 (quite a mouthful, isn't it?), more often known as the VCAP-DCV 2022 Design certification, as per VMware, validates that you have advanced knowledge of end-user computing environments and components, and are able to recommend and design VMware solutions to meet specific goals and requirements.

Lets break this down further.

During upgrades of VCF components from the SDDC Manager, we often run into situations where a component upgrade bundle does not show up for a particular VCF version upgrade. This can also present itself as skipping a component through the upgrade process. In this article, I will explain what compatibility sets are, how they work, and how they affect the VCF component upgrade process.

What are Compatibility Sets?

Compatibility sets are set of version entries for the 3 primary core VCF products - namely vCenter, ESXi and NSX(T/V), which are marked as compatible with one another. Lets break it down further.

Each entry in the compatibility sets contains one version each for the vCenter, ESXi and NSX component.

For example, from the screenshot above, the final entry indicates that vCenter version 7.0.3.00300, ESXi version 7.0.3, and NSX-T version 3.1.3.7.4 are basically marked as compatible to work with each other in a VCF environment.

The VerionAlias.yml file on a SDDC Manager is referenced quite a bit across several VMware KBs, mostly in the context of "change this value here" or "add this value there" while troubleshooting VCF upgrades. There really aren't any articles online describing what this file really means or how it works. In this post, I will describe the various elements that are in a VersionAlias.yml file, and how to interpret the information available here.

Lets start with the different versions of each VCF component that we are dealing with:

HyTrust (now Entrust) KeyControl is a Key Management Server (KMS) that essentially manages encryption keys for virtual machines, including their rotation, sharing, access etc.
The reason I chose this KMS for use with vCenter is essentially due to the availability of a 60 day trial, which then let me try all the encryption options available within vCenter 6.7

To start off, I downloaded the ISO and uploaded it to a datastore that my ESXi hosts can access. I created a new VM, in this case I called it HyTrust_KeyControl_Test, with the following configuration:

As discussed in this blog post, PSC replication primarily involves the vmdird - VMware Directory Service.

This VMware Directory Service provides a multitenant, peer-replicating LDAP directory service that stores authentication, certificate, lookup, and license information. If your domain contains more than one PSC or embedded VC instance, an update of vmdir content in one vmdir instance is propagated to all other instances of vmdir.

All of this vmdir information is stored in a data.mdb file. This data.mdb file and its contents are essentially what are replicated.

The size of this file is usually about 15-20MB per node – it really should not be beyond 150-200 MB in 99% of the cases.

How Replication is Broken

There are two ways that we see vmdird replication breaking:

In our previous posts, we configured a KMS to use with a vCenter, and configured a VM storage policy for encryption.

Here, we are going to encrypt a VM using a VM encryption storage policy.
To start, we need to login to the vCenter vSphere Client and go to the Hosts and Clusters section.

In this example, I am going to encrypt the "RHEL7_1" VM.
To encrypt an existing VM, we are going to have to change its storage policy from its current one, to a VM Storage Policy for Encryption. The VM does need to be powered off for this operation, since we are changing the storage policy affiliated with the VMDKs.

The simplest way to run through this is to Right Click on the VM in question > VM Policies > Edit VM Storage Policies

Virtual machine storage policies control which type of storage is provided for the virtual machine and how the virtual machine is placed within storage. They also determine data services that the virtual machine can use.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.storage.doc/GUID-A8BA9141-31F1-4555-A554-4B5B04D75E54.html

In our previous posts, we installed a KMS and configured it for use with vCenter. Here, we are going to configure a new VM Storage Policy to use for encryption.

We start by heading to Policies and Profiles, by heading to Menu > Policies and Profiles
In there, we are going to select VM Storage Policies and then Create VM Storage Policy.

A platform services controller, or a PSC (which can be an external appliance, or embedded into VC) handles vSphere single sign-on (SSO), licensing, tagging, global permissions, custom roles, and certificate management. More Info:

https://blogs.vmware.com/vsphere/2017/10/platform-services-controller-psc-6-x-faq-now-available.html

We often read about PSCs replicating between each other, but more often than not, it is unclear WHAT exactly is being replicated between the PSCs. When you deploy multiple PSCs (or vCenters with embedded PSCs) in the same SSO domain, the PSCs will be replicating VMDIR data with each other, depending on how replication is configured.

Based on VMware documentation available here:

 https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.psc.doc/GUID-FE4E0496-A14C-4331-A7D6-1200F7C068A5.html 

VMDIR data includes: authentication, certificate, lookup, and license information. If your domain contains more than one Platform Services Controller instance, an update of VMDIR content in one VMDIR instance is propagated to all other instances of VMDIR (i.e the other PSCs or embedded VCs).

In this example, we are using 3x 6.7 PSCs, all part of the same "vsphere.local" domain.