A recent post from VMware talked about a new malware for ESXi, based on information published by Mandiant - which is a cyber-security firm (a subsidiary of Google). You can read the VMware KB here.
The full posts from Mandiant can be found here:
https://www.mandiant.com/resources/blog/esxi-hypervisors-malware-persistence
https://www.mandiant.com/resources/blog/esxi-hypervisors-detection-hardening
In this post, I am going to discuss the key aspects of this issue.
What are Mandiant's findings?
Mandiant found malware on ESXi hosts that was basically installed using unsigned VIBs. The unsigned VIBs contain backdoors which then compromise the ESXi host. At that point, anything on the host can be considered as compromised - commands can be sent for execution on Guest VMs, files can be transferred between ESXi and the Guest VMs etc.
What is an unsigned VIB?
A VIB or vSphere Installation Bundle is basically a software package used to install components on an ESXi hosts.
You can read more about it here: What's in a VIB?
As mentioned in that post, VIBs have the following acceptance levels:
- VMwareCertified: VIBs created and tested by VMware. VMware Certified VIBs undergo thorough testing by VMware.
- VMwareAccepted: VIBs created by a VMware partners that are approved by VMware. VMware relies on partners to perform the testing, but VMware verifies the results.
- PartnerSupported: VIBs created and tested by a trusted VMware partner. The partner performs all testing. VMware does not verify the results.
- CommunitySupported: VIBs created by individuals or partners outside of the VMware partner program. These VIBs do not undergo any VMware or trusted partner testing and are not supported by VMware or its partners.
An unsigned VIB is not signed by VMware or its partners - and therefore is not supported, vetted, or guaranteed by any of these parties.
VMware does not recommend using or installing unsigned VIBs on ESXi hosts.
The default acceptance level for installation of VIBs on an ESXi host is PartnerSupported. [Reference]
We can check the acceptance level on a host by running the following command:
esxcli software acceptance get
We can also change this acceptance level using the following command:
esxcli software acceptance set --level=<acceptance_level>
For example:
esxcli software acceptance set --level=VMwareAccepted
While there can be unsigned VIBs that are of a higher acceptance level, the vast majority of unsigned VIBs will likely be of the CommunitySupported acceptance level.
To check if a VIB is signed or unsigned, we can run the following command:
esxcli software vib signature verify
If the Signature Verification fails for a VIB - it is unsigned.
How does an unsigned VIB get installed?
To be able to install an unsigned VIB on a host, you need to have root access.
A VIB, any VIB, whether signed or unsigned, has to installed on the host via root credentials for the host.
The command for installing a VIB directly is:
esxcli software vib install -v <path_to_VIB>To be able to install a CommunitySupported VIB, we will first need to change the acceptance level on the host.
However the acceptance level can be set to CommunitySupported, only if Secure Boot is disabled:
Additionally, the install of the VIB will likely be forced using the --force flag:
esxcli software vib install -v <path_to_VIB> --forceSo to be able to install and run an unsigned VIB on an ESXi host, an entity has to have:
- root access to the ESXi host
- Secure Boot disabled on the host
- Acceptance level set to the lowest - CommunitySupported
What is the CVE associated with this?
There isn't one. Why? Because no ESXi vulnerability is being exploited here to be able to install the VIB. Mandiant has not found any evidence of an ESXi exploit or vulnerability being leveraged to gain root access to ESXi or install the VIBs.
How do I fix this ?
If you have found and isolated a VIB that is determined to be the malware - then you need to uninstall the VIB.
VIBs can be uninstalled using the following command [Reference]:
esxcli software vib remove --vibname-<vibname>But that just deals with the immediate issue. It does not address the bigger concern. The only way this VIB was installed was if someone already had complete administrative access (root) of the ESXi host. Until that is resolved, and the we have ensured that access to the host is secured, uninstalling the VIB doesn't do much. If the entity still has root access, they can just re-install the VIB again. and again - till that host is re-secured.
How do I mitigate against this?
Enable Secure Boot on the ESXi hosts.
With secure boot enabled, unsigned VIBs cannot be installed on the host, and even if they are installed, the ESXi host will not boot up if a reboot is performed.
Check for presence of unsigned VIBs on existing hosts.
We can check for VIB signatures using the command listed above. Since that has to be on each host, VMware has made a script available on this KB to connect to the vCenter via PowerCLI and scan all hosts in the environment to determine what hosts have unsigned VIBs installed.
Secure access to the environment
Since this malware fundamentally depends on having root access to the ESXi hosts, ensure that access to the environment is secured, controlled and restricted. Ensure to have all products patched to the latest versions available to minimize the risk of exploitation via other products.
I have unsigned VIBs! Are my hosts compromised?
Not exactly. The presence of unsigned VIBs on hosts does not equate to the presence of the malware installed on the hosts. You need to track what was the origin of the unsigned VIBs - where they came from, what do they contain, why were they installed, and who installed them. It is entirely possible that an unsigned VIB was installed by a trusted administrator for a genuine reason.
However, it does come down to being able to identify and account for each unsigned VIB.
If there is a VIB is not accounted for or is completely unknown, then you need to determine its source and take appropriate actions to based on your incident response protocol, and treat it as a potential compromise.