PSC

If you are running a VMware environment with multiple vCenters in Enhanced Linked Mode, then chances are, you have inevitably taken snapshots of vCenters and caused a replication issue due to the vmdir DBs being out of sync. In previous posts, I have talked about how this replication works, and how to not break this replication.

In this post, I will explain the quickest way to repair and recover from a broken replication state and bring all the vCenters back in sync.

VMware GSS has multiple internal scripts and KBs to repair the vmdir DB replication in place. However, there is a publicly accessible utility that also lets us repair the replication. Its called cross-domain repoint.

https://blogs.vmware.com/vsphere/2019/10/repointing-vcenter-server-to-another-sso-domain.html

As discussed in this blog post, PSC replication primarily involves the vmdird - VMware Directory Service.

This VMware Directory Service provides a multitenant, peer-replicating LDAP directory service that stores authentication, certificate, lookup, and license information. If your domain contains more than one PSC or embedded VC instance, an update of vmdir content in one vmdir instance is propagated to all other instances of vmdir.

All of this vmdir information is stored in a data.mdb file. This data.mdb file and its contents are essentially what are replicated.

The size of this file is usually about 15-20MB per node – it really should not be beyond 150-200 MB in 99% of the cases.

How Replication is Broken

There are two ways that we see vmdird replication breaking:

A platform services controller, or a PSC (which can be an external appliance, or embedded into VC) handles vSphere single sign-on (SSO), licensing, tagging, global permissions, custom roles, and certificate management. More Info:

https://blogs.vmware.com/vsphere/2017/10/platform-services-controller-psc-6-x-faq-now-available.html

We often read about PSCs replicating between each other, but more often than not, it is unclear WHAT exactly is being replicated between the PSCs. When you deploy multiple PSCs (or vCenters with embedded PSCs) in the same SSO domain, the PSCs will be replicating VMDIR data with each other, depending on how replication is configured.

Based on VMware documentation available here:

 https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.psc.doc/GUID-FE4E0496-A14C-4331-A7D6-1200F7C068A5.html 

VMDIR data includes: authentication, certificate, lookup, and license information. If your domain contains more than one Platform Services Controller instance, an update of VMDIR content in one VMDIR instance is propagated to all other instances of VMDIR (i.e the other PSCs or embedded VCs).

In this example, we are using 3x 6.7 PSCs, all part of the same "vsphere.local" domain.