A common error that VCF admins may encounter is the reject HostKey error. This indicates a mismatch or incorrect SSH Host Key entry stored in the SDDC Manager's known_host file(s). In this post, we'll walkthrough the process of addressing and resolving this issue.
Caused by: com.jcraft.jsch.JSchException: reject HostKey: 127.0.0.1
at com.jcraft.jsch.Session.checkHost(Session.java:799)
at com.jcraft.jsch.Session.connect(Session.java:345)
at com.jcraft.jsch.Session.connect(Session.java:183)SDDC Manager uses the jsch library, which is a pure Java implementation of SSH2. jsch allows you to connect to a sshd server and use port forwarding, file transfer, scp, sftp etc
The reject HostKey error indicates a discrepancy between the expected Host Key and the one presented during an SSH connection attempt. This can occur due to various reasons, such as system restores, upgrades, manual key changes, manual re-installs etc, leading to authentication failures. These authentication failures can affect operations and workflows throughout the VCF environment.
The resolution to the issue is simple enough - we need to update the bad Host Key entries in the known_hosts file on the SDDC Manager. That being said, there are multiple known_hosts files on the SDDC Manager, and different workflows check the Host Keys against different files. Additionally, a particular component may present multiple Host Keys with different algorithms - making the process of manually updating these multiple entries across multiple files fairly cumbersome.
fix_known_hosts.sh
Fortunately, there is a script available to remediate and update all Host Keys for a given component across all the known_hosts files on an SDDC Manager. The fix_known_hosts.sh script is available here.
It is strongly recommended to take a snapshot of the SDDC Manager before proceeding with any changes.
Download the fix_known_hosts.sh script, and upload it to the SDDC Manager
Update the file with executable permissions: chmod +x fix_known_hosts.sh
Or run the file directly with bash: /bin/bash fix_known_hosts.sh
The script will prompt to input the FQDN and IP address of the node for which we need to update the Host Keys.
The script should delete any existing Host Key entries for the FQDN and IP Address provided, and add new ones by running the ssh-keyscan command against them.
Certain caveats to keep in mind:
- The script assumes that SSH is enabled for the node in question.
- The script will not work if FIPs is enabled on the nodes in the VCF Environment.