As previously established, I do want to maintain correct certificates for my infrastructure here. When I installed VMWare vCenter Server it created a new interface to work with. That interface actually generates and maintains its own set of certificates. I wanted to get at least the vCenter Server web interface to use my own custom certificates to maintain at least what I had before on the individual ESXi box. I honestly thought this would be relatively easy at first. It had a web GUI for actually changing custom certificates. I should have known better that it would never be that easy.
I started by navigating to administration.
Then I navigated to the Certificate Management section.
Like some other end clients, it had a section for trusted root certificates. So I wanted to add my step-ca root certificate. I clicked on the Add.
Browsed to and selected the root_ca.crt (which Is kept on my Certificates Share). Then Clicked Add.
Now my new root certificate was present. I clicked on Actions for the Machine SSL Certificate and then Import and Replace Certificate.
Then I selected Replace with external CA certificate(requires private key). This looks exactly like what some of the more manual processes were.
Next I added the cert.pem in the Machine SSL certificate, the fullchain.pem in the Chain of trusted root certificates, and the privkey.pem in the Private Key. Remember that all of my certificates are based on how letsencrypt works. This appears like it would be the correct set.
And it told me I had a Common Name issue.
This was annoying. I knew from the previous work on this that the common name is *.internal . It was a bit weird the wildcard didn’t work, but I did know how to generate a certificate for just vcenter.internal. I did that and attempted to import it again.
I don’t know what that means. Even Google couldn’t help me on that one. Eventually, I gave up and searched for another method to actually import a certificate. I did some searching, and many were not recommending I do this. It seems like they say I should just download the vCenter root certificate and move on. I really didn’t like that, as I have already built up a certificate infrastructure, and don’t want to maintain two just for the sake of it.
I found a couple articles on this issue [1] [2]. The second being the far more useful one. First I copied the certificates over to the vCenter VM from the issuing ca1.internal vm:
scp /etc/letsencrypt/live/vcenter.internal/* root@vcenter.internal:/root/*Then I ssh’d over to the vCenter machine. I needed to use the local root login, not the administrator@vsphere.internal login. Next I selected the shell.
Last login: Tue Dec 15 06:02:25 2020 from 192.168.2.156
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"
Command> shell
Shell access is granted to rootThen I ran the certificate-manager.
/usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.8 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| NOTE: Solution user certs will be deprecated in a future |
| release of vCenter. Refer to release notes for more details.|
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 1
Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@vsphere.internal
Enter password:
1. Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate
2. Import custom certificate(s) and key(s) to replace existing Machine SSL certificate
Option [1 or 2]: 2
Please provide valid custom certificate for Machine SSL.
File : /root/certs/cert.pem
Please provide valid custom key for Machine SSL.
File : /root/certs/privkey.pem
Please provide the signing certificate of the Machine SSL certificate
File : /root/certs/chain.pem
You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : y
Command Output: /root/certs/cert.pem: OK
Get site nameCompleted [Replacing Machine SSL Cert...]
default-site
Lookup all services
It listed all the services as it replaced the certificates everywhere. Then it succeeded. I went back and checked the web interface.
Failed. It looks like it didn’t report the chain. I did actually re-step through that process a couple of times, but the entire machine kept adding more certificates. I looked into deleting certificates [3]. Honestly, I couldn’t find out how to do this. It doesn’t look like anyone ever considered just removing a certificate! I ended up having to give up and reinstall a clean vCenter VM to actually clear the certificates.
I also learned something useful along the way. vCenter maintains a certificate chain among all of the ESXi hosts.
It overrode what I did. I was actually able to fix this, but I did mess it up the first time, replacing only the certificate and not the private key. I then had to learn how to boot ESXi into a barely running mode, re-generate a set of self-signed certificates which actually work together, restart all the services; then I was able to reassert control over everything [4] [5].
This is what everyone meant about maintaining its own set of certificates. This is for vCenter and ESXi hosts to communicate securely among themselves. It is highly recommended to not attempt to replace this system. VMWare does a good job here, but changing the management system for them is not easy. It is okay to change just the vCenter Machine certificate though, as that can work, without the need to actually go back and change anything it is managing. I can still manage the ESXi webhost directly from the IP address.
In any event. I reinstalled a blank vCenter Server and tried again. Since it appeared that the first Machine SSL was just echoed back, I decided to see if I just gave it the fullchain.pem there would echo back the entire chain now including the intermediate certificate.
I did forget to add the root certificate at first.
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"
Command> shell
Shell access is granted to root
#ls
certs
#/usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.8 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| NOTE: Solution user certs will be deprecated in a future |
| release of vCenter. Refer to release notes for more details.|
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 1
Please provide valid SSO and VC privileged user credential to perform certificat e operations.
Enter username [Administrator@vsphere.local]:administrator@vsphere.internal
Enter password:
1. Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate
2. Import custom certificate(s) and key(s) to replace existing Machine SSL certificate
Option [1 or 2]: 2
Please provide valid custom certificate for Machine SSL.
File : /root/certs/fullchain.pem
Please provide valid custom key for Machine SSL.
File : /root/certs/privkey.pem
Please provide the signing certificate of the Machine SSL certificate
File : /root/certs/fullchain.pem
You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : y
/root/certs/fullchain.pem: CN = Smallstep Intermediate CA
error 2 at 1 depth lookup:unable to get issuer certificate
Error in verifying certificate: /root/certs/fullchain.pemSo I had to pop back and re-add the step-ca root certificate.
And tried again.
/usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.8 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| NOTE: Solution user certs will be deprecated in a future |
| release of vCenter. Refer to release notes for more details.|
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 1
Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@vsphere.internal
Enter password:
1. Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate
2. Import custom certificate(s) and key(s) to replace existing Machine SSL certificate
Option [1 or 2]: 2
Please provide valid custom certificate for Machine SSL.
File : /root/certs/fullchain.pem
Please provide valid custom key for Machine SSL.
File : /root/certs/privkey.pem
Please provide the signing certificate of the Machine SSL certificate
File : /root/certs/fullchain.pem
You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : y
Command Output: /root/certs/fullchain.pem: OK
Get site nameCompleted [Replacing Machine SSL Cert...]
default-site
Lookup all services
...
Updated 42 service(s)
Status : 100% Completed [All tasks completed successfully]
Which worked.
Success!
[1] https://kb.vmware.com/s/article/2112277
[2] https://jjasghar.github.io/blog/2017/11/14/vcenter-vcsa-and-using-lets-encrypt/
[4] https://docs.vmware.com/en/VMware-vSphere/5.5/com.vmware.vsphere.security.doc/GUID-EA0587C7-5151-40B4-88F0-C341E6B1F8D0.html[5] https://communities.vmware.com/t5/VMware-vCenter-Discussions/ESXI-Web-UI-not-working/m-p/490832