SSL peer certificate validation failed after renewing certificate

Kaspars_K · 2024-09-04T19:48:06.620Z

Hi, I am building and Managing multiple MongoDB environments and I have ran into some odd issue with “expired” internal SSL certificates for MongoDB servers.

Back in May our internal SSL Certificates were about to expire, and I ordered these certificates to be renewed for 2 of our Environments(environment A and Environment B).
Certificates were renewed and were applied to both environments. Everything seemed to be working just fine, until I noticed in July that in MongoDB Environment B we get following error messages in our logs:

{“t”:{“$date”:“2024-09-03T15:40:05.038+03:00”},“s”:“W”, “c”:“NETWORK”, “id”:23222, “ctx”:“conn18”,“msg”:“Peer certificate expiration information”,“attr”:{“peerSubjectName”:“CN=hostname1”,“daysDays”:-103}}
{“t”:{“$date”:“2024-09-03T15:40:05.038+03:00”},“s”:“E”, “c”:“NETWORK”, “id”:23279, “ctx”:“conn18”,“msg”:“SSL peer certificate validation failed”,“attr”:{“errorCode”:“800B0101”,“error”:“A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.”}}

Oddly enough there were no issues in MongoDB Environment A, also both environments work fine, there are no startup issues, its just in MongoDB Environment B we get spammed by these error messages in logs, yet every thing works just fine.
In the error message this line sticks out “daysDays”:-103, which is a direct countdown to the day our old certificates expired. It almost feels like Mongo is looking at old certificates instead of new ones, but I can see in the logs that new certificates are being used with different thumbprint and expiration dates etc.

I have tried following things to resolve this:
Ordered completely new certificates with small changes to them, but the same error message persists.
Deleted both db and MongoDB from server and reinstalled it(This environment is in the build phase no important data was stored here yet)
Changed replica set name.
Compared settings/certificates/ time zones for MongoDB Environment A and MongoDB Environment B.

Setup:

MongoDB Environment A:

MongoDB EA 6.0.14
3 Windows machines(VMs in Azure) (1 Primary, 1 Secondary, 1 Arbiter)

MongoDB Environment B:

MongoDB EA 6.0.14
3 Windows machines(VMs in Azure) (1 Primary, 2 Secondaries)

We are using our internal SSL certificates, they are being signed by our own CA(Root CA + Signing Cert).

Certificates setup.

Previous Certificates(each machine had its own cert):

CN=hostname1
SAN: hostname1, hostname1_IP_Address
Extended Key usage: Client Authentication; Server Authentication

New Certificates(1 Cert that contains SAN for all 3 machines):

CN=MongoDB
SAN: FQDN1, FQDN2, FQDN3, hostname1, hostname2, hostname3
Extended Key usage: Client Authentication; Server Authentication

Mongodb Config:

tls:
mode: requireTLS
certificateKeyFile: C:\Cert\ServerCert.pem
CAFile: C:\Cert\CA-cert.pem

ServerCert.pem contains Server Cert + unencrypted private key(only way for windows)
CA-Cert.pem contains RootCA and Signing CA

I have ran out of ideas on how to solve this, perhaps someone knows something or can point me in a direction that might provide some answers.
In our logs I can see that the correct certificates are being used when MongoDB server starts/restarts.

Kaspars_K · 2024-10-24T12:08:35.057Z

Issue was resolved.

My colleague had installed Telegraf for monitoring purposes and since certificates for that app were stored somewhere else, I did not notice they were expired, since Mongodb logs didn’t really point towards that application, I had no idea we had expired certificates on the same server.

Thankfully once certificates for the Telegraf application were updated this issue resolved itself, and no longer Mongodb logs are being populated by the certificates expired error message