I suddenly became unable to connect to mongodb with shell and unable to dump db

Ghrib_Ahmed · 2021-06-22T10:20:45.288Z

Hello everyone,
I have been using mongoDb for quite a while.
And only a few days ago I was able to dump mongodb.

I usually dump my mongodb database using this command and it works perfectly:

mongodump --uri mongodb+srv://name:password@cluster0.fklgt.mongodb.net/database_name --archive="dump-copy-name" --forceTableScan

However, I have been trying to dump different databases and it just keeps logging this:

2021-06-21T18:43:39.206+0100 error dialing cluster0-shard-00-00.fklgt.mongodb.net:27017: SSL errors: SSL routines:ssl3_get_server_certificate:certificate verify failed

I went and checked whether I am connected to the database using mongo command. And it turns-out I wasn’t. So I tried to connect again using this:

mongo "mongodb+srv://cluster0.fklgt.mongodb.net/MY_DATA_BASE_NAME" --username MY_USER_NAME

But, I keep getting a similar error:

2021-06-22T11:01:08.849+0100 E NETWORK [thread1] SSL peer certificate validation failed: unable to get local issuer certificate

I find this quite strange. Everything was working fine. And out of nowhere this happened.

Any idea what’s going-on?

Jason_Tran · 2021-06-22T11:40:33.962Z

Hi @Ghrib_Ahmed,

Welcome to the community!

Based off the sudden failures of mongodump (as well as mongo connection attempts) in addition to the SSL errors, it’s possible that this may be caused by the TLS certificates changes for your cluster(s) which could have recently been renewed to the ISRG Root X1 certificate, as required by the upcoming change being made by Let’s Encrypt, the authority which provides the TLS certificates for Atlas clusters.

There are a few things you can try:

Reverting to the DST Root CA X3 and performing another mongodump to see if the errors are still present.
Using the latest versions of mongodump and mongo (if not already done so).

Would you also be able to also provide the versions of mongodump and mongo you used when the errors were being returned? This can be done by running mongodump --version and mongo --version.

Kind Regards,
Jason

Ghrib_Ahmed · 2021-06-22T14:20:31.045Z

Jason_Tran:

mongo --version

Hello these are the requested information:

mongodump --version
mongodump version: r3.6.3
git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
Go version: go1.7
os: linux
arch: amd64
compiler: gc
OpenSSL version: OpenSSL 1.0.1 14 Mar 2012

mongo --version
MongoDB shell version v3.6.3
git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1204
distarch: x86_64
target_arch: x86_64

Jason_Tran · 2021-06-23T06:13:09.423Z

Thanks @Ghrib_Ahmed,

From looking at the hostnames from the error logs you’ve provided initially it appears this would be an Atlas cluster. Please correct me if I am wrong here. The Atlas clusters should have a minimum MongoDB version of 4.0 which is a bit ahead of the mongodump and mongo commands being run.

I would recommend trying either of the following if you haven’t already done so for troubleshooting purposes:

Reverting to the DST Root CA X3 and performing another mongodump to see if the errors are still present.
Using the latest versions of mongodump and mongo. You can find the latest version of the MongoDB Database tools here. You can find the mongo shell within the bin folder after downloading and extracting the corresponding version of MongoDB community server.

After trying the above, please send any errors you receive when trying to perform mongodump and mongo shell connection attempts again.

Hope this helps.

Kind Regards,
Jason

Ghrib_Ahmed · 2021-06-25T09:30:59.500Z

Hello again,

I’m really not that comfortable with Linux file manipulations and found the documentation rather daunting.

So I tried to update mongo using this command:

sudo apt update

All other packages were updated successfully, except for mongo. I had this logged:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http:// repo.mongodb .org/apt/ubuntu trusty/mongodb-org/3.2 Release: The following signatures were invalid: KEYEXPIRED 1570654450
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https:// repo.mongodb .org/apt/ubuntu precise/mongodb-org/3.6 Release: The following signatures were invalid: KEYEXPIRED 1544811256
W: Failed to fetch http:// repo.mongodb. org/apt/ubuntu/dists/trusty/mongodb-org/3.2/Release.gpg The following signatures were invalid: KEYEXPIRED 1570654450
W: Failed to fetch https:// repo.mongodb .org/apt/ubuntu/dists/precise/mongodb-org/3.6/Release.gpg The following signatures were invalid: KEYEXPIRED 1544811256
W: Some index files failed to download. They have been ignored, or old ones used instead.

So I tried to do a normal installation using this command:

sudo apt install mongodb

But, when I ran this command:

sudo systemctl status mongodb

I got this:

● mongodb.service - High-performance, schema-free document-oriented database
Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2021-06-25 10:06:32 CET; 1min 29s ago
Main PID: 13502 (code=exited, status=2)

Jun 25 10:06:32 ahmed-Aspire-E5-573G systemd[1]: Started High-performance, schema-free document-oriented database.
Jun 25 10:06:32 ahmed-Aspire-E5-573G mongod[13502]: Unrecognized option: processManagement.timeZoneInfo
Jun 25 10:06:32 ahmed-Aspire-E5-573G mongod[13502]: try ‘/usr/bin/mongod --help’ for more information
Jun 25 10:06:32 ahmed-Aspire-E5-573G systemd[1]: mongodb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 25 10:06:32 ahmed-Aspire-E5-573G systemd[1]: mongodb.service: Unit entered failed state.
Jun 25 10:06:32 ahmed-Aspire-E5-573G systemd[1]: mongodb.service: Failed with result ‘exit-code’.

Strangely enough, mongo and mongodump version reverted to a previous version:

mongodump --version
version 2.6.10

mongo --version
MongoDB shell version: 2.6.10

Also when I run mongo I get this:
MongoDB shell version: 2.6.10

connecting to: test
2021-06-25T10:31:42.628+0100 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2021-06-25T10:31:42.628+0100 Error: couldn’t connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
exception: connect failed

So my question is, is there a way to do the update without having to do a manual file manipulation?
In other words, just run a simple command?

Jason_Tran · 2021-06-25T09:55:40.056Z

Thanks for providing that information @Ghrib_Ahmed. Seems the mongo and mongodump reverting to a previous version is a seperate issue to the one originally reported here.

So my question is, is there a way to do the update without having to do a manual file manipulation?
In other words, just run a simple command?

Have you tried downloading the up to date mongo and mongodump tools and run them directly from their bin folders? This way you won’t have to manually manipulate any files or configurations. It would just be for troubleshooting connectivity using the more up to date version of the tools.

For example, I have mongo version 4.4.6 added to PATH within my environment. However, I can run mongo version 4.4.4 directly by running it from the bin folder within the mongodb version 4.4.4 installation folder shown below:

07:44 pm JT@MacbookPro [~] $mongo --version
MongoDB shell version v4.4.6
Build Info: {
    "version": "4.4.6",
    "gitVersion": "72e66213c2c3eab37d9358d5e78ad7f5c1d0d0d7",
    "modules": [],
    "allocator": "system",
    "environment": {
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}
07:44 pm JT@MacbookPro [~] $./mongodb-macos-x86_64-4.4.4/bin/mongo --version
MongoDB shell version v4.4.4
Build Info: {
    "version": "4.4.4",
    "gitVersion": "8db30a63db1a9d84bdcad0c83369623f708e0397",
    "modules": [],
    "allocator": "system",
    "environment": {
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

You can try downloading the corresponding version of MongoDB community server which matches the MongoDB version of the cluster you’re attempting to connect to. Once downloaded and unpacked, you should find mongo within the bin folder. You can then run the newer version of mongo directly by using a command such as below:

$ <MongoDB Bin Folder Path>/mongo "mongodb+srv://cluster0.fklgt.mongodb.net/MY_DATA_BASE_NAME" --username MY_USER_NAME

Where is the full path of the bin folder within the unpacked MongoDB server file.

In addition to this, are you able to provide what operating system + OS version you’re attempting to connect from?

Lastly, did you try to attempt reverting back to DST Root CA X3 to see if mongodump and mongo connected without SSL errors?

Jason

Ghrib_Ahmed · 2021-06-25T10:59:35.538Z

First, I’d like to thank you for the detailed response. Here’s what I did now:
1. TRYING TO USE THE LATEST VERSION OF MONGO

I have downloaded the mongoDb community server, and I only find this file in /bin folder:

So I didn’t find the mongo file.
I tried running mongod anyway, but this is what I got:

~/Downloads/mongodb-org-server-4.4.6-1.amzn2.aarch64/usr/bin $ ./mongod
bash: ./mongod: cannot execute binary file: Exec format error

./mongod --version
bash: ./mongod: cannot execute binary file: Exec format error

2. TRYING TO USE THE LATEST VERSION OF MONGO-DUMP

I downloaded the mongodb database tools
I tried to do a mongo-dump:

/mongodb-database-tools-amazon-x86_64-100.3.1/bin $ ./mongodump --uri ‘mongodb+srv://ACCOUNT_NAME:PASSWORD@cluster0.fklgt.mongodb.net/DATABASE_NAME’ --archive=“mongodump-prod-db” --forceTableScan
2021-06-25T11:51:26.829+0100 Failed: can’t create session: could not connect to server: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: cluster0-shard-00-00. fklgt.mongodb. net :27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : x509: certificate signed by unknown authority }, { Addr: cluster0-shard-00-01.fklgt.mongodb. net: 27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : x509: certificate signed by unknown authority }, { Addr: cluster0-shard-00-02.fklgt.mongodb. net: 27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : x509: certificate signed by unknown authority }, ] }

~/Downloads/mongodb-database-tools-amazon-x86_64-100.3.1/bin $ ./mongodump --version
mongodump version: 100.3.1
git version: 32632b931f9c41d8314b75ecc88e551b012b1e30
Go version: go1.15.8
os: linux
arch: amd64
compiler: gc

3. SYSTEM INFO

Operating System: Linux Mint 18.2
Kernel: Linux 4.8.0-53-generic
Architecture: x86-64

4. REVERTING BACK TO DST Root CA X3
Honestly, I didn’t try this. I decided to leave it as a last resort if nothing else works because it seems to me too complicate it and may complicate things even more.

Thank you.

Ghrib_Ahmed · 2021-06-25T11:18:11.674Z

On a side note, a colleague of mine has mongodump v100.3.1 and mongo v4.4.4 and he was capable of successfully dumping the db.

So I suppose deleting everything and installing it from scratch should solve my problems.

Jason_Tran · 2021-06-25T11:41:02.706Z

I have downloaded the mongoDb community server, and I only find this file in /bin folder:

Apologies, you may have to change the package on the download page to shell:

On a side note, a colleague of mine has mongodump v100.3.1 and mongo v4.4.4 and he was capable of successfully dumping the db.

Glad to hear it worked for your colleague. If you do end up finding the solution regarding your own system (whether re-installation solved it), please update this post here.

Apologies I was not able to help further.

Kind Regards,
Jason

Ghrib_Ahmed · 2021-06-25T13:02:23.238Z

Hello again,

I hope I can continue the conversation because I finally managed to do what you asked me.
I downloaded the files and tested them:

This is the result for mongo command:

/mongodb-linux-x86_64-ubuntu1604-4.4.6/bin $ ./mongo “mongodb+srv://cluster0.fklgt.mongodb.net/DATABASENAME” --username USERNAME
MongoDB shell version v4.4.6
Enter password:
connecting to: mongodb://cluster0-shard-00-00.fklgt.mongodb.net:27017,cluster0-shard-00-01.fklgt.mongodb. net:27017,cluster0-shard-00-02.fklgt .mongodb.net:27017/ %3Cdbname%3E?authSource=admin&compressors=disabled&gssapiServiceName=mongodb&replicaSet=atlas-zuxkh7-shard-0&ssl=true
{“t”:{“$date”:“2021-06-25T12:56:36.945Z”},“s”:“E”, “c”:“NETWORK”, “id”:23256, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“SSL peer certificate validation failed”,“attr”:{“error”:“SSL peer certificate validation failed: unable to get local issuer certificate”}}
{“t”:{“$date”:“2021-06-25T12:56:36.945Z”},“s”:“E”, “c”:“NETWORK”, “id”:23256, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“SSL peer certificate validation failed”,“attr”:{“error”:“SSL peer certificate validation failed: unable to get local issuer certificate”}}

This is the result for the mongodump command:

/mongodb-database-tools-ubuntu1604-x86_64-100.3.1/bin $ ./mongodump --uri mongodb+srv://USER_NAME@cluster0.fklgt.mongodb.net/DATABASE_NAME --archive=“mongodump-db” --forceTableScan
2021-06-25T13:52:54.986+0100 Failed: can’t create session: could not connect to server: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: cluster0-shard-00-02.fklgt. mongodb .net:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : x509: certificate signed by unknown authority }, { Addr: cluster0-shard-00-00.fklgt. mongodb .net:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : x509: certificate signed by unknown authority }, { Addr: cluster0-shard-00-01.fklgt. mongodb .net:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: i/o timeout }, ] }

I find it strange that my colleague managed to the dump and connect and I didn’t even with the latest versions of the commands.

Jason_Tran · 2021-06-27T09:31:36.052Z

Hi @Ghrib_Ahmed,

Have you compared certificate stores of your own system to that of your colleague?

4. REVERTING BACK TO DST Root CA X3
Honestly, I didn’t try this. I decided to leave it as a last resort if nothing else works because it seems to me too complicate it and may complicate things even more.

There are some instructions on how to do this linked in my previous comment which are straight forward and can be easily reverted using the same steps by switching back to ISRG Root X1. All of this can be done via the Atlas UI, it should not take more than 2-3 minutes each time. To provide a step by step guide after logging into Atlas:

Select the Clusters
Click ... on the Cluster you are trying to perform mongodump on.
Click Edit Configuration
Select Additional Settings
Select More configuration options
On the dropdown menu, change ISRG Root X1 to DST Root CA X3
Wait for changes to apply
Retry mongo and mongodump
If this still fails, follow steps 1-6 again but change from DST Root CA X3 back to ISRG Root X1.

Jason

Ramachandra_Tummala · 2021-06-27T10:57:10.852Z

Ghrib_Ahmed:

I find it strange that my colleague managed to the dump and connect and I didn’t even with the latest versions of the commands

May be your network/ISP not supporting SRV type string and also in some geographic locations SRV will not work

SSL peer certificate validation failed: unable to get local issuer certificate

Ghrib_Ahmed · 2021-06-27T14:52:52.131Z

Thank you all.

The thing is my colleague is connected on a different PC under the exact same network.

Another colleague who is in a different country was capable of both connecting and dumping the database.

This is clearly not a problem with the database itself, but with my own PC.

For this reason, I’d rather not tweak any database configuration and try to see what’s wrong with my PC.

Thank you.