Mongo Export Failing

Hari_N_A2 · 2024-06-20T13:04:12.644Z

Im trying to backup my Cloud:MongoDB Atlas Cluster.

As of now my logical size less than 100MB
I am using the M0 cluster to host my databases.
I have multiple collections within the database.
One of the collections accounts for 80% of the logical size

My current idea is to create a backup of the collections using a ps1 file on task schduler.
To employ this idea I had created a script that uses mongoexport

$backupCommand = "mongoexport --uri='$mongoURI' --out='$backupFile' --db='$db_to_backup' --collection='$collection'"

However for that one huge collection within the DB, it is unable to be fully exported and the connection is terminated at any point during the export.

2024-06-20T10:28:49.663+0800    [........................]  db.collection  0/305251  (0.0%)
2024-06-20T10:28:50.670+0800    [........................]  db.collection  0/305251  (0.0%)
2024-06-20T10:28:50.932+0800    [........................]  db.collection  101/305251  (0.0%)
2024-06-20T10:28:50.933+0800    Failed: connection(shard.mongodb.net:27017[-10]) incomplete read of message header: read tcp LOCAL.IP:MONOGO_PORT->CLOUD.IP:CLOUD_PORT: wsarecv: An existing connection was forcibly closed by the remote host.

This error is happening at random points, sometimes it can be at 30% of the data or it could be at 70%. The connection. In the excrpt above it happend at 0.0%. The other collections with a lesser amount of data are able to be properly backed up.

I have had cases where I ran this exact script and it was able to backup the full collection. However, it took at least 10 minutes to run, which seems long for a <100mb database.
I have tried mongodump but it leads to the same error, even if i try to archive or gzip the output.
Im wondering if Im doing something wrong or would I need to setup something on my cluster to allow this.

Note:

I am not hitting the connection limit of 500, my connections fluctuate at around 50-100
My internet speed is above 100mbps
I dont plan on upgrading my mongo cluster

Thank you in advance for any help.

Samuel_84194 · 2024-06-21T01:08:57.802Z

Hello, welcome to MongoDB Community!

Can you share your options on URI ?

Note> For good practice, we end up using mongodump to backup your instance, it tends to be faster and better ;D

steevej · 2024-08-20T14:37:52.942Z

Hari_N_A2:

I am using the M0 cluster to host my databases.

M0 throttles traffic because it is free and shared. It is possible that your reach the thresholds where Atlas is slowing your traffic and causes the failure.