Im trying to backup my Cloud:MongoDB Atlas Cluster.
As of now my logical size less than 100MB
I am using the M0 cluster to host my databases.
I have multiple collections within the database.
One of the collections accounts for 80% of the logical size
My current idea is to create a backup of the collections using a ps1 file on task schduler.
To employ this idea I had created a script that uses mongoexport
$backupCommand = "mongoexport --uri='$mongoURI' --out='$backupFile' --db='$db_to_backup' --collection='$collection'"
However for that one huge collection within the DB, it is unable to be fully exported and the connection is terminated at any point during the export.
2024-06-20T10:28:49.663+0800 [........................] db.collection 0/305251 (0.0%)
2024-06-20T10:28:50.670+0800 [........................] db.collection 0/305251 (0.0%)
2024-06-20T10:28:50.932+0800 [........................] db.collection 101/305251 (0.0%)
2024-06-20T10:28:50.933+0800 Failed: connection(shard.mongodb.net:27017[-10]) incomplete read of message header: read tcp LOCAL.IP:MONOGO_PORT->CLOUD.IP:CLOUD_PORT: wsarecv: An existing connection was forcibly closed by the remote host.
This error is happening at random points, sometimes it can be at 30% of the data or it could be at 70%. The connection. In the excrpt above it happend at 0.0%. The other collections with a lesser amount of data are able to be properly backed up.
I have had cases where I ran this exact script and it was able to backup the full collection. However, it took at least 10 minutes to run, which seems long for a <100mb database.
I have tried mongodump but it leads to the same error, even if i try to archive or gzip the output.
Im wondering if Im doing something wrong or would I need to setup something on my cluster to allow this.
Note:
- I am not hitting the connection limit of 500, my connections fluctuate at around 50-100
- My internet speed is above 100mbps
- I dont plan on upgrading my mongo cluster
Thank you in advance for any help.
