Missing Indexes while Mongo restoration

Dinesh_Reddy_Sanikommu · 2024-08-15T12:33:52.410Z

I am doing a Backup of Data from Production to Lower Environments using mongodump and mongorestore, In the process of restoration, The Indexes are not getting restored and in the log file it shows the below output. Also the creation of remaining Indexes are also getting hold, Is there a way to make

Fabio_Ramohitaj · 2024-08-15T14:27:29.237Z

Hi @Dinesh_Reddy_Sanikommu,
I ran a test and I get indexes restored correctly as well. Can you better explain what problem you are experiencing? What version of MongoDB are you working on?

Best Regards

Dinesh_Reddy_Sanikommu · 2024-08-15T17:28:48.989Z

Hi @Fabio_Ramohitaj , Thanks for the reply,

So when I clone the data (mongo dump) from Prod and restore in my Lower Environments (mongo restore), I believe there are few Indexes missing, Is this an expected behavior.

If not, please let me know what are validation steps should if perform?

Regards,
Dinesh

Dinesh_Reddy_Sanikommu · 2024-08-15T17:30:27.988Z

Also My Atlas Mongo DB version is 5.

Fabio_Ramohitaj · 2024-08-15T19:14:37.680Z

Hi @Dinesh_Reddy_Sanikommu,

I simulated the following situation:

Created the collection with some document:

db.collection.insertMany([{name: "Fabio"},{name: "Marco"},{name: "Luca"}])
{
 acknowledged: true,
 insertedIds: {
   '0': ObjectId('66acb407fe3ec6f3c4079058'),
   '1': ObjectId('66acb407fe3ec6f3c4079059'),
   '2': ObjectId('66acb407fe3ec6f3c407905a')
 }
}

Create an index on the field name:

db.collection.createIndex({name:1})

Dump the database with the mongodump utility:

mongodump -d database

the dump folder will be created in the current directory and in turn it will contain a folder with the name of the database and inside it will be the bson of the collection + a file containing the metadata

dump/database/collection.metadata.json
dump/database/collection.bson

Finally we go to restore:

mongorestore dump/

To verify that the index has been imported, we can run the following command:

db.collection.getIndexes()

I hope it helps to solve the problem!

Best Regards

Yilmaz_Durmaz · 2024-08-16T17:59:00.472Z

Dinesh_Reddy_Sanikommu:

in the log file it shows the below output

you forgot to add this detail. please edit your question and add it.

Dinesh_Reddy_Sanikommu · 2024-09-16T17:55:44.745Z

Hi @Fabio_Ramohitaj and @Yilmaz_Durmaz ,

Here are the steps which i follow for taking Dump and Restore the MongoDB from My Prod to Stage environment.

Dump From Prod::
nohup /share/infa/AdminScripts/infa/Mongo_DB_Refresh/mongotools/mongodb-database-tools-rhel70-x86_64-100.8.0/bin/mongodump mongodb+srv://:@datahub-prod-g-pl-0.ldvl.mongodb.net/ --db=PTFPIMSPS --numParallelCollections=391 --gzip --out=/share/infa/Backup/mongo_export/PTFPIMSPS_SG_${START_DATE}/ >> /share/infa/Backup/mongo_export/PTFPIMSPS_SG_log_${START_DATE}/PTFPIMSPS_SG_${START_DATE}.log &

It now generates both collection.bson and collection.metadata.json file

Restore in Stage::

In the restore part, when i execute the below command I facing the following issue’s:

nohup /share/infa/AdminScripts/infa/Mongo_DB_Refresh/mongotools/mongodb-database-tools-rhel70-x86_64-100.8.0/bin/mongorestore mongodb+srv://:@datahub-stage-g-pl-0.a94l.mongodb.net/?appName=FF_STFPIMSQS_${BATCHDATE} --nsFrom PTFPIMSPS.* --nsTo STFPIMSQS.* --numParallelCollections=100 --numInsertionWorkersPerCollection=10 --gzip --dir /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}${BATCHDATE}/ --drop >> /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}log${BATCHDATE}/${DB_Name}${Identifier}_${BATCHDATE}.log &

Error:

Failed: STFPIMSQS.SSE036: error creating collection STFPIMSQS.SSE036: error running create command:
(NamespaceExists) Collection STFPIMSQS.SSE036 already exists.

Can you please help me on this.

Regards,
Dinesh S

Fabio_Ramohitaj · 2024-09-17T16:16:22.235Z

Hi @Dinesh_Reddy_Sanikommu,

Please, format the commands for a better readability.

Formatting code and log snippets in posts - About the Community / Getting Started - MongoDB Developer Community Forums

Can you paste the dbs that are present in the target cluster?
The restore command looks like ok.

Best Regards

Dinesh_Reddy_Sanikommu · 2024-09-17T17:06:28.650Z

Sure @Fabio_Ramohitaj , I taking the Dump from PTFPIMSPS database which is in Production and Restoring it in to STFPIMSQS db.

mongorestore mongodb+srv://:@datahub-stage-g-pl-0.a94l.mongodb.net/?appName=FF_STFPIMSQS_${BATCHDATE} --nsFrom PTFPIMSPS.* --nsTo STFPIMSQS.* --numParallelCollections=100 --numInsertionWorkersPerCollection=10 --gzip --dir /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}${BATCHDATE}/ --drop >> /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}log${BATCHDATE}/${DB_Name}* ${Identifier}_${BATCHDATE}.log &

Please let me know if any required.

Also I have the following error while the Index restoration and remaining Indexes are not created.

Failed: STFPIMSQS.SFF004: error creating indexes for STFPIMSQS.SFF004: createIndex error: (DuplicateKey)
Index build failed: c3565a46-31ed-4271-b67d-cbed6f5db828: Collection STFPIMSQS.SFF004 ( b0257688-2d84-40c5-bd60-43d0c851acac )
:: caused by :: E11000 duplicate key error collection: STFPIMSQS.SFF004 index: SFF004_RBA_1 dup key: { SFF004_RBA: 187214632 }

Fabio_Ramohitaj · 2024-09-17T17:25:50.930Z

Hey @Dinesh_Reddy_Sanikommu,
You should format the code and show me what databases are in the target cluster.

Regards

Yilmaz_Durmaz · 2024-09-17T18:21:25.006Z

Dinesh_Reddy_Sanikommu:

Collection STFPIMSQS.SSE036 already exists.

Dinesh_Reddy_Sanikommu:

duplicate key error collection: STFPIMSQS.SFF004

it seems you are trying to restore to an already existing collection/database/index, and --drop does not drop them properly.

a longer error log would be better to help you, as you mentioned in your first post but forgot to attach. please add one.

and for code segments and logs lines, select them and click on the "preformatted text button. or place them between backtick quotes.

Dinesh_Reddy_Sanikommu · 2024-09-17T20:01:21.655Z

Hi @Fabio_Ramohitaj ,

Here are the list of databases in my stage (target) cluster.

STFPIMSQS, STFPIMSSS, admin, config, local and featureflags

Regards

Fabio_Ramohitaj · 2024-09-17T20:20:16.188Z

Hi @Dinesh_Reddy_Sanikommu
The --drop option is ineffective when the nsFrom and nsTo options are used, I hypothesize because maybe an attempt is made to drop the nsFrom db (which does not exist) or something similar (you would have to add verbosity to verify that this is effectively the case), so to solve the problem, you will have to manually drop the STFPIMSQS database into the target cluster in order for the restore to be executed correctly.
If it is necessary, make a backup of the database that you will have to drop before the drop.

Regards

Dinesh_Reddy_Sanikommu · 2024-09-19T17:07:38.160Z

Hi @Fabio_Ramohitaj , By manually dropping the STFPIMSQS database would drop all the collections, where in the above process of the restore command which I have shared will only restore the collections existing from Prod to Stage, and will not impact the rest of the stage collections(Not exist in Prod). Could you just any other feasible way’s.

Yilmaz_Durmaz · 2024-09-19T17:39:42.087Z

using --drop you are already telling the command to drop the collections before restoring from backup, which means you are already impacting them.

please refer to official documentation:
mongorestore - MongoDB Database Tools

Laura-Emilia_Toporas · 2024-10-15T15:10:53.110Z

Hi,

I am having a similar issue trying to automate a backup/restore procedure. I was doing several tests and on one occasion I backed up the database, then manually deleted the database and afterwards restored the backup. In this case only the custom indexes were not restored, even though the version of the database that I backed up did include the custom indexes. Only the default index was restored.
If I do the same procedure with the --drop option on mongorestore, the custom indexes are restored.
This is not to discuss the opportunity of manually deleting a database but accidents happen, so I would expect that all the metadata is backed up and restored, that includes custom indexes.
The following are the commands I am using for backup/restore:

mongodump --uri=‘{uri}’
–username=‘{accessKeyId}’
–password=‘{secretAccessKey}’
–awsSessionToken=‘{sessionToken}’
–db=‘{database}’
–gzip
-vvv
mongorestore --uri=‘{uri}’
–username=‘{accessKeyId}’
–password=‘{secretAccessKey}’
–awsSessionToken=‘{sessionToken}’
–db=‘{database}’
–drop
–gzip
./dump
-vvv
So I have several questions
not saving/restoring custom indexes when the database is manually deleted is by design or is it a missed corner case?
how is this possible? Are the custom indexes linked somehow to the pre-existence of the database? or are the mongodump/mongorestore commands perform some inner logic on the custom indexes specifically?

Thank you