Mongo repair not working, data corrupted when server shutdown accidently

Joby_Joseph · 2025-03-04T07:42:43.899Z

we faced this issue multiple times. we have mongodb community 6.0.20. when our server run out of space mongodb stops. this is ok. when we made space and restart mongodb. mongo is starting correctly but all data on the mongodb is corrupted and lost.

when we use mongosh command we get error like below

Current Mongosh Log ID: 67c6ae4a80b60365f9e43268
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.3.9
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017

we tried

sudo rm -rf /tmp/mongodb-27017.sock
sudo rm /var/lib/mongodb/mongod.lock
sudo mongod --dbpath /var/lib/mongodb --repair

all these, no luck. it wont work and we lost all data. and we had to reinstall our mongodb to fix this. luckily all these issue happened on development server. we are worried how to recover if this happened on our production server. please help

dogs_Cute · 2025-03-04T08:06:19.334Z

If your data directory still exists. Then it can be restored. Secondly, you should provide the log file content.

Joby_Joseph · 2025-03-04T08:07:47.311Z

{“t”:{“$date”:“2025-03-04T07:51:41.195+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:20698, “ctx”:“-”,“msg”:“***** SERVER RESTARTED **“}
{“t”:{”$date":“2025-03-04T07:51:41.197+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2025-03-04T07:51:41.197+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4915701, “ctx”:“main”,“msg”:“Initialized wire specification”,“attr”:{“spec”:{“incomingExternalClient”:{“minWireVersion”:0,“maxWireVersion”:17},“incomingInternalClient”:{“minWireVersion”:0,“maxWireVersion”:17},“outgoing”:{“minWireVersion”:6,“maxWireVersion”:17},“isInternalClient”:true}}}
{“t”:{“$date”:“2025-03-04T07:51:41.197+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648601, “ctx”:“main”,“msg”:“Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.”}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“TenantMigrationDonorService”,“namespace”:“config.tenantMigrationDonors”}}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“TenantMigrationRecipientService”,“namespace”:“config.tenantMigrationRecipients”}}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“ShardSplitDonorService”,“namespace”:“config.tenantSplitDonors”}}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:5945603, “ctx”:“main”,“msg”:“Multi threading initialized”}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:{“pid”:128980,“port”:27017,“dbPath”:“/var/lib/mongodb”,“architecture”:“64-bit”,“host”:“swimphonyserver”}}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“6.0.20”,“gitVersion”:“ef4efd469fb0c5626c629c4f0f21398c9b5dd08f”,“openSSLVersion”:“OpenSSL 3.0.2 15 Mar 2022”,“modules”:[],“allocator”:“tcmalloc”,“environment”:{“distmod”:“ubuntu2204”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:{“os”:{“name”:“Ubuntu”,“version”:“22.04”}}}
{“t”:{“$date”:“2025-03-04T07:51:41.206+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:{“options”:{“config”:“/etc/mongod.conf”,“net”:{“bindIp”:“127.0.0.1”,“port”:27017},“processManagement”:{“timeZoneInfo”:“/usr/share/zoneinfo”},“storage”:{“dbPath”:“/var/lib/mongodb”},“systemLog”:{“destination”:“file”,“logAppend”:true,“path”:“/var/log/mongodb/mongod.log”}}}}
{“t”:{“$date”:“2025-03-04T07:51:41.207+00:00”},“s”:“W”, “c”:“STORAGE”, “id”:22271, “ctx”:“initandlisten”,“msg”:“Detected unclean shutdown - Lock file is not empty”,“attr”:{“lockFile”:“/var/lib/mongodb/mongod.lock”}}
{“t”:{“$date”:“2025-03-04T07:51:41.207+00:00”},“s”:“F”, “c”:“STORAGE”, “id”:50922, “ctx”:“initandlisten”,“msg”:“An incomplete repair has been detected! This is likely because a repair operation unexpectedly failed before completing. MongoDB will not start up again without --repair.”}
{“t”:{“$date”:“2025-03-04T07:51:41.207+00:00”},“s”:“F”, “c”:“ASSERT”, “id”:23091, “ctx”:“initandlisten”,“msg”:“Fatal assertion”,“attr”:{“msgid”:50922,“file”:“src/mongo/db/storage/storage_engine_init.cpp”,“line”:106}}
{“t”:{“$date”:“2025-03-04T07:51:41.207+00:00”},“s”:“F”, “c”:“ASSERT”, “id”:23092, “ctx”:“initandlisten”,“msg”:"\n\naborting after fassert() failure\n\n”}

this is from mongo db log when attempting mongodb start
tried sudo rm /var/lib/mongodb/mongod.lock
sudo mongod --dbpath /var/lib/mongodb --repair

this give a big output in screen here is the final messages

{“t”:{“$date”:“2025-03-04T08:06:29.357+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22327, “ctx”:“initandlisten”,“msg”:“Verify succeeded. Not salvaging.”,“attr”:{“uri”:“table:collection-15681–2079246058934227174”}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“I”, “c”:“INDEX”, “id”:20303, “ctx”:“initandlisten”,“msg”:“validating collection”,“attr”:{“namespace”:“local.orphan.collection_15681__2079246058934227174”,“uuid”:{“uuid”:{“$uuid”:“866591c0-272e-4074-9cb6-cf79129ab4a6”}}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“I”, “c”:“INDEX”, “id”:20306, “ctx”:“initandlisten”,“msg”:“Validation complete for collection. No corruption found”,“attr”:{“namespace”:“local.orphan.collection_15681__2079246058934227174”,“uuid”:{“uuid”:{“$uuid”:“866591c0-272e-4074-9cb6-cf79129ab4a6”}}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:21028, “ctx”:“initandlisten”,“msg”:“Collection validation”,“attr”:{“results”:{“ns”:“local.orphan.collection_15681__2079246058934227174”,“nInvalidDocuments”:0,“nNonCompliantDocuments”:0,“nrecords”:0,“nIndexes”:0,“keysPerIndex”:{},“indexDetails”:{}},“detailedResults”:{“valid”:true,“repaired”:false,“warnings”:,“errors”:,“extraIndexEntries”:,“missingIndexEntries”:,“corruptRecords”:}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:4934002, “ctx”:“initandlisten”,“msg”:“Validate did not make any repairs”,“attr”:{“collection”:“local.orphan.collection_15681__2079246058934227174”}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:21027, “ctx”:“initandlisten”,“msg”:“Repairing collection”,“attr”:{“namespace”:“local.orphan.collection_18528__2079246058934227174”}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“E”, “c”:“WT”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error message”,“attr”:{“error”:22,“message”:{“ts_sec”:1741075589,“ts_usec”:359522,“thread”:“130047:0x7f9a1fd2ccc0”,“session_name”:“WT_SESSION.verify”,“category”:“WT_VERB_DEFAULT”,“category_id”:9,“verbose_level”:“ERROR”,“verbose_level_id”:-3,“msg”:“__wt_schema_get_table_uri:85:‘table:collection-18528–2079246058934227174’ cannot be used until all column groups are created”,“error_str”:“Invalid argument”,“error_code”:22}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22328, “ctx”:“initandlisten”,“msg”:“Verify failed. Running a salvage operation.”,“attr”:{“uri”:“table:collection-18528–2079246058934227174”}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“E”, “c”:“WT”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error message”,“attr”:{“error”:22,“message”:{“ts_sec”:1741075589,“ts_usec”:359572,“thread”:“130047:0x7f9a1fd2ccc0”,“session_name”:“WT_SESSION.salvage”,“category”:“WT_VERB_DEFAULT”,“category_id”:9,“verbose_level”:“ERROR”,“verbose_level_id”:-3,“msg”:“__wt_schema_get_table_uri:85:‘table:collection-18528–2079246058934227174’ cannot be used until all column groups are created”,“error_str”:“Invalid argument”,“error_code”:22}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“W”, “c”:“STORAGE”, “id”:22351, “ctx”:“initandlisten”,“msg”:“Salvage failed. The file will be moved out of the way and a new ident will be created.”,“attr”:{“uri”:“table:collection-18528–2079246058934227174”,“error”:{“code”:2,“codeName”:“BadValue”,“errmsg”:“Salvage failed: 22: Invalid argument”}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“W”, “c”:“STORAGE”, “id”:22353, “ctx”:“initandlisten”,“msg”:“Rebuilding ident”,“attr”:{“ident”:“collection-18528–2079246058934227174”}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“E”, “c”:“WT”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error message”,“attr”:{“error”:-31803,“message”:{“ts_sec”:1741075589,“ts_usec”:359640,“thread”:“130047:0x7f9a1fd2ccc0”,“session_dhandle_name”:“file:WiredTiger.wt”,“session_name”:“WT_CURSOR.search”,“category”:“WT_VERB_DEFAULT”,“category_id”:9,“verbose_level”:“ERROR”,“verbose_level_id”:-3,“msg”:“__schema_create_collapse:106:metadata information for source configuration "colgroup:collection-18528–2079246058934227174" not found”,“error_str”:“WT_NOTFOUND: item not found”,“error_code”:-31803}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“E”, “c”:“STORAGE”, “id”:22357, “ctx”:“initandlisten”,“msg”:“Rebuilding ident failed: failed to get metadata”,“attr”:{“uri”:“table:collection-18528–2079246058934227174”,“error”:{“code”:4,“codeName”:“NoSuchKey”,“errmsg”:“Unable to find metadata for table:collection-18528–2079246058934227174”}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“F”, “c”:“STORAGE”, “id”:21030, “ctx”:“initandlisten”,“msg”:“Failed to repair database”,“attr”:{“db”:“local”,“error”:{“code”:4,“codeName”:“NoSuchKey”,“errmsg”:“Unable to find metadata for table:collection-18528–2079246058934227174”}}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“F”, “c”:“ASSERT”, “id”:23095, “ctx”:“initandlisten”,“msg”:“Fatal assertion”,“attr”:{“msgid”:4805001,“error”:“NoSuchKey: Unable to find metadata for table:collection-18528–2079246058934227174”,“file”:“src/mongo/db/startup_recovery.cpp”,“line”:557}}
{“t”:{“$date”:“2025-03-04T08:06:29.359+00:00”},“s”:“F”, “c”:“ASSERT”, “id”:23096, “ctx”:“initandlisten”,“msg”:“\n\n***aborting after fassert() failure\n\n”}

Joby_Joseph · 2025-03-04T08:24:02.618Z

my mongodb is back after i removed all my database files.

sudo rm -rf /var/lib/mongodb/*
sudo systemctl restart mongod

is there anyway to do this without removing the full data?

Joby_Joseph · 2025-03-04T08:56:23.454Z

i tried a lot repair isnot working