Failed to start WiredTiger after system upgrade

Russell_Almond · 2020-12-15T20:55:42.225Z

I recently was having so many problems with my computer (gdm issues unrelated to mongo), that I wound up upgrading my OS from Ubuntu 20.04 to Pop_OS 20.10. In the process, I needed to reinstall mongo-org. Fortunately, my old database was on a secondary drive and hence did not get refreshed.

However, when I try to start mongod, it is failing to start. I’m getting the following message in the log file.

{"t":{"$date":"2020-12-15T15:27:43.546-05:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/home1/mongo/db/","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/home1/mongo/log/mongod.log"}}}}
{"t":{"$date":"2020-12-15T15:27:43.590-05:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/home1/mongo/db/","storageEngine":"wiredTiger"}}
{"t":{"$date":"2020-12-15T15:27:43.590-05:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2020-12-15T15:27:43.590-05:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15515M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2020-12-15T15:27:43.974-05:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2020-12-15T15:27:43.974-05:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
{"t":{"$date":"2020-12-15T15:27:43.975-05:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":1123}}
{"t":{"$date":"2020-12-15T15:27:43.975-05:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

I’m guessing that I upgraded mongo in the process of upgrading the rest of the system, and that somehow the old database is not adjusting itself for compatability?

Is there some kind of database repair I can run to fix this problem? (Fortunately, I can restore any mission critical data from backup if need be).

BM_Sharma · 2020-12-16T05:42:15.804Z

Hi Russell_Almond,

Share the below detail-
1.) Version of MongoDB server before upgrade.
2.) Version of MongoDB server after upgrade.

Thanks
Braj Mohan

chris · 2020-12-16T12:58:22.004Z

Russell_Almond:

Is there some kind of database repair I can run to fix this problem? (Fortunately, I can restore any mission critical data from backup if need be).

No, you’re going to have to run a mongod that can handle that compatibility level. Best use the last version that you ran. Then upgrade following the procedures in the release note.

Russell_Almond · 2020-12-17T18:19:14.858Z

I’m still not sure of the original version, but given that I’m at version 3.6 on my laptop, it looks like a multistep process.

Fortunately, in my case, the critical data is backed up as JSON files, so it is probably faster to kill the old database and then rebuild from JSON.

Is there an easy way to tell what version of Mongo the database was built under? That might help for a more informative error message.

chris · 2020-12-17T18:25:31.552Z

@Russell_Almond see this post. It show which version it successfully ran under last. Use the most recent metrics file.

How to check mongo version from volumes that I've set in the docker-compose.yml file? Working with Data

Hello @tanapone_kanongsri Yes it is, sort of. Note that this will only show what version last successfully started the database not what FCV it is at. On startup mongod will write out the results of some commands to the diagnostic.data folder, this can be parsed to see what version it was running at that time. This is an example of a data directory that does not start with mongod 4.4 I’m not sure if there are any other tricks to find the info. When I tried a 3.6 database with mongod 4.2 it…

hb0 · 2021-05-04T09:10:28.169Z

Just because this pops up first when searching for the Error

Failed to start WiredTiger after system upgrade

In our case we had mongo-4.2.8 running and wanted to upgrade to mongo-4.4.4 which should work just fine, but the featureCompatibilityVersion was still set to “4.0” instead of “4.2” so the upgrade did not succeed.

You can solve this easily with:

downgrade to 4.2.8 (in our case) again
Follow the migration steps to set the featureCompatibilityVersion to 4.2
Shut down your database (or replica set) cleanly as stated in the migration guide
Upgrade the binary to 4.4.4 (in our case)
Everything should start correctly. Now you set the featureCompatibilityVersion to 4.4 for future upgrades.

Stefan_Schmidt · 2022-07-01T09:41:59.393Z

thx, this is really helping