Mongo v6.0.0 immediately exits with featureCompatibilityVersion error

Altay · 2022-08-16T20:30:43.429Z

Since the last reboot of my server I can longer access mongo at all. When I attempt to run the mongodb service I see the following error logged in /var/log/mongodb/mongod.log:

{
  "t": {"$date":"2022-08-16T12:45:18.536-04:00"},
  "s": "F",
  "c": "CONTROL",
  "id": 20573,
  "ctx": "initandlisten",
  "msg": "Wrong mongod version",
  "attr": {
    "error": "UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"4.4\" }. See https://docs.mongodb.com/master/release-notes/5.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value, expected '5.0' or '5.3' or '6.0. See https://docs.mongodb.com/master/release-notes/5.0-compatibility/#feature-compatibility.). If the current featureCompatibilityVersion is below 5.0, see the documentation on upgrading at https://docs.mongodb.com/master/release-notes/5.0/#upgrade-procedures."
  }
}

The instructions for upgrading at the provided link require me to run an adminCommand from the mongo shell, which I cannot do because I cannot access it at all.
The mongo shell is gone:

$ mongo
-bash: mongo: command not found

And the mongosh shell doesn’t work:

$ mongosh
Current Mongosh Log ID: [redacted]
Connecting to:	        mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.5.4
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017

My lock file is empty, but I attempted to repair mongod anyway to no avail.

$ ls -l /var/lib/mongodb/mongod.lock
-rw------- 1 mongodb daemon 0 Aug 16 12:45 /var/lib/mongodb/mongod.lock

I don’t really care about recovering my databases, just getting mongo back into a working state. However, I don’t even know how I can wipe my databases without being able to start the mongo service at all. How can I get mongo running again?

Doug_Duncan · 2022-08-16T20:38:13.193Z

Is the mongod process running? It looks like that it’s not since you’re getting a connection refused.

The FCV in the error message shows that it’s 4.4. Are you trying to upgrade from 4.4, or does your 5.0 database still have the FCV set to 4.4?

Altay · 2022-08-16T20:43:02.083Z

Doug_Duncan:

Is the mongod process running?

No, it immediately terminates with the fatal featureCompatibilityVersion error whenever I attempt to start it.

Doug_Duncan:

Are you trying to upgrade from 4.4, or does your 5.0 database still have the FCV set to 4.4?

I haven’t attempted to perform an update before, so I’m assuming my databases still have FCV set to 4.4. Is there a way I can check?

Doug_Duncan · 2022-08-16T21:31:52.827Z

I have been able to replicate this error by starting up MongoDB 4.4.15. Once it was running I stopped it. I then started up MongoDB 6.0.0 and pointed it at the same path that the 4.4.15 version wrote its database files to and I got the FCV error and the process stopped. This seems to be what you are seeing.

If you want to save your data, you can fix this, by downloading the compressed archive of 5.0.x and extract it. From this extracted folder you can run ./bin/mongod --dbpath <current database path>. Connect to this instance with mongosh and run db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } ). This will change the FCV for you. You can then exit mongosh and then shutdown the mongod instance and finally start your version 6.0.0 server. You will want to change the FCV here as well to be 6.0.

If you don’t care about any of your data, you can just delete the entire database directory that the current data is stored in.

Altay · 2022-08-16T22:05:16.834Z

Doug_Duncan:

If you want to save your data, you can fix this, by downloading the compressed archive of 5.0.x and extract it. From this extracted folder you can run ./bin/mongod --dbpath <current database path> . Connect to this instance with mongosh and run db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } ) . This will change the FCV for you. You can then exit mongosh and then shutdown the mongod instance and finally start your version 6.0.0 server. You will want to change the FCV here as well to be 6.0 .

Thanks! This worked like a charm. Since I’m on Arch Linux I just needed to install mongodb50-bin from the AUR to downgrade, ran the adminCommand, then installed mongodb-bin again to upgrade my binary back to 6.0.0.

Is it good practice to update the FCV after each major release of mongo? Is there a reason it doesn’t happen automatically?

Doug_Duncan:

If you don’t care about any of your data, you can just delete the entire database directory that the current data is stored in.

Good to know in case I run into similar issues in the future.

Doug_Duncan · 2022-08-16T22:08:31.627Z

Altay:

Is it good practice to update the FCV after each major release of mongo? Is there a reason it doesn’t happen automatically?

I think the reason for that is that if you need to roll back due to a failed upgrade having the FCV automatically change could cause even more problems. It’s better to leave the FCV at the upgraded from version for a bit to make sure things work as planned and then only change that to the upgraded to version.

system · 2022-08-21T22:08:41.510Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.