Mongodb upgrade from 4.0 to 4.2 incompatible wire version:

shirisha_medi · 2024-07-15T22:58:53.160Z

Hello

I have 7 node cluster, I tried to update one of the replica node from 4.0 to 4.2, but I see the mongod service is not starting up due to below error.

Mon Jul 15 19:00:47.495 F STORAGE [initandlisten] Unable to start up mongod due to missing featureCompatibilityVersion document.
Mon Jul 15 19:00:47.495 F STORAGE [initandlisten] Please run with --repair to restore the document.
Mon Jul 15 19:00:47.495 F - [initandlisten] Fatal Assertion 40652 at src/mongo/db/repair_database_and_check_version.cpp 608
Mon Jul 15 19:00:47.495 F - [initandlisten] \n\n***aborting after fassert() failure\n\n

Mon Jul 15 19:00:47 UTC 2024 : exited with status 3584 (code 14)

From the primary I see the lastHeartbeatMessage for the secondary node which was tried to upgrade as below. How can I resolve this issue ?
Note: Feature Compatibility Version is 4.0 for the replica set on all nodes.

remote host has incompatible wire version: Server min and max wire version (8,8) is incompatible with client min wire version (7,7).You (client) are attempting to connect to a node (server) that no longer accepts connections with your (client’s) binary version. Please upgrade the client’s binary version.",

chris · 2024-07-15T23:10:38.774Z

Hi @shirisha_medi

Please take care to format code and log samples correctly when posting:

Formatting code and log snippets in posts Getting Started

If you are including code and log snippets in a post, consider if there are ways to make these more readable using some of the available formatting options. Well formatted posts will help other users understand and engage with your discussion. Some quick tips for improved formatting: Include short code and log snippets as inline text rather than images. Text is searchable for other users who might encounter a similar issue or need the exact text to help investigate the issue you are experien…

The error is likely due to a previous incomplete upgrade from 3.6 to 4.0.

Rollback the mongodb binaries to 4.0 and complete the upgrade.

Enable backwards-incompatible 4.0 features:

mongodb.com

Upgrade a Replica Set to 4.0 — MongoDB Manual

shirisha_medi · 2024-07-15T23:38:55.895Z

@chris Can you please more information on how the upgrade is verified to be in- complete ?
This MongoDB replica set was upgraded from 4.0.20 to 4.0.28 patch release version last year and didn’t had any issue reported. Feature Compatibility Version was 4.0 for the replica set on all nodes.

Also, I have done the same upgrade the non-prod environment with 5 nodes (upgrading from 4.0 to 4.2) and didn’t see any issue.

I have cleared all the data files on the node where upgrade was showing error, and tried to restart the node. Now the node shows featureCompatibilityVersion 4.2, does changing this back to 4.0 and restarting the node will make the node join the replica set ?

chris · 2024-07-15T23:48:04.678Z

shirisha_medi:

I have cleared all the data files on the node where upgrade was showing error, and tried to restart the node. Now the node shows featureCompatibilityVersion 4.2, does changing this back to 4.0 and restarting the node will make the node join the replica set ?

That depends. What is the state of the rest of the cluster?

If this was just one of the members being upgraded it would be incorrect that featureCompatibilityVersion is already at 4.2 as that is the last step once all members are running the new binary version.

Check the rs.status() and the db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

shirisha_medi · 2024-07-16T00:01:13.385Z

On the node where upgrade was done.

> rs.status();
{
	"ok" : 0,
	"errmsg" : "no replset config has been received",
	"code" : 94,
	"codeName" : "NotYetInitialized"
}
> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } );
{ "featureCompatibilityVersion" : { "version" : "4.2" }, "ok" : 1 }

Rest of the cluster shows one primary node, 4 secondary nodes and the last one as state (not reachable/healthy) and rest of nodes show featureCompatibilityVersion as 4.0

shirisha_medi · 2024-07-16T00:05:50.240Z

If this was just one of the members being upgraded it would be incorrect that featureCompatibilityVersion is already at 4.2 as that is the last step once all members are running the new binary version.

Yes this was the only one member being upgraded and featureCompatibilityVersion was already at 4.2 while rest of the cluster shows 4.0

chris · 2024-07-16T15:30:17.893Z

Looks like you’ll need to take that host back to 4.0. If it is in the replicaSet config(check the other memebers) it should initial sync.