Mongodb upgrade from 4.0 to 4.2 incompatible wire version:

Hello

I have 7 node cluster, I tried to update one of the replica node from 4.0 to 4.2, but I see the mongod service is not starting up due to below error.

Mon Jul 15 19:00:47.495 F STORAGE [initandlisten] Unable to start up mongod due to missing featureCompatibilityVersion document.
Mon Jul 15 19:00:47.495 F STORAGE [initandlisten] Please run with --repair to restore the document.
Mon Jul 15 19:00:47.495 F - [initandlisten] Fatal Assertion 40652 at src/mongo/db/repair_database_and_check_version.cpp 608
Mon Jul 15 19:00:47.495 F - [initandlisten] \n\n***aborting after fassert() failure\n\n

Mon Jul 15 19:00:47 UTC 2024 : exited with status 3584 (code 14)

From the primary I see the lastHeartbeatMessage for the secondary node which was tried to upgrade as below. How can I resolve this issue ?
Note: Feature Compatibility Version is 4.0 for the replica set on all nodes.

remote host has incompatible wire version: Server min and max wire version (8,8) is incompatible with client min wire version (7,7).You (client) are attempting to connect to a node (server) that no longer accepts connections with your (client’s) binary version. Please upgrade the client’s binary version.",

@chris Can you please more information on how the upgrade is verified to be in- complete ?
This MongoDB replica set was upgraded from 4.0.20 to 4.0.28 patch release version last year and didn’t had any issue reported. Feature Compatibility Version was 4.0 for the replica set on all nodes.

Also, I have done the same upgrade the non-prod environment with 5 nodes (upgrading from 4.0 to 4.2) and didn’t see any issue.

I have cleared all the data files on the node where upgrade was showing error, and tried to restart the node. Now the node shows featureCompatibilityVersion 4.2, does changing this back to 4.0 and restarting the node will make the node join the replica set ?

On the node where upgrade was done.

> rs.status();
{
	"ok" : 0,
	"errmsg" : "no replset config has been received",
	"code" : 94,
	"codeName" : "NotYetInitialized"
}
> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } );
{ "featureCompatibilityVersion" : { "version" : "4.2" }, "ok" : 1 }

Rest of the cluster shows one primary node, 4 secondary nodes and the last one as state (not reachable/healthy) and rest of nodes show featureCompatibilityVersion as 4.0