MongoDB 4.4 FCV not completeing sharded cluster

Hello,

I appreciate any help in this mater. We have upgraded our sharded cluster (16 shards) to Mongo 4.4.25. This has been running fine on 4.4.25 for a while so going to set the FCV to 4.4 in order to test the upgrade to Mongo 5.0

I have connected to a mongos as usual and ran the command

db.adminCommand( { setFeatureCompatibilityVersion: "4.4" } )

The command doesn’t seem to complete. When I check on the first 4 shrads have their FCV upgrade to 4.4, but all the remaining shards are at 4.2.

I am seeing the following error repeated the logs

{“t”:{“$date”:“2024-01-31T01:50:41.403+00:00”},“s”:“E”, “c”:“RDELETER”, “id”:23765, “ctx”:“range-deleter”,“msg”:“Unable to find shard key index”,“attr”:{“keyPattern”:{“_id”:“hashed”},“namespace”:“abc123.documents”}}

but I am also seeing that on the shards that have had their FCV upgrade.

Any help or direction would be greatly appreciated.