Unable to setFeatureCompatibilityVersion to MongoDB 4.2

Ops and Admin Installation & Upgrades
1

Hi guys,
we’re facing very strange error. Long story short - we’ve really old database on mongo 3.4.
Finally we have the time and resources to start upgrading the same as much as we can. However, to avoid downtime, we decided to go the long path and do upgrades version by version.
No issues with 3.4->3.6, 3.6->4.0 (after each upgrade we did setFeatureCompatibilityVersion to the respective version). However, once we switched to 4.2 and tried to setFeatureCompatibilityVersion we get an error
BSON field 'setFCV-find-matching-sessions.lastWriteOpTime is missing but a required field`.

It took us some time to find out what exactly is the problem, but finally our best guess are 2 records in config.transactions`` collection. All records, except those two, have lastWriteOpTime field, while those two havelastWriteOpTimeTs` field (rest don’t). But those transactions seems “stucked”. No progress and transaction reaper doesn’t abort them.

Any suggestions how we can proceed and fix this so that we can continue with the upgrade?

2

Some update from my side.

NOTE: I’m not recommending this fix in any way. Still trying to understand all the implications of it. Probably it’s safe, but still don’t know for sure.

After spending few hours trying to understand what and why is happening, I decide to moved on with manually fix those “transactions” entries. However, in replicaset, config.transactions cannot be modified, so what I did was:

  • Remove on of the replicas from the set and start it in standalone mode.
  • Delete the 2 entries.
  • Restart again as part of the replicaset (--replSet rs0 flag) and add it back. Wait to replicate changes.
  • Do the same with the second replica.
  • Stepdown the primary and do the same.

After that, cluster was able to upgrade to v4.2 FCV and looks like working correctly and without any downtime.

Still, I’m open if there is other solutions for this problem and any feedback regarding above.

1 Like