I am relatively new to MongoDB – greener still in developing for reliability in a multi-server deployment scenario. I understand (and have now experienced in practice) that with a replicaSet (of say, 3 nodes) and using a readPreference of secondaryPreferred, a quick write-read can result in a successful write of a new document followed by a failed read of the same from a sufficiently lagging secondary.
My question: Is there a way to configure a replicaSet such that there is a simulated lag in replication that would allow me to (more) reliably reproduce failures of this sort but which are not so extreme as to cause the lagging secondaries to be ruled out for reads altogether? I feel like this would give me more confidence as I develop my application that I’ve followed best practices (using atomic update operations, forcibly reading from the primary when I absolutely must, and so on).
I’ve tried toying with configuring the secondaries with a non-zero secondaryDelaySecs (I tried 5 seconds), but that seemed merely to result in the secondaries being not targeted for reads at all – a find() with a readPreference of secondary in that setup hangs until some 30s timeout is hit.
Any recommendations here?