Mongodb intermittent spikes in fetch latencies

Samarth_Goyal · 2020-06-09T13:37:36.914Z

I am facing the issue of intermittent spikes in response latencies while fetching data in a sharded cluster. Data is always being queried via indexed fields.

Also seeing high memory usage on secondary boxes (memoryUsage:99% loadAvg:2.5) at exact same time as the latencies.

Client: Morphia Client with ReadPreference = PRIMARY, ReadConcern = Local, WriteConcern Default. As per my understanding secondaries should not affect the response if ReadConcern is Local.

Is there any correlation here?

MongoDB version 4.2.2
mongod instance running on 8 core, 40 GB RAM machine, wiredtiger cache 11 GB.

Can anybody suggest what might be the issue or how to debug this?

Justin · 2020-06-10T00:54:45.086Z

Hi Samarth_Goyal,

Unfortunately, we don’t have enough information to diagnose the problem. Memory usage is unlikely to be related unless you’re experiencing cache pressure (in fact, 99% memory usage could simply be filesystem cache if you’re checking with free -m).

Latency can be caused by a lot of things. It includes the network initiation, network round trip, query execution, and client-side deserialization. Do you have any measurements available to isolate the issue? Generally, intermittent spikes are network based. If the spikes are more regular, it may be a cron job. Are you instantiating a lot of connections all at once when you’re seeing the spikes? It could be CPU exhaustion.

Ultimately, there are so many things it could be that we’ll need a lot more information. Are you using Atlas? The profiler and performance adviser are invaluable tools for helping to diagnose performance problems with the mongod instance itself.

Stennie_X · 2022-01-31T03:26:17.989Z