MAX System CPU Spiking

Andy_Hynes · 2024-09-18T16:59:55.437Z

I have a mongo database deployed in production and recently I have been noticing the Max System CPU spiking unexpectedly. From analysing my metrics it looks like there are two types of spikes;

There are significant spikes in Max System CPU when there are small spikes in Normalized System User CPU. For example, if my normalized system user cpu increases from 3.8% to 4.2% my max System CPU goes from 100% to 200% CPU. Is this normal?
The Max System CPU arbitrarily spikes without any corresponding increase in normalized system cpu. This seems to happen once every week or so and I am struggling to identify any metrics which indicate why this might be. My one suspicion is that it could be cause by some sort of backup/migration happening on the node, but I am not sure how to validate this suspicion.

Can anyone help me debug these issues?
Thanks!

Artur_57972 · 2024-09-18T19:56:13.527Z

Are you using micro-services architecture?

steevej · 2024-09-18T21:01:18.300Z

Andy_Hynes:

seems to happen once every week or so

Do you have any TTL index? The regular spike might corresponds to the time when documents are automatically removed by the index.

Andy_Hynes · 2024-09-25T19:56:47.312Z

Hi, Thanks for the responses!

No we have a monolith i.e. one server and one database, but I am interested why a microservice architecture might impact this?

No we don’t have any TTL for any collection. I had thought it might be down to some periodic job but nothing jumps out at me either

Artur_57972 · 2024-09-25T20:22:27.289Z

Quite hard to point out what could be generating this CPU usage with more information about the application. Other possible questions:

How many indexes do you have on your biggest and/or most used collection?
Do you have documents with fields of type array with many items (more than 100) and this field is also indexed?

Regarding the microservice, the issue is that every time it executes, it’ll need to create an authenticated connection to the DB, the process of authenticating is CPU intensive, so, if you have a microservice that executes thousands of times a second, thousands of authentication would be happening, using a ton of CPU.