Disk Usage Issues Post MongoDB Upgrade on some Nodes

Michael_Ruf · 2024-09-27T10:15:37.345Z

Hi,

we are currently facing a significant issue with disk space consumption on certain nodes of our MongoDB cluster after upgrading from version 5 over 6 to version 7.0.12. Our cluster consists of three nodes. The normal disk growth is about 3-4GB per day on an affected node it is about 80GB per day.

We monitored the behaviour on a secondary and on the primary. After resyncing either of the affected node, the disk space consumption is back to normal. But after some day it started again on the first affected node, which was fine after the resync.

We observed that the on-disk files of collections and indexes show considerable differences in size between the nodes. Specifically, the “file bytes available for reuse” is much smaller on the affected nodes, even though the overall file size is much larger. So the file will always append, even if data is deleted.
We think that the problem is not related to https://jira.mongodb.org/browse/SERVER-91013 , because we don’t see any problems with the _tmp directory.

We would greatly appreciate any advice or hints in diagnosing the root cause of this issue. For know we are resyncing the affected nodes, before the disk is full.

Michael_Ruf · 2024-11-11T03:03:43.049Z

With the update to 7.0.15 the problem seems to be resolved.