Extra space occupied by Indexes

Mehul_Sanghvi · 2025-01-27T06:39:49.860Z

Hi Community,

I recently removed several documents from a collection due to a business logic requirement. However, I noticed that instead of a reduction in index size, it has actually increased.

Given that the data size has decreased, I expected the index size to shrink as well, since the removed documents were the basis for those indexes.

Here are the details:

Index size before the removal process: 1.67 GB
Index size after the removal process: 2.71 GB

Note: These values were collected weekly, and the documents were removed at the end of the week. Even with this timeline, I expected a gradual reduction in index size compared to the previous week since approximately 5.2 million documents were deleted, leaving only 200,000 documents in the collection.

Any insights or suggestions on why this might be happening?

steevej · 2025-02-01T01:47:07.802Z

Size matters. But 1.1 gig should not. More importantly, did you see performance degradation?

In most cases, released memory is not returned to the OS because performance is better when memory reused within the application. In the case of MongoDB an update to a document involves writing back the whole document, in a unused block. If freed block are returned to the OS new blocks are then requested from the OS for new writes. This is inefficient.

Take a look at

mongodb.com

compact - MongoDB Manual v8.0

May be it can help.

Mehul_Sanghvi · 2025-02-05T09:24:55.855Z

steevej:

Size matters. But 1.1 gig should not. More importantly, did you see performance degradation?

Answer to above question, no any changes in performance (Neither degraded nor improved).

steevej:

In most cases, released memory is not returned to the OS because performance is better when memory reused within the application. In the case of MongoDB an update to a document involves writing back the whole document, in a unused block. If freed block are returned to the OS new blocks are then requested from the OS for new writes. This is inefficient.

If this is the scenario, then when should one return the memory to OS as if we won’t return it then there might be chance that it can lead to Memory Fragmentation Problem?

steevej:

Take a look at

https://www.mongodb.com/docs/manual/reference/command/compact/

Yes I know about this command, but before implementing it I want to gather as much as information about Memory Fragmentation & how to fetch how much a memory is fragmentated, what would be the scenario when we need to return memory to OS? And all such questions.

steevej · 2025-02-06T18:02:12.682Z

As far as I know, the store engine is quite efficient at reusing blocks that it marked as deleted.

More detail than that is above my pay grade.

You may want to dig in the following for more insights.

https://source.wiredtiger.com/develop/arch-block.html

Mehul_Sanghvi · 2025-02-10T08:23:57.678Z

Ok, still thanks for sharing your knowledge.