MongoDB TTL Index. doesn't delete old documents.

Bar_Shauli · 2025-01-21T18:34:10.669Z

Subject: Index Issue with MongoDB Collection

Hi Team,

We have a schema X that contains collection Y. Here is an example of a document:

{
    "_id": ObjectId('XXX'),
    "artifact": "XX-YY",
    "date": 1730509200015,
    "fileName": "google",
    "logFile": "a5c9.gz",
    "version": "1",
    "date_created": ISODate('2024-11-02T01:00:00.015Z'),
    "originalFilenames": ["google"],
    "platforms": ["viewer"],
    "requestsCounter": 90,
    "responseSize": 16,
    "url": "/services/editor.html"
}

The indexes on this collection are as follows:

[
  { "v": 2, "key": { "date": 1 }, "name": "date_1" },
  { "v": 2, "key": { "_id": 1 }, "name": "_id_" },
  {
    "v": 2,
    "key": { "date_created": -1 },
    "name": "date_created_-1",
    "background": true,
    "expireAfterSeconds": 864000
  }
]

The Problem:

The index date_created_-1 does not delete old documents (older than 864000 seconds). We have tried dropping and recreating this index. Our MongoDB version is 6, and we upgraded this cluster from version 4.4. Since the upgrade, we have encountered this issue. The size of the collection has increased to 1.5TB.

How can we fix this? Could it be related to the upgrade? This issue did not occur in our other clusters.

Please advise.

chris · 2025-01-22T02:32:24.956Z

Hi @Bar_Shauli

TTL indexes are very slow on descending order indexes in MongoDB versions before 7.0

Upgrade to MongoDB 7.0 (or 8.0) or recreate the the TTL index in ascending order.

It is also possible something else is wrong, review the log file for errors relating to TTL index.

The progress of TTLMonitor can be observed via db.serverStatus().metrics.ttl the deleted documents. Passes should increase regularly, every minute or so.

ref: https://jira.mongodb.org/browse/SERVER-56274

Bar_Shauli · 2025-01-22T20:11:21.888Z

Hi Chris,
Thanks a lot for your detailed response.

Recreated the index with asending :
db.X.getIndexes()
[
{ v: 2, key: { date: 1 }, name: ‘date_1’ },
{ v: 2, key: { _id: 1 }, name: ‘id’ },
{
v: 2,
key: { date_created: 1 },
name: ‘date_created_1’,
background: true,
expireAfterSeconds: 864000
}
]
The command : db.serverStatus().metrics.ttl
{ deletedDocuments: Long(‘0’), passes: Long(‘0’) }

And yet the problem didn’t resolved.
Could it be related to version 6 ?

Bar_Shauli · 2025-01-23T14:14:40.988Z

We also upgraded to version 7 - which didn’t help.
Can you please advise what else can we check ?

chris · 2025-01-23T16:07:34.450Z

Check the mongod log for anything relating to TTL.

A couple of other things to check:

TTL Monitor has been disabled. Check the mongod.conf(or command line parameters) for setParameter and TTLMonitorEnabled: false or db.adminCommand({getParameter:1,ttlMonitorEnabled:1})
If this host was previously part of a replicaSet and is subsequently running in standalone then the TTLMonitor will be disabled, this is a logged condition. If this is the case and the host will remain a standalone the drop the local database and restart mongodb db.getSiblingDB('local').dropDatabase()

ref:
https://www.mongodb.com/docs/manual/reference/parameters/#mongodb-parameter-param.ttlMonitorEnabled
https://www.mongodb.com/docs/manual/core/index-ttl/#mongod-in-standalone-mode