I’m using Mongo 6.0 (shared replica set environment). Application creates temporary collections to stage the intermediate data to perform data massaging before pushing the data to destination collections.
The data staged in temporary collections are voluminous and have to be retained for a specified time frame (per compliance) and later have to purged to save storage space.
Is there any automated way to drop the entire collection after a specific time period ?
Below are some of the analysis done from my end to solve this problem
- TTL indexes helps here but it removes every document and doesn’t drop the entire collection and this could turn out to be performance issue if temporary collection has voluminous data to remove.
- ObjectId.getTimestamp () - Get the latest timestamp for temporary collection and drop the collection if it exceeds the time threshold.
- mongostat - command line tool and DBA activity so skipping this option.
- Introduce a date field in mono collections to reflect the last modified timestamp and use this as an identifier to classify if a collection is idle or not,
– but the drawback here, there will also be read operations happening in the collection, so need to monitor both read and write operations to classify it as a stale collection. - $collStats - not very helpful for my use case as metrics have to separately captured and cross verified to check if there are changes.
- top command - not very helpful for my use case as metrics have to separately captured and cross verified to check if there are changes.
- db.collection.watch() - change streams
– How to monitor the read operations is again a question - MongoDB Atlas - The application has not subscribed to Atlas.
Can you tag me to posts/analysis on similar lines and help me with an effective approach.