Suboptimal bucket creation in timeseries collection due to cache pressure

ktverdov · 2023-10-19T14:13:02.978Z

Brief description:

When using timeseries-collections, there comes a point when new buckets are created non-optimally and contain only 1 document ( but if you initiate inserting of such collection on mongo without load ( e.g. just after start ), documents with same metadata will be grouped to one bucket ).

Server parameters: 128 GB RAM, memory usage during the experiment does not exceed 80 GB.
Mongo settings: all default ( WiredTiger Cache : 50% RAM, timeseriesIdleBucketExpiryMemoryUsageThreshold set to 3 GB). Pymongo driver is used.

Experiment Description:
Loading is performed into several collections (50 in total).

The general schema of collections provided below (collections are created with the parameter bucketMaxSpanSeconds": 31536000, date is rounded to day ):

# Type 1:
{"metadata": {"id_1": "string", "id_2": "string"}, 
	"date": "Date", "d_0": "int", "d_i": "int", "d_2": "int"}
# 500_000 unique pairs of ("id_1", "id_2"), avg 15 records in month for id pair, 12 months


# Type 2: # more measurement values, 20 instead of 2 - one record size is larger
{"metadata": {"id_1": "string", "id_2": "string"}, 
	"date": "Date", "d_0": "int", "d_i": "int", "d_i": "int", ... , "d_N": "int"} # N = 20
# 25_000 unique pairs ("id_1", "id_2"), avg 15 records in month for id pair, 12 months

Load testing: during the load testing 50 collections are created, after that data in them is inserted by chunks ( chunk contains all dates from specific month for set of id_pairs ), starting with one month, then proceeding to the entire following month, and so on. Operations across different collections are interleaved. Dates within each chunk are sorted. Chunks of specific collection also inserted in time-order.

Issue:
If a new collection is created and a Type 2 collection chunk inserted at the beginning of the load testing ( or before it ), the collstats results are: {'timeseries.bucketCount': 25,000, 'avgNumMeasurementsPerCommit': 17}.

However, if a new collection is created and a Type 2 collection chunk is loaded after loading has been performed in other collections ( during load testing ), the collstats results are vastly different: {'timeseries.bucketCount': 420,000 # (slightly less than the number of elements in the chunk), 'avgNumMeasurementsPerCommit': 1, numBucketsClosedDueToCachePressure: 415,000}. This means that records with the same id pair were not merged, resulting in buckets with a size of 1.

As a result, buckets with a size of 1 are created, although without loading they were of size 17 (for a single chunk) and 200 (for all inserted months). Consequently, the collection begins to occupy significantly more space ( at least 3 times more ) compared to the same collection loaded without any system stress. So bucket effective size optimization seems to create even more load.

Possible reasons:
As far as I understand, when adding a new document, there is a check for the ability to add it to an already opened bucket in the catalog ( check by time, the number of measurements, and so on ). One of the checks involves comparing the size after a potential addition of a new document to the bucket with the effective maximum size of the bucket: bucket.size + sizeToBeAdded > effectiveMaxSize.

# bucket_catalog_internal.cpp, determineRolloverAction function
auto bucketMaxSize = getCacheDerivedBucketMaxSize(opCtx->getServiceContext()->getStorageEngine(), 
	catalog.numberOfActiveBuckets.load()); # bucketMaxSize = storageCacheSize / (2 * numberOfActiveBuckets)

int32_t effectiveMaxSize = std::min(gTimeseriesBucketMaxSize, bucketMaxSize);

The effective size of the bucket, in turn, depends on bucketMaxSize. bucketMaxSize becomes smaller with more active buckets. With time it becomes comparable to the size of one record.

The graphs showing the changes in the number of buckets and memory during load testing are presented below. Generated based on the statistics from db.runCommand({ serverStatus: 1 }).bucketCatalog.

buckets_num1833×809 83.9 KB

Questions:

Is this behavior expected, or is it some kind of a bug (for example, numberOfActiveBuckets should be used instead of numberOfOpenedBuckets or there should be a bottom limit: gTimeseriesBucketMinSize - which is defined but not used )?
Is there anything that can be done about this, or our use case just not suitable for timeseries-collections?
Is the assumption about the cause of the problem accurate or I misunderstanding what’s going on?

ktverdov · 2023-10-19T14:15:03.575Z

“New users can add only one media.”
So closed buckets graph:

closed_buckets1827×793 62.7 KB

ktverdov · 2023-10-19T14:16:12.019Z

bucketCatalog memory usage:

bucket_catalog_memory1829×783 65.5 KB

cagatay_erem · 2024-02-22T07:28:49.173Z

I am having same issue here. I have been playing around with wiredTiger eviction settings but It looks like this issue is correlated with bucket memory usage.

bothside3732×812 189 KB

Around 1.25GB BucketMemory is my max value before starting cachePressure . And It causes a lot Space usage on disk and slow memory in long run.

Everytime I restart, this issue resolves for about 3 days for my setup.

Is there any recommendation how to resolve it?

Zongji_Li · 2024-03-03T14:45:46.413Z

Pretty much same problem here…

Sandeep_Muthangi · 2024-10-18T07:19:26.803Z

Timeseries buckets definitely have an issue. Way more buckets get created. Here is my post about it…

Buggy buckets when using custom bucketing with timeseries Working with Data

Testing bucketing on a timeseries collection. Why are multiple buckets being created even though I am inserting exactly the same data? Collection Info: [image] Data that I am inserting: [image] Multiple buckets for the same timestamp and metadata! [image] Inserted the same set of recs 2x. Instead of 2 buckets, 3 buckets are getting created. A list of 2 records - inserted 2x - is creating 3 buckets? How?

The way one inserts also has a significant impact. Inserts within a short span of time are impacting bucket creation. But, if you space them out - and send them as time-gapped inserts, then the buckets get created properly.

Simon_He · 2025-02-24T15:59:43.162Z

I am having same issue… Bucket catalog memory seems also full. Is there any configuration to set now?

cagatay_erem · 2025-03-03T07:05:57.250Z

I couldn’t find any server param to change bucket catalog memory. It looks like this is correlated via the current ram capacity.

And also, It would be great manually to close buckets for specific collections.
This could be done only by restarting nodes. Once we restart, wiredtiger closes buckets.

Timeseries datas in some cases used as collection iterations rather than a single collection.