Increasing $lookup Stage Limit for Large Aggregations

Victoria_Oberascher · 2025-01-29T15:51:15.816Z

Hi everyone,

I’m using MongoDB as the backend for FiftyOne, and I’m running into an issue when adding a large amount of data to the database. Specifically, I encounter the following error:

OperationFailure: PlanExecutor error during aggregation :: caused by :: Total size of documents in 'collection' matching pipeline's $lookup stage exceeds 104857600 bytes, full error: {'ok': 0.0, 'errmsg': "PlanExecutor error during aggregation :: caused by :: Total size of documents in frames.samples.648aedc2e54c5d1599ee16b1 matching pipeline's $lookup stage exceeds 104857600 bytes", 'code': 4568, 'codeName': 'Location4568'}

From what I understand, the $lookup stage has a limit of 100MB for the total size of documents being processed. Is there any way to increase this limit, either through configuration settings or by adjusting how the aggregation pipeline is structured?

If not, are there any best practices to work around this issue when handling large datasets?

Thanks in advance for any insights!

steevej · 2025-01-30T14:06:47.746Z

While you may use allowDiskUse to probably avoid this issue, you still might risk to reach the 16MB size limit of a single document.

Victoria_Oberascher:

by adjusting how the aggregation pipeline is structured

Probably, but to get help with this you will need to share your aggregation and some sample documents so that we can work it. The best way to share is to create a playground.

steevej · 2025-02-07T03:07:03.226Z

@Victoria_Oberascher, I would appreciate a followup on the reply I wrote more than a week ago.

Thanks