Efficiently Querying Large Collections with Complex Conditions - Emmanuel Katto

emmanuelkatto · 2024-07-16T04:39:52.413Z

Hi All, I am Emmanuel Katto. I’m working on a project that involves storing and querying a large collection of documents in MongoDB. The collection contains around 10 million documents, and I’m struggling to query it efficiently when using complex conditions.

I have a document structure that looks like this:

{
“_id”: ObjectId,
“name”: String,
“categories”: [String],
“tags”: [String],
“createdAt”: Date,
“updatedAt”: Date
}
I need to query this collection to retrieve all documents that match the following conditions:

The name field contains the string “example”
The categories array contains at least one element that matches “categoryA” or “categoryB”
The tags array contains at least one element that matches “tag1” or “tag2”
The createdAt field is within a specific date range (e.g. last 24 hours)

I’ve tried using the $regex operator to query the name field, and the $elemMatch operator to query the categories and tags arrays. However, my queries are taking a long time to execute and returning a large number of results.

Is there a more efficient way to query my collection using MongoDB’s aggregation pipeline? Please let me know.

Thanks!
Emmanuel

Lokesh_Reddy1 · 2024-07-16T08:19:01.703Z

Hi @emmanuelkatto ,

Please provide some additional information below, that may helps to give the better solutions.

Are you tried to get documents by partial matching on name field or need to query with exact match?
Have you created any indexes in that collection to enhance the query performance ?
Is there any other stages after the filtered data ?
Please provide the sample query that you have written earlier.

Regards,
Lokesh.