Slow queries on large number of documents

Raphael_Canin · 2024-09-20T19:25:54.345Z

Hello,

I have a 6.4M documents database with an average size of 8kB.

A document has a schema like this :

{"group_ulid": str, "position": int, "..."}

I have 15 other columns that are :

dict with 5-10 keys
small list (max 5 elements) of dict with 5-10 keys

I want to retrieve all documents of a given group_ulid (~5000-10000 documents) but it is slow (~1.5 seconds). I’m using pymongo :

res = collection.find({"group_ulid": "..."}) 
res = list(res)

I am running mongo using Docker on a 16 GB and 2 vCPU instance.

I have an index on group_ulid, ascendant. The index is like 30MB.

Are there some ways to make it faster ? Is this a normal behavior ?

Thanks

Shane · 2024-09-20T19:57:37.514Z

1.5 seconds seems slow. Does your app reuse a single MongoClient? If not, recreating a client for each find() will add latency due to setting up new connections each time.

Otherwise, could you try profiling the find() via py-spy, cProfile, and/or memray to help isolate the slow code?