Hybrid search + pagination

Francisco_Gonzalez_Jara · 2024-11-14T19:54:08.814Z

Hello everyone, Im currently working on a hybrid search implementation and one of the requirements is pagination. After looking into the documentation seems likes there’s no way to do it yet. Is there any workaround or suggestion you can make to solve this problem?

Henry_Weller · 2024-11-22T20:28:12.103Z

Hi @Francisco_Gonzalez_Jara! Thank you for posting, and sorry for the delay in responding.

I am assuming you are following along with the hybrid search tutorial here. May I ask what limit value you are planning on using within your $vectorSearch stage? We have seen <30ms latency for rather large limits (500+) for pure vector search, even against reasonably large datasets (100k-1M vectors) so I suspect that pagination might not strictly necessary. I want to get a better understanding for your anticipated query pattern first before I provide any suggestions.

Francisco_Gonzalez_Jara · 2024-11-26T22:52:49.080Z

Hi @Henry_Weller thanks for replying. Within the vector search stage we are planning to use a limit around and 100 to 150 . Sadly pagination is a strong requirement that I have for the hybrid search responses even if the latency is not a problem. Currently, we are trying to find workarounds like paginate the full response after get all the documents at the same time.

Ramiro_Enriquez · 2024-12-12T18:49:32.966Z

Hi Francisco,

You could look at how to use pagination on the $search stage.

mongodb.com

Paginate the Results - MongoDB Atlas

Retrieve $search results before or after a given reference point.

Francisco_Gonzalez_Jara · 2024-12-12T19:16:00.249Z

yeah I reviewed the documentation, but if im not wrong that would only apply for the fulltext search part of the pipe. Or am I wrong?