1

Hello MongoDB Community,

I’m currently working on an application that includes a Rust-based HTTP backend, a React frontend, and MongoDB as our database. Our dataset consists of several thousand documents which frequently change due to updates and deletions.

To enhance our web application’s performance and improve user experience, we plan to implement infinite scrolling with backend pagination. The basic idea is to load documents in batches of 50; as the user scrolls to the bottom of the list, the next 50 documents are fetched, and similarly, scrolling to the top fetches the previous set of 50 documents.

From my initial research, I understand that MongoDB supports pagination using the skip() and limit() methods, like so: skip(pageCount * 50).limit(50). However, given the dynamic nature of our data, this approach has raised some concerns:

  1. Assume the initial request fetches the first 50 documents (PAGE 0).
  2. If one of these documents is deleted from the database, the subsequent documents should ideally shift forward to fill the gap.
  3. When the user scrolls down, the frontend requests PAGE 1, which, if we use skip(50).limit(50), would now skip the new 50th document, leading to inconsistent results.

This scenario could result in the user missing out on certain documents unless they reload PAGE 0, which now includes what was originally the 51st document.

I’ve read about using cursors in MongoDB, which can iterate over the documents in increments of 50, based on a snapshot of the database. This seems like it would solve the inconsistency issue by maintaining a consistent view of the data as it was at the start of the query. However, this approach introduces statefulness to what would ideally be a stateless REST endpoint, which is something I’m hoping to avoid.

Does anyone have suggestions or insights on how to handle pagination in a high-churn environment without losing the benefits of statelessness? Any experiences or alternative approaches you could share would be greatly appreciated.

I read something about searchBefore and searchAfter which seems to be exactly what I need. But it seems to be Atlas exclusive (???):

Thank you in advance for your help!

Best regards,

1 Like
2

Hi @Tim_McTa

I’ve been looking for the same solution for some time, but this behavior is impossible to achieve in the current implementation of MongoDB drivers.

I believe that this functionality could be implemented in the MongoDB drivers or MongoDB Database. Every cursor created has an id attached to it, so, if we want to get more data from that cursor, a method to create a new one or reconnect to an existing one using the id would work.