MongoDB Data federation limitation

Manish_Bisht · 2022-09-26T16:11:21.868Z

Hello, I m relatively new to MongoDB.
I am working on an application and I have a certain use case where I have to use mongodb lookups on two different collections stored in two different databases. Since, lookups dont have support for cross databases, I decided to use MongoDB Data Federation.
The query works now using federated instance. I read up more about the limitation of Data federation by reading this article data-federation-limitations.

One of the limitations state that federated instance can only have 60 guaranteed concurrent connections per region and only 30 concurrent queries. So if my application were to be used by thousands of users , would there be a performance issue??
If so, how can I do this task differently?

Benjamin_Flast · 2022-09-26T22:04:40.976Z

Hello Manish,

Ben here, I’m the PM for Atlas Data Federation. It’s true we have limits on concurrent queries and concurrent connections, but we can expand these limits depending on your requirements. Additionally we plan on offering more configuration options in the future that would help with this.

That said, I’m curious if you can share more about your specific use case. Is the $lookup across databases a very common query? I wonder if there are more appropriate data modeling options that would alleviate the need for a cross DB $lookup.

Best,
Ben

Manish_Bisht · 2022-09-27T01:59:33.955Z

Hey Ben, Thanks for the reply.

yes, the $lookup is a very common query for now. The user has a “favorites” section where they maintain their favorite items. The issue is the item and user collections are being maintained in different databases because of some business constraints. Do you think moving both of them to a same DB would be a better idea? Also, if we were to do so, how would it affect the scalability of the application?

Benjamin_Flast · 2022-09-28T14:15:29.658Z

Got it. Can I ask how “long running” your $lookup queries are? Do you expect there to be contention over the 30 concurrent query limit and do you know how much we would need to raise the limit by to support your use case?

Roro · 2023-02-21T15:57:15.414Z

Hi @Benjamin_Flast
I’ve got the same issue. Not that I’m doing $lookup, but here’s our use case:

We use Online Archive for one of our collections, and archive documents older than 365 days
Still, our customers can request their documents without any date filter. In this case, we use the combined connection string (cluster + online archive). If there’s a date filter, we would use either the archive only connection string or the cluster only connection string (no issue for this last one of course)

The issue here is that 30 simultaneous queries is way too low for our needs. That would mean that we can’t allow more than 30 customers to do this kind of query, which is ridiculously low.
How can we increase this to a more “usable” value ?
Thanks in advance and have a nice day