Hi Team,
I am looking to track changes in my MongoDB collections. Currently, I have around 5 collections, but this number could grow to 10 or more in the future. I want to identify the best approach for this task.
I am using Python with FastAPI and currently leveraging the pymongo module to interact with MongoDB. This solution needs to be suitable for production-level deployment, so minimal overhead is crucial. While I am currently using pymongo, I am open to switching to another module if required.
Thank you for your assistance.
Hi @Omprakash_Sahu - welcome to the MongoDB community forums!
If you want to track changes across multiple collections then there are two approaches:
This will allow you to monitor all the changes across all of your collections in a single place in your code. If you have lots of changes in collections you don’t care about, then this is not going to be super-efficient, but at least it keeps all your code in one place. You can (and should!) provide an aggregation pipeline to the function so that only the events you’re interested in are sent to your application.
You’d need to have various places in your code where you call this, one for each collection you wish to monitor. This is the most efficient code. The only complexity is that you’ll need to run a thread for each collection you’re watching, so that all are watching simultaneously.
A better approach, but one that’s maybe a little far-reaching, is to switch to using async methods, which are supported by FastAPI. In this case, you would use the Motor library instead of PyMongo.
In this situation, you can start a lightweight coroutine for each collection you wish to monitor, and then any time an event is provided, you can either process each event within each coroutine, or add the event document to a queue that is being monitored by a single coroutine for handling all events.
I hope this helps!
Mark
2 Likes