Use multiple CDC handlers to transfer data

Rafal_Zabrowarny · 2025-03-19T12:04:43.740Z

Hi,

I would like to sync data from mongodb to second storage.
I tough I could use CDC for that - watching every change on my collection and push transformed data to secondary storage.

Can I subscribe more than 1 handler to transfer data ?
Will this result with multiplication of data in secondary storage ?

Tim_Kelly · 2025-03-20T11:26:50.812Z

Hi,

Yes, you can have multiple consumers listening to a MongoDB Change Stream, but whether this leads to data duplication depends on how you handle the events.

Do keep in mind, with multiple handlers on the same Change Stream, each handler will receive the same events. If multiple handlers write identical transformed data to secondary storage, you will likely end up with duplicate records.

To avoid duplication, if your secondary storage supports upserts you can ensure that updates are idempotent rather than inserting duplicates. I’m not sure on your exact use case but if you’re appending data (e.g., logging events), duplicates will naturally occur unless you introduce deduplication logic.

I can recommend considering message queues or streaming pipelines. So forwarding change events to something like Kafka lets you process and transform data before writing to secondary storage.

If you’re working with high-throughput data or need reliability in event processing, a message queue or stream processor is often a better approach than directly writing from multiple consumers but this is absolutely situation dependant. Does this help you?

Rafal_Zabrowarny · 2025-03-21T08:46:55.848Z

Thank you for anwser. I find it very helpful.

system · 2025-03-26T08:47:05.118Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.