Produce events to Single kafka topic from multiple database and collections from source connector

Hitesh_Musale · 2024-03-20T13:18:29.635Z

Hello MongoTeam,
We want to setup MongoDB Source connector to produce to Single topics from multiple collection.
Is this possible ?

So far i have tried using
“topic.namespace.map”: “{"^database_pattern_[0-9]{4}[0-1][0-9]:collection_pattern_[0-9]{4}[0-1][0-9]$": "singletopicname"}”,

But it creates multiple topics based on default config of topic creation.

FYI:
we have databases and collection name like :
database_202204.collection_202204
database_202205.collection_202205, …etc

Any help will be appreciated !!

Hitesh_Musale · 2024-06-11T17:51:11.572Z

After some Trial and error, I was able to achieve this using below configs:

Using pipeline to listen for change events on multiple MongoDB collections, and publish them to a Kafka topic.
“pipeline”: “[{"$match":{"$and":[{"ns.db":{"$regex":/^database_[0-9]{4}[0-1][1-9]$/}},{"ns.coll":{"$regex":/^collection_pattern_/}}]}}]”
Using topic.namespace.map and pipeline to publish events from multiple source into single kafka topic.
“topic.namespace.map”: “{"/database_\d{6}(?:\.collection_pattern_\d{6})?":"single-topicname"}”