1

Hello MongoTeam,
We want to setup MongoDB Source connector to produce to Single topics from multiple collection.
Is this possible ?

So far i have tried using
“topic.namespace.map”: “{"^database_pattern_[0-9]{4}[0-1][0-9]:collection_pattern_[0-9]{4}[0-1][0-9]$": "singletopicname"}”,

But it creates multiple topics based on default config of topic creation.

FYI:
we have databases and collection name like :
database_202204.collection_202204
database_202205.collection_202205, …etc

Any help will be appreciated !!

2

After some Trial and error, I was able to achieve this using below configs:

  1. Using pipeline to listen for change events on multiple MongoDB collections, and publish them to a Kafka topic.
    “pipeline”: “[{"$match":{"$and":[{"ns.db":{"$regex":/^database_[0-9]{4}[0-1][1-9]$/}},{"ns.coll":{"$regex":/^collection_pattern_/}}]}}]”

  2. Using topic.namespace.map and pipeline to publish events from multiple source into single kafka topic.
    “topic.namespace.map”: “{"/database_\d{6}(?:\.collection_pattern_\d{6})?":"single-topicname"}”