MongoDB Kafka Connect document delete source and sink configurations

Anjana_Jayakumar · 2023-02-03T17:10:22.139Z

Hi,
I have the following source and sink configurations for MongoDB Kafka connect. Create and update changes are reflected correctly in the change stream and the target database gets the modifications. However, deletions on the source mongodb database does not produce any tombstone messages and the changes are not reflected in the target DB either. Requesting assistance with this.

Source configuration:

{
    "name": "del_connect_source",
    "config": {
      "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
      "key.converter": "org.apache.kafka.connect.storage.StringConverter",
      "value.converter": "org.apache.kafka.connect.storage.StringConverter",
      "connection.uri": “mongodb://localhost:27017”,
      "database": "j_trials",
      "collection": "kafka_connect",
      "pipeline": "[{'$match': {'operationType': {'$in': ['insert', 'update', 'replace', 'delete'] }}}, {'$project': {'_id': 1,'fullDocument': 1,'ns': 1,} } ]",
      "publish.full.document.only": true,
      "publish.full.document.only.tombstones.on.delete": true,
      "topic.prefix": "del_",
      "topic.namespace.map": "{\"*\":\"j_trials.kafka_connect\"}",
      "copy.existing": "true",
      "tasks.max": 1,
      "mongo.errors.log.enable": true
    }
}

Sink configuration:

{
    "name": "del_connect_sink",
    "config": {
      "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
      "key.converter": "org.apache.kafka.connect.storage.StringConverter",
      "value.converter": "org.apache.kafka.connect.storage.StringConverter",
      "topics": "del_.j_trials.kafka_connect",
      "connection.uri": “mongodb://localhost:27017”,
      "database": "j_trials_sink",
      "collection": "kafka_connect",
      "schema.enable": "false"
    }
  }

Roshan_Joyce · 2024-05-31T02:24:20.675Z

I am seeing the same problem where no tombstone messages are published to the kafka topic.
Did you get to make this work? If so, please share any information you may have.

Thanks and best regards.

Roshan_Joyce · 2024-06-04T09:16:08.280Z

The problem was using the wrong configuration property name “publish.full.document.only.tombstones.on.delete”.
The correct property name to use is “publish.full.document.only.tombstone.on.delete”. And with this setting, it works as documented.

This was pointed out to me when I created the ticket [KAFKA-417] MongoSourceConnector v1.12.0 does not produce tombstone messages - MongoDB Jira.

Posting reply here so that the community is aware.