MongoDB Kafka source connector - how reprocessed all documents from collection again

Andrei_Zakabluk · 2024-08-20T10:18:32.442Z

Hi ! I use MongoDB Kafka source connector and I can’t force connector reread mongodb collection again.
I deleted and recreated connector with another name. Reseted offset. Deleted topic but nothin help me.

    class: com.mongodb.kafka.connect.MongoSourceConnector
    config:
      "connection.uri": uri
      "database": db
      "collection": "products"
      "key.converter": "org.apache.kafka.connect.storage.StringConverter"
      "value.converter": "org.apache.kafka.connect.json.JsonConverter"
      "change.stream.full.document": "updateLookup"
      "pipeline": pipline
      "mongo.errors.log.enable": "true"
      "mongo.errors.deadletterqueue.topic.name": "name
      "heartbeat.interval.ms": 30000
``

Hardik_Agrawal · 2024-08-22T10:44:14.140Z

You need to set copy.existing flag for the task to read all documents from mongo first as opposed to reading from oplog directly. When all the mongo documents are read, the task will resume reading from oplog.

mongodb.com