Hi, I use the Kafka Connect source connector to push to Kafka events from a change stream.
How do I configure the connector so the events are put in more than one partition, in a round robin way for instance.
My goal is to have many consumers consuming these events (hence my need for several partitions). Each event must be consumed only once.
If I create the topic myself before launching the connector, then I can have as many partitions as I need. I wish the connector could accept configuration to do it itself.
Anyway, now what I need is to distribute the events evenly to all partitions, in a round robin way. How can I do that?
EDIT: I used partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner but the result is not perfect, the events are not spread evenly on all partitions
Hi Jean
I think i have the answer to this query of yours
“If I create the topic myself before launching the connector, then I can have as many partitions as I need. I wish the connector could accept configuration to do it itself.”
I used something like
“topic.creation.enable”: “true”,
“topic.creation.default.replication.factor”: 1,
“topic.creation.default.partitions”: 1
Hope it helps you !