Several partitions for the source connector

Jean_Francois · 2024-01-07T20:49:01.062Z

Hi, I use the Kafka Connect source connector to push to Kafka events from a change stream.
How do I configure the connector so the events are put in more than one partition, in a round robin way for instance.
My goal is to have many consumers consuming these events (hence my need for several partitions). Each event must be consumed only once.

Jean_Francois · 2024-01-07T21:18:47.187Z

If I create the topic myself before launching the connector, then I can have as many partitions as I need. I wish the connector could accept configuration to do it itself.
Anyway, now what I need is to distribute the events evenly to all partitions, in a round robin way. How can I do that?

EDIT: I used partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner but the result is not perfect, the events are not spread evenly on all partitions

Hitesh_Musale · 2024-06-11T17:50:23.148Z

Hi Jean
I think i have the answer to this query of yours

“If I create the topic myself before launching the connector, then I can have as many partitions as I need. I wish the connector could accept configuration to do it itself.”

I used something like
“topic.creation.enable”: “true”,
“topic.creation.default.replication.factor”: 1,
“topic.creation.default.partitions”: 1

Hope it helps you !