Trying to use Relational Migrator with Kafka cluster

Marcos_Flores · 2024-05-05T19:22:27.738Z

Hi there!

I’m trying to use Relational Migrator to migrate from SQL Server to MongoDB. I want to use Kafka just for academic purposes. I alredy have a Kafka cluster running locally. I know is working fine because i used the same Kafka server with Apache Flink, also for academic purposes.

The point is, i am following this guide:https://www.mongodb.com/docs/relational-migrator/installation/kafka-deployments/install-kafka-cluster/

I also checked with Kafka Connect that the new connectors are working.

My problem is, when i try to open Relational Migrator, there’s an error next to ‘Create Migration Job’ that says that Kafka has some kind of error. If a check the kafka server, i can see this:

WARN [SocketServer listenerType=ZK_BROKER, nodeId=0] Unexpected error from /127.0.0.1 (channelId=127.0.0.1:9092-127.0.0.1:50255-2); closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 904857600)

I suppose is related to the http connection. I tried to increment the socket.request.max.bytes in the server.properties of Kafka server, but doing that caused an out of memory error.

Here is the user.properties conf

spring.profiles.active: kafka
server.port: 8278
migrator.kafka.bootstrap.servers:localhost:9092
migrator.kafka.connect.url:localhost:9092

Any suggestions?

Ty very much for your time.

Jason_Vo · 2024-05-06T00:39:28.428Z

Thanks for trying out RM using Kafka!

A few follow up questions to help you diagnose the issue:

The error shown near the ‘Create Migration Job’ button is a check which we do to attempt to validate the configuration provided. You mention that the new connectors are working, so are you able to start a job but you still see the error within the UI?
Could you describe your Kafka setup detailing how many brokers are there and the version of Kafka?
I notice the bootstrap servers and the connect url provided have the same port. Could you clarify how your Kafka Connect cluster is running?

Marcos_Flores · 2024-05-06T07:27:54.199Z

Hello

First, thank you for your time.

Let’s answer the questions:

No, i’m not able to start a job in Relational Migrator (i can actually with Apache Flink), but i can run the kafka connect command and i can see the statement in the logs, like ‘Added plugin…’.
Since im running kafka standalone locally, i only have one broker.
I think i answered your question in the previous point. I’m using kafka 3.7
Maybe i have to change the connect url for localhost:8080?

I also tried to increment the socket.request.max.bytes in the server.properties of Kafka server. This causes and out of memory error. I tried to increment the memory of the JVM with export KAFKA_HEAP_OPTS=“-Xmx4G -Xms2G” and then i was able to start a job and the error disappeared. The problem was that, although Kafka didn’t show any errors, Relational Migrator showed this:

INFO org.apache.http.impl.execchain.RetryExec [pool-1-thread-1] I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://localhost:9092: The target server failed to respond
INFO org.apache.http.impl.execchain.RetryExec [pool-1-thread-1] Retrying request to {}->http://localhost:9092

And the running job never ends.

I hope this info helps.

Ty for your time again.

Jason_Vo · 2024-05-07T01:00:50.225Z

We have a reference docker-compose file which will guide you through an example setup running in Kafka mode. https://downloads.mongodb.org/migrator/1.6.0/docker-compose-migrator-kafka.yml

Note that the Kafka connect runs a seperate service so I expect that the connect url would be different from the bootstrap server. The default port for Kafka connect is usually 8083. Can you confirm you’re running Kafka connect as a seperate service?

Hope this reference docker file helps.