we started to use atlas mongodb in our project with combination of kafka connect + debezium source connector.
We use private endpoints for services and debezium connection string also uses private endpoints.
The way that debezium works is that it reads config database’s shards collection
But entries in shards collection contain adresses that are only available using public endpoints which are blocked in our setup.
Basically I am able to connect to shard directly using connection string in shards collection where it is allowed to access using public links. But obviously debezium source can’t connect to shard because we only allow private links for our services (debezium included).
Is it possible for Atlas mongo to have private endpoint accessible uris in shards collection in config database?
If the connector you’re connecting with needs direct shard (mongod) level access instead of working through the mongos (sharded cluster router) tier, you will need to connect via public IP or network peering. Note that you can connect via private endpoints and also one or both of those other options for different parts of your application. Note that I am not familiar with Debezium but as a general rule if the connection isn’t going through a mongos there is risk of accessing orphan data which could lead to data quality issues.
An alternative strategy could be to explore using MongoDB’s Kafka Connector which can connect via private endpoints through the mongos directly.
Hello @Andrew_Davidson, we’re in the process of migrating to Atlas. Currently, our connectors are streaming data from MongoDB hosted on our AWS accounts. According to Debezium configuration, we’re connecting to the config servers. However, with Atlas, it seems we should connect to the mongos instead. Is this correct? Additionally, is there a method to switch or update our connectors to Atlas without any data loss?
set up a proxy within your network that act as a link between your service and MongoDB Atlas. The proxy can translate request between the private and public end point are needed, allowing your service to access the MongoDB atlas.