Cannot connect to replica set

Ops and Admin Replication
,
1

Hi,

For starters I’m sorry if I post this topic in the wrong category.
I have recently built a mongodb replica-set with two replication nodes and one arbiter.

all three nodes have a similar configuration:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /opt/mongo_path
  journal:
    enabled: true

processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

net:
  port: 27017
  bindIp: 0.0.0.0

security:
  keyFile: /opt/mongo_path_to_keyfile. 

replication:
  replSetName: MongoDB-Cluster
  enableMajorityReadConcern: false

The replica set gives the correct status for all three nodes, even when one is shut down the other one will take it over. Replication over the internal network seems to work.

Now when tying to connect from the public IP, I can get the http page for each node seperately.
Saying : "It looks like you are trying to access MongoDB over HTTP on the native driver port. "

But when I try to connect to the replicaset with following connection -string using mongodb compass :

mongodb://user:DifficultPassword@x.x.x.x:27017,x.x.x.x:27017/?authSource=admin&replicaSet=MongoDB-Set

The logs show:

2020-08-31T21:14:20.262+0000 I  NETWORK  [listener] connection accepted from My.Home.Public.IP:57252 #22 (4 connections now open)
2020-08-31T21:14:20.269+0000 I  NETWORK  [conn22] received client metadata from My.Home.Public.IP:57252 conn22: { driver: { name: "nodejs", version: "3.5.6" }, os: { type: "x", name: "x", architecture: "x", version: "18.7.0" }, platform: "'Node.js v12.4.0, LE (unified)", application: { name: "MongoDB Compass Community" } }
2020-08-31T21:14:20.394+0000 I  ACCESS   [conn22] Successfully authenticated as principal user on admin from client My.Home.Public.IP:57252
2020-08-31T21:14:20.440+0000 I  NETWORK  [conn22] end connection My.Home.Public.IP:57252 (3 connections now open)

MongoDB Compass shows “Connection timeout”.
I have been stuck on this issue for quite some time and am not able to connect to my replica-set so that if my primary goes down, I’m able to write to my secondary.

2

Hi @bjorn_massoels,

It looks like according to the log the connection is established correctly but then the client close the connection.

What is the operating system you use to run compass? What is the compass version?

Have you tried using a " fill form" compass connection and not a connection string specifying only primary?

Can you also try using the same dns/ip you have in the replica set rs.conf?

Thanks
Pacel

3

Hi @Pavel_Duchovny,

The ip addresses in my rs.conf() are IP’s of the internal private cloud subnet.
I cannot try to connect to them from outside the network.

I tried following things:
Both MongoDB Compass and Robo 3T.
In both I tried by entering a connection string with all nodes of the replicaset (primary, secondary and arbiter), a connection string with primary and secondary and the fill in form to connect to one node separately.

Are there any other things I need to consider to make the replicaset available through an internet gateway ?
I binded port 27017 to both the private ip and localhost ip of my instances.

When I create a connection string do I also add the arbiter node to the string or only primary and secondary ?

I am using MongoDB 4.2 btw.

Thanks for the help !

4

I am using both tools from a laptop outside the cloud network using Mac OSX.
The nodes use Amazon Linux 2 as OS.

5

Hi @bjorn_massoels,

Can you try to show developers tools output for this timeout in compass?

I want to see if there are any specific errors for connection to not make it through and attach the primary log of that time.

Best
Pavel

6

Hi @Pavel_Duchovny,

The Developer tools show no difference when I make a connection.
Not on the source nor on the network tab.

Can you maybe elaborate a bit on where I should look ?

7

@bjorn_massoels Is there anything on the console log?

Have you tried the mongo shell connection? Can you post this test here…

Best regards,
Pavel

8

Hi,

It seems like a connection can be made. But then it redirects to the internal IP-address, which isn’t available from the outside.

connecting to: mongodb://PublicIP:27017,PublicIP:27017/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=MongoDB-Cluster
2020-09-01T19:24:07.199+0200 I NETWORK [js] Starting new replica set monitor for MongoDB-Cluster/PublicIP:27017,PublicIP:27017
2020-09-01T19:24:07.199+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to PublicIP:27017
2020-09-01T19:24:07.199+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to PublicIP:27017
2020-09-01T19:24:07.331+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for MongoDB-Cluster is MongoDB-Cluster/PrivateIP:27017,PrivateIP:27017
2020-09-01T19:24:07.331+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to PrivateIP:27017
2020-09-01T19:24:07.331+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to PrivateIP:27017
2020-09-01T19:24:17.722+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set MongoDB-Cluster
2020-09-01T19:24:17.722+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set MongoDB-Cluster. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
2020-09-01T19:24:23.229+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set MongoDB-Cluster
2020-09-01T19:24:23.229+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set MongoDB-Cluster. Please check network connectivity and the status of the set. This has happened for 2 checks in a row.
2020-09-01T19:24:23.229+0200 E QUERY [js] Error: connect failed to replica set MongoDB-Cluster/PublicIP:27017,PublicIP:27017 :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-09-01T19:24:23.233+0200 F - [main] exception: connect failed
2020-09-01T19:24:23.233+0200 E - [main] exiting with code 1

Maybe I added the nodes in a wrong way ? Or do I have to specify their public IP somewhere ?

9

Hi @bjorn_massoels,

I think that when there is a replica set connection type the drivers/shell are getting the values based on rs.conf and build the monitor threads.

You can check by just using ip:port connection it will work.

If this is the case you need to have resolvable dns names in the replica config from yoyr client side. Files like etc/hosts might help.

Best
Pavel

1 Like
10

Thank, you helped me.
You need to add replicas on cluster like or domains, but domains must be discovered from public.

11

Just to be little more clear for future audience , <mongo-connection-url>&directConnection=true made the trick.

thanks

1 Like
12

This is only really useful from administration usage. A robust application that can continue to operate after a member failure/stepdown will need to be able to connect to all members without directConnection=true .

1 Like