I set up a K8S cluster on three servers and then deployed a MongoDB Replica Set (v4.4.29) on it using a StatefulSet. After initializing the Replica Set, CRUD operations work fine and everything seems normal. However, I need to conduct a stress test, which involves shutting down all the servers, then restarting them after restoring power. After waiting for K8S to start and MongoDB pod to be ready, I check the Replica Set status, and the following error message appears.
> rs.status()
{
"ok" : 0,
"errmsg" : "Our replica set config is invalid or we are not a member of it",
"code" : 93,
"codeName" : "InvalidReplicaSetConfig"
}
>
Normally, after logging into the Replica Set from one of pods should obtain the role of Primary or Secondary, as follows.
root@mongodb-0:/# mongo
MongoDB shell version v4.4.29
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("7f82960f-6c1e-4c3e-a259-d3966ee64b3c") }
MongoDB server version: 4.4.29
---
The server generated these startup warnings when booting:
2024-10-21T08:13:33.395+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2024-10-21T08:13:34.465+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2024-10-21T08:13:34.465+00:00: You are running this process as the root user, which is not recommended
---
rs0:SECONDARY>
However, it didn’t actually become the Primary or Secondary.
root@mongodb-0:/# mongo
MongoDB shell version v4.4.29
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b40a7c2e-0404-46e8-84b8-3cee11efd21d") }
MongoDB server version: 4.4.29
---
The server generated these startup warnings when booting:
2024-10-21T08:38:16.339+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2024-10-21T08:38:17.372+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2024-10-21T08:38:17.372+00:00: You are running this process as the root user, which is not recommended
---
>
I checked the Replica Set config, all the previous settings were still here.
> rs.config().members
[
{
"_id" : 0,
"host" : "mongodb-0.mongodb-service.my_ns:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "mongodb-1.mongodb-service.my_ns:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "mongodb-2.mongodb-service.my_ns:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
]
Does anyone konw what could be causing this?