How to handle mongodb pod or node failures in Kubernetes?

Sudhir_Harikant · 2022-01-27T12:24:47.440Z

I am exploring deployment of mongodb in a kubernetes cluster (standalone or stateful set). I deployed a standalone mongodb POD and used a NFS share for the database volume mount. While testing a node failure scenario, I observed that the mongodb database POD remained in the terminating state and a new mongodb POD on an alive worker node remained in a retry state as the mongodb instance was not able to come online due to existing lock files in the database directory. Is this expected?
What should the deployment model for Mongodb be in Kubernetes? Does it support a node/pod failure scenario by bringing up the pod on another node in the cluster or need to use the mongodb replica feature?

Aasawari · 2022-03-04T10:36:02.494Z

Hi @Sudhir_Harikant
Welcome to the MongoDB Community!!

This is an expected behaviour from MongoDB as the mongod.lock prevents

two instances of mongod to write on the same FS.
the datapath is not accessed by multiple mongod simultaneously

It is recommended to use replica set feature for redundancy or else have a graceful shut down for the pod.
Please find the documentation here: Graceful Shut down

Let us know if you have any more questions.

Regards
Aasawari

Sudhir_Harikant · 2022-03-08T18:12:20.208Z

Hi @Aasawari ,

Thank you for confirming that behavior. This makes it clear to me now.

Regards,
Sudhir

system · 2022-03-13T18:12:55.062Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.