APP DB doesnt appear when deploying ops manager

black_albus · 2021-06-05T18:48:49.180Z

I’m trying to deploy the ops-manager in my kubernetes cluster.
The cluster has been setup using kubeadm and has a master and a worker node.

I have been following the steps mentioned in this link.
https://www.mongodb.com/blog/post/running-mongodb-ops-manager-in-kubernetes

However, the STATE (APPDB) is always in pending. The following is the output of different commands which I feel will be helpful in further debugging:

$ kubectl get namespaces
mongodb Active 47m

$ kubectl get om -n mongodb
NAME          REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
ops-manager   1          4.4.3                          Pending                          47m

$ kubectl get sc 
NAME                  PROVISIONER           RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   nfs-dynamic-storage   Delete          WaitForFirstConsumer   false                  50m

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                           STORAGECLASS          REASON   AGE
data-ops-manager-db-0                      5Gi        RWO            Retain           Available                                   managed-nfs-storage            44m
data-ops-manager-db-1                      5Gi        RWO            Retain           Available                                   managed-nfs-storage            44m
data-ops-manager-db-2                      5Gi        RWO            Retain           Available                                   managed-nfs-storage            44m

$ kubectl get pvc -n mongodb
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
data-ops-manager-db-0   Bound    pvc-48501c9d-0582-4410-9c0d-30fa9ae24ed1   16G        RWO            managed-nfs-storage   44m
data-ops-manager-db-1   Bound    pvc-f5adc6bf-5978-4b3c-97cb-a9077363f9f6   16G        RWO            managed-nfs-storage   44m
data-ops-manager-db-2   Bound    pvc-24732b52-1656-4e12-9589-fea8f8bd47b5   16G        RWO            managed-nfs-storage   44m

$ kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-enterprise-operator-6dd9b65cdd-hq72w   1/1     Running   0          49m
ops-manager-db-0                               2/2     Running   0          48m
ops-manager-db-1                               2/2     Running   0          38m
ops-manager-db-2                               2/2     Running   0          32m

$ kubectl describe om -n mongodb
Name:         ops-manager
Namespace:    mongodb
Labels:       <none>
Annotations:  <none>
API Version:  mongodb.com/v1
Kind:         MongoDBOpsManager
Metadata:
  Creation Timestamp:  2021-06-05T17:50:37Z
  Generation:          1
  Managed Fields:
    API Version:  mongodb.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:adminCredentials:
        f:applicationDatabase:
          .:
          f:additionalMongodConfig:
            .:
            f:operationProfiling:
          f:members:
          f:podSpec:
            .:
            f:cpu:
          f:version:
        f:backup:
          .:
          f:enabled:
        f:configuration:
          .:
          f:automation.versions.source:
          f:mms.adminEmailAddr:
          f:mms.fromEmailAddr:
          f:mms.ignoreInitialUiSetup:
          f:mms.mail.hostname:
          f:mms.mail.port:
          f:mms.mail.ssl:
          f:mms.mail.transport:
          f:mms.minimumTLSVersion:
          f:mms.replyToEmailAddr:
        f:externalConnectivity:
          .:
          f:type:
        f:replicas:
        f:version:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-06-05T17:50:37Z
    API Version:  mongodb.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:applicationDatabase:
          .:
          f:lastTransition:
          f:message:
          f:observedGeneration:
          f:phase:
          f:version:
    Manager:         mongodb-enterprise-operator
    Operation:       Update
    Time:            2021-06-05T17:51:07Z
  Resource Version:  16618
  UID:               b0009425-2826-4f77-8cdf-727420385fa5
Spec:
  Admin Credentials:  adminusercredentials
  Application Database:
    Additional Mongod Config:
      Operation Profiling:
        Mode:  slowOp
    Members:   3
    Pod Spec:
      Cpu:    0.25
    Version:  4.2.6-ent
  Backup:
    Enabled:  false
  Configuration:
    automation.versions.source:  mongodb
    mms.adminEmailAddr:          support@example.com
    mms.fromEmailAddr:           support@example.com
    mms.ignoreInitialUiSetup:    true
    mms.mail.hostname:           email-smtp.us-east-1.amazonaws.com
    mms.mail.port:               465
    mms.mail.ssl:                true
    mms.mail.transport:          smtp
    mms.minimumTLSVersion:       TLSv1.2
    mms.replyToEmailAddr:        support@example.com
  External Connectivity:
    Type:    NodePort
  Replicas:  1
  Version:   4.4.3
Status:
  Application Database:
    Last Transition:      2021-06-05T18:40:08Z
    Message:              Application Database Agents haven't reached Running state yet
    Observed Generation:  1
    Phase:                Pending
    Version:
Events:                   <none>

I wanted to know if there is something that I’m missing here because of which the APP DB doesn’t appear in the running state ?

Gabriel_Gubis · 2023-10-18T07:41:21.494Z

Hi, we faced the same situation and found more hints in details of application databases pods. Just switch between containers of those pods and check the logs. I think in our case it was the agent pod which showed problems with SSL certificate - missing CA certificates in ConfigMap. After updating the config map with CAs it worked.