MongoDB kubernetes operator does not upgrade statefulset

karolizi · 2024-09-02T11:32:35.559Z

Hello all, only thing I change is replicas number in the CR yaml manifest.

That is strange because according to operator I should be able to do this. My statefulset is not scaling. I checked last applied configurations, last success apply and no any differences were there. Is this some kind of bug or what?

Using 6.0.4 MongoDB community version.
Using 0.7.8 operator version.

I tried removing annotations for existing CRD which states as failed after apply and reapply again - no result. Nothing was changed so previous setup is still online and working. However, it prompts errors without any reasons.

The error I get:
ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps “mongodb” is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden

karolizi · 2024-09-03T06:02:46.335Z

@michael_hoeller Sorry for bothering you, perhaps you could help me on this one or assign anyone?

Nam_Nguyen4 · 2024-09-03T08:22:12.793Z

Hey @karolizi thanks for reaching out. Can you either open an issue at the GH repo or follow the issue template and provide the given information?

github.com

mongodb/mongodb-kubernetes-operator/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

---
name: Bug report
about: File a report about a problem with the Operator
title: ''
labels: ''
assignees: ''

---
**What did you do to encounter the bug?**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**What did you expect?**
A clear and concise description of what you expected to happen.

**What happened instead?**
A clear and concise description of what happened instead

This file has been truncated. show original

Its very likely that you have set in your statefulset an override which is not supported.

karolizi · 2024-09-04T09:44:13.007Z

Hi @Nam_Nguyen4 I have submitted git issue with as many details as possible:

github.com/mongodb/mongodb-kubernetes-operator

Operator falsely errors and does not let to upgrade MongoDB cluster replicas

opened 09:43AM - 04 Sep 24 UTC

KarooolisZi

Hello, I was trying to adjust replicas number in the CR MongoDB yaml manifest…. **Only change** to current CR was changing **'replicas' from 2 to 3**. That is strange because according to operator I should be able to do this. My statefulset is not scaling. I checked last applied configurations, last success apply and no any differences were there. I compared these configuration to my VCS configuration. **No changes were detected**. Using **6.0.4 MongoDB community version**. Using **0.7.8 operator version**. I tried removing annotations for existing CRD which states as 'failed' after apply and reapply again - no result. Nothing was changed so previous setup is still online and working. However, it prompts errors on operator without any reasons even after applying same configuration with 2 'replicas' again. I have another environment with same specific operator and MongoDB versions. I was able to add replica and even arbiter to specs there. That was also the **only** change made to CR of MongoDB. **_The error I get_**: `ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps “mongodb” is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden` **What did you do to encounter the bug?** Steps to reproduce the behavior: 1. Changed my CR MongoDB cluster yaml manifest _database.yaml_ `spec.members: 2` to `spec.members: 3` 2. Use `kubectl apply -f database.yaml` 3. Checked MongoDB statefulset pods (no replica was added). 4. Checked MongoDB community operator logs. Observed the error. **What did you expect?** I expected operator to add an additional member to existing MongoDB cluster in statefulset. Making members number 3 instead of existing 2. **What happened instead?** Statefulset of MongoDB still had 2 members. MongoDB operator threw error (pasted in description). **Screenshots** If applicable, add screenshots to help explain your problem. **Operator Information** - 0.7.8 - docker.io/mongo:6.0.4 **Kubernetes Cluster Information** - AWS EKS - 1.28 If possible, please include: - The operator logs: `sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227 2024-09-04T07:11:25.133Z INFO controllers/replica_set_controller.go:137 Reconciling MongoDB {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.134Z DEBUG controllers/replica_set_controller.go:139 Validating MongoDB.Spec {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.134Z DEBUG controllers/replica_set_controller.go:148 Ensuring the service exists {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.134Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.142Z INFO controllers/replica_set_controller.go:463 Create/Update operation succeeded {"ReplicaSet": "mongodb-<NAME>/mongodb", "operation": "updated"} 2024-09-04T07:11:25.142Z DEBUG controllers/replica_set_controller.go:409 Scaling up the ReplicaSet, the StatefulSet must be updated first {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.142Z INFO controllers/replica_set_controller.go:330 Creating/Updating StatefulSet {"ReplicaSet": "mongodb-<NAME>/mongodb"} 2024-09-04T07:11:25.151Z ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps "mongodb" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden github.com/mongodb/mongodb-kubernetes-operator/controllers.messageOption.ApplyOption /workspace/controllers/mongodb_status_options.go:104 github.com/mongodb/mongodb-kubernetes-operator/pkg/util/status.Update /workspace/pkg/util/status/status.go:25 github.com/mongodb/mongodb-kubernetes-operator/controllers.ReplicaSetReconciler.Reconcile /workspace/controllers/replica_set_controller.go:200 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227` - Below we assume that your replicaset database pods are named `mongo-<>`. For instance: ``` ❯ k get pods NAME READY STATUS RESTARTS AGE mongo-0 2/2 Running 0 19h mongo-1 2/2 Running 0 19h ❯ k get mdbc NAME PHASE VERSION mongo Running 4.4.0 ``` - yaml definitions of your MongoDB Deployment(s): - `kubectl get mdbc -oyaml` ``` apiVersion: v1 items: - apiVersion: mongodbcommunity.mongodb.com/v1 kind: MongoDBCommunity metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"mongodb","namespace":"mongodb-<SENSITIVE>"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":3,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"NodeGroup","operator":"In","values":["<SENSITIVE>"]}]}]}},"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["mongodb"]}]},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"name":"mongod","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"500m","memory":"1Gi"}}}]}},"volumeClaimTemplates":[{"metadata":{"name":"data-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"70G"}},"storageClassName":"ebs-sc"}},{"metadata":{"name":"logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10G"}},"storageClassName":"ebs-sc"}}]}},"type":"ReplicaSet","users":[{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"read"},{"db":"<SENSITIVE>","name":"read"},{"db":"<SENSITIVE>","name":"changestreamrole"}],"scramCredentialsSecretName":"<SENSITIVE>"}],"version":"6.0.4"}} mongodb.com/v1.lastAppliedMongoDBVersion: 6.0.4 creationTimestamp: "2024-01-03T07:47:03Z" generation: 18 labels: k8slens-edit-resource-version: v1 name: mongodb namespace: mongodb-<SENSITIVE> resourceVersion: "386378973" uid: 8dbc92a1-061b-4ebb-a2be-d1b5dd6d696b spec: additionalMongodConfig: storage.wiredTiger.engineConfig.journalCompressor: zlib members: 3 security: authentication: ignoreUnknownUsers: true modes: - SCRAM statefulSet: spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: NodeGroup operator: In values: - <SENSITIVE> podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - mongodb topologyKey: kubernetes.io/hostname weight: 100 containers: - name: mongod resources: limits: cpu: "1" memory: 2Gi requests: cpu: 500m memory: 1Gi volumeClaimTemplates: - metadata: name: data-volume spec: accessModes: - ReadWriteOnce resources: requests: storage: 70G storageClassName: ebs-sc - metadata: name: logs-volume spec: accessModes: - ReadWriteOnce resources: requests: storage: 10G storageClassName: ebs-sc type: ReplicaSet users: - db: <SENSITIVE> name: <SENSITIVE> passwordSecretRef: name: <SENSITIVE> roles: - db: <SENSITIVE> name: clusterAdmin - db: <SENSITIVE> name: userAdminAnyDatabase scramCredentialsSecretName: <SENSITIVE> - db: <SENSITIVE> name: <SENSITIVE> passwordSecretRef: name: <SENSITIVE> roles: - db: <SENSITIVE> name: readWrite scramCredentialsSecretName: <SENSITIVE> - db: <SENSITIVE> name: <SENSITIVE> passwordSecretRef: name: <SENSITIVE> roles: - db: <SENSITIVE> name: readWrite scramCredentialsSecretName: <SENSITIVE> - db: <SENSITIVE> name: <SENSITIVE> passwordSecretRef: name: <SENSITIVE> roles: - db: <SENSITIVE> name: readWrite - db: <SENSITIVE> name: readWrite scramCredentialsSecretName: backup-user-scram - db: <SENSITIVE> name: <SENSITIVE> passwordSecretRef: name: <SENSITIVE> roles: - db: <SENSITIVE> name: dbAdmin - db: <SENSITIVE> name: userAdminAnyDatabase - db: <SENSITIVE> name: readWrite - db: <SENSITIVE> name: readWrite - db: <SENSITIVE> name: readWrite scramCredentialsSecretName: <SENSITIVE> - db: <SENSITIVE> name: <SENSITIVE> passwordSecretRef: name: <SENSITIVE> roles: - db: <SENSITIVE> name: read - db: <SENSITIVE> name: read - db: <SENSITIVE> name: changestreamrole scramCredentialsSecretName: <SENSITIVE> version: 6.0.4 status: currentMongoDBMembers: 2 currentStatefulSetReplicas: 2 message: 'Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps "mongodb" is invalid: spec: Forbidden: updates to statefulset spec for fields other than ''replicas'', ''ordinals'', ''template'', ''updateStrategy'', ''persistentVolumeClaimRetentionPolicy'' and ''minReadySeconds'' are forbidden' mongoUri: <SENSITIVE>,<SENSITIVE>/?replicaSet=mongodb phase: Failed version: 6.0.4 kind: List metadata: resourceVersion: "" ``` - yaml definitions of your kubernetes objects like the statefulset(s), pods (we need to see the state of the containers): - `kubectl get sts -oyaml` ``` apiVersion: v1 items: - apiVersion: apps/v1 kind: StatefulSet metadata: creationTimestamp: "2024-01-03T07:47:03Z" generation: 31 labels: app: mongodb-<SENSITIVE> name: mongodb namespace: mongodb-<SENSITIVE> ownerReferences: - apiVersion: mongodbcommunity.mongodb.com/v1 blockOwnerDeletion: true controller: true kind: MongoDBCommunity name: mongodb uid: <SENSITIVE> resourceVersion: "<SENSITIVE>" uid: <SENSITIVE> spec: persistentVolumeClaimRetentionPolicy: whenDeleted: Retain whenScaled: Retain podManagementPolicy: OrderedReady replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app: mongodb-<SENSITIVE> serviceName: mongodb-<SENSITIVE> template: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2024-09-02T08:55:35Z" creationTimestamp: null labels: app: mongodb-<SENSITIVE> spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: NodeGroup operator: In values: - <SENSITIVE> podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - mongodb topologyKey: kubernetes.io/hostname weight: 100 containers: - command: - /bin/sh - -c - |2+ #run post-start hook to handle version changes /hooks/version-upgrade # wait for config and keyfile to be created by the agent while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ; # start mongod with this configuration exec mongod -f /data/automation-mongod.conf; env: - name: AGENT_STATUS_FILEPATH value: /healthstatus/agent-health-status.json image: docker.io/mongo:6.0.4 imagePullPolicy: IfNotPresent name: mongod resources: limits: cpu: "1" memory: 2Gi requests: cpu: 500m memory: 1Gi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /data name: data-volume - mountPath: /healthstatus name: healthstatus - mountPath: /hooks name: hooks - mountPath: /var/log/mongodb-mms-automation name: logs-volume - mountPath: /var/lib/mongodb-mms-automation/authentication name: mongodb-keyfile - mountPath: /tmp name: tmp - command: - /bin/bash - -c - |- current_uid=$(id -u) AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)" declare -r current_uid if ! grep -q "${current_uid}" /etc/passwd ; then sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd export NSS_WRAPPER_PASSWD=/tmp/passwd export LD_PRELOAD=libnss_wrapper.so export NSS_WRAPPER_GROUP=/etc/group fi agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs 24 -logLevel DEBUG env: - name: AGENT_STATUS_FILEPATH value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json - name: AUTOMATION_CONFIG_MAP value: mongodb-config - name: HEADLESS_AGENT value: "true" - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1 imagePullPolicy: Always name: mongodb-agent readinessProbe: exec: command: - /opt/scripts/readinessprobe failureThreshold: 40 initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "1" memory: 500M requests: cpu: 500m memory: 400M securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /opt/scripts name: agent-scripts - mountPath: /var/lib/automation/config name: automation-config readOnly: true - mountPath: /data name: data-volume - mountPath: /var/log/mongodb-mms-automation/healthstatus name: healthstatus - mountPath: /var/log/mongodb-mms-automation name: logs-volume - mountPath: /var/lib/mongodb-mms-automation/authentication name: mongodb-keyfile - mountPath: /tmp name: tmp dnsPolicy: ClusterFirst initContainers: - command: - cp - version-upgrade-hook - /hooks/version-upgrade image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6 imagePullPolicy: Always name: mongod-posthook resources: {} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /hooks name: hooks - command: - cp - /probes/readinessprobe - /opt/scripts/readinessprobe image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12 imagePullPolicy: Always name: mongodb-agent-readinessprobe resources: {} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /opt/scripts name: agent-scripts restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccount: mongodb-<SENSITIVE> serviceAccountName: mongodb-<SENSITIVE> terminationGracePeriodSeconds: 30 volumes: - emptyDir: {} name: agent-scripts - name: automation-config secret: defaultMode: 416 secretName: mongodb-<SENSITIVE> - emptyDir: {} name: healthstatus - emptyDir: {} name: hooks - emptyDir: {} name: mongodb-<SENSITIVE> - emptyDir: {} name: tmp updateStrategy: type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null name: data-volume spec: accessModes: - ReadWriteOnce resources: requests: storage: 50G storageClassName: ebs-sc volumeMode: Filesystem status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null name: logs-volume spec: accessModes: - ReadWriteOnce resources: requests: storage: 10G storageClassName: ebs-sc volumeMode: Filesystem status: phase: Pending status: availableReplicas: 2 collisionCount: 0 currentReplicas: 2 currentRevision: mongodb-<SENSITIVE> observedGeneration: 31 readyReplicas: 2 replicas: 2 updateRevision: mongodb-<SENSITIVE> updatedReplicas: 2 kind: List metadata: resourceVersion: "" ``` - `kubectl get pods -oyaml` ``` apiVersion: v1 items: - apiVersion: v1 kind: Pod metadata: annotations: agent.mongodb.com/version: "6" kubectl.kubernetes.io/restartedAt: "2024-09-02T08:55:35Z" creationTimestamp: "2024-09-02T08:56:31Z" generateName: mongodb- labels: app: mongodb-<SENSITIVE> apps.kubernetes.io/pod-index: "0" controller-revision-hash: mongodb-<SENSITIVE> statefulset.kubernetes.io/pod-name: <SENSITIVE> name: mongodb-0 namespace: mongodb-<SENSITIVE> ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: StatefulSet name: mongodb uid: 934952c3-22f8-4b93-a6e3-34c85bc85fc0 resourceVersion: "386287077" uid: 5c442610-203d-45ca-b280-dec81b9a923d spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: NodeGroup operator: In values: - <SENSITIVE> podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - mongodb topologyKey: kubernetes.io/hostname weight: 100 containers: - command: - /bin/sh - -c - |2+ #run post-start hook to handle version changes /hooks/version-upgrade # wait for config and keyfile to be created by the agent while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ; # start mongod with this configuration exec mongod -f /data/automation-mongod.conf; env: - name: AGENT_STATUS_FILEPATH value: /healthstatus/agent-health-status.json image: docker.io/mongo:6.0.4 imagePullPolicy: IfNotPresent name: mongod resources: limits: cpu: "1" memory: 2Gi requests: cpu: 500m memory: 1Gi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /data name: <SENSITIVE> - mountPath: /healthstatus name: healthstatus - mountPath: /hooks name: hooks - mountPath: /var/log/mongodb-mms-automation name: logs-volume - mountPath: /var/lib/mongodb-mms-automation/authentication name: <SENSITIVE> - mountPath: /tmp name: <SENSITIVE> - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-t6sf8 readOnly: true - command: - /bin/bash - -c - |- current_uid=$(id -u) AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)" declare -r current_uid if ! grep -q "${current_uid}" /etc/passwd ; then sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd export NSS_WRAPPER_PASSWD=/tmp/passwd export LD_PRELOAD=libnss_wrapper.so export NSS_WRAPPER_GROUP=/etc/group fi agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs 24 -logLevel DEBUG env: - name: AGENT_STATUS_FILEPATH value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json - name: AUTOMATION_CONFIG_MAP value: mongodb-config - name: HEADLESS_AGENT value: "true" - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1 imagePullPolicy: Always name: mongodb-agent readinessProbe: exec: command: - /opt/scripts/readinessprobe failureThreshold: 40 initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "1" memory: 500M requests: cpu: 500m memory: 400M securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /opt/scripts name: agent-scripts - mountPath: /var/lib/automation/config name: automation-config readOnly: true - mountPath: /data name: data-volume - mountPath: /var/log/mongodb-mms-automation/healthstatus name: healthstatus - mountPath: /var/log/mongodb-mms-automation name: logs-volume - mountPath: /var/lib/mongodb-mms-automation/authentication name: mongodb-keyfile - mountPath: /tmp name: tmp - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-t6sf8 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostname: <SENSITIVE> initContainers: - command: - cp - version-upgrade-hook - /hooks/version-upgrade image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6 imagePullPolicy: Always name: mongod-posthook resources: {} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /hooks name: hooks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-t6sf8 readOnly: true - command: - cp - /probes/readinessprobe - /opt/scripts/readinessprobe image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12 imagePullPolicy: Always name: mongodb-agent-readinessprobe resources: {} securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /opt/scripts name: agent-scripts - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-t6sf8 readOnly: true nodeName: <SENSITIVE> preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: <SENSITIVE> runAsNonRoot: true runAsUser: <SENSITIVE> serviceAccount: <SENSITIVE> serviceAccountName: <SENSITIVE> subdomain: <SENSITIVE> terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: data-volume persistentVolumeClaim: claimName: data-volume-mongodb-0 - name: logs-volume persistentVolumeClaim: claimName: logs-volume-mongodb-0 - emptyDir: {} name: agent-scripts - name: automation-config secret: defaultMode: 416 secretName: <SENSITIVE> - emptyDir: {} name: healthstatus - emptyDir: {} name: hooks - emptyDir: {} name: <SENSITIVE> - emptyDir: {} name: tmp - name: kube-api-access-t6sf8 projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: <SENSITIVE> - configMap: items: - key: <SENSITIVE> path: <SENSITIVE> name: <SENSITIVE> - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2024-09-02T08:56:38Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2024-09-02T08:56:55Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2024-09-02T08:56:55Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2024-09-02T08:56:31Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://5a01d0a5116ac5af31cee1f7b0fe6b516d22b5e223042adfb035ee3fcaf7d284 image: docker.io/library/mongo:6.0.4 imageID: docker.io/library/mongo@sha256:a4f2db6f54aeabba562cd07e5cb758b55d6192dcc6f36322a334ba0b0120aaf1 lastState: {} name: mongod ready: true restartCount: 0 started: true state: running: startedAt: "2024-09-02T08:56:38Z" - containerID: containerd://fab2952924c2028fab4117bb26aaf53715a1993792a5730226823925fb9146db image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1 imageID: quay.io/mongodb/mongodb-agent@sha256:00dc01552af1eac8a457d7214ae4f913dd2a14409c0125dead25808e40cc0d62 lastState: {} name: mongodb-agent ready: true restartCount: 0 started: true state: running: startedAt: "2024-09-02T08:56:38Z" hostIP: <SENSITIVE> initContainerStatuses: - containerID: containerd://42c48fb8a9556f01b3ab8d61b985aec62c4e177dc53b2b65e0afecaa3991dbe0 image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6 imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:5c1483fa22cfb772f186a30180fcfa4af91ff3a638b28a37cc9f997f8ac046f9 lastState: {} name: mongod-posthook ready: true restartCount: 0 started: false state: terminated: containerID: containerd://42c48fb8a9556f01b3ab8d61b985aec62c4e177dc53b2b65e0afecaa3991dbe0 exitCode: 0 finishedAt: "2024-09-02T08:56:36Z" reason: Completed startedAt: "2024-09-02T08:56:36Z" - containerID: containerd://144d11be92c06da8c8c69faa841708f4caf73ebb9a9255cecfa499e8a05befa4 image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12 imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:ef6c144de962bc976161b41972cc85988189870e05ad4ada0798200c0a988547 lastState: {} name: mongodb-agent-readinessprobe ready: true restartCount: 0 started: false state: terminated: containerID: containerd://144d11be92c06da8c8c69faa841708f4caf73ebb9a9255cecfa499e8a05befa4 exitCode: 0 finishedAt: "2024-09-02T08:56:37Z" reason: Completed startedAt: "2024-09-02T08:56:37Z" phase: Running podIP: <SENSITIVE> podIPs: - ip: <SENSITIVE> qosClass: Burstable startTime: "2024-09-02T08:56:31Z" kind: List metadata: resourceVersion: "" ``` - The agent clusterconfig of the faulty members: - `kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json` ``` {"version":6,"processes":[{"name":"mongodb-0","disabled":false,"hostname":"<SENSITIVE>","args2_6":{"net":{"port":27017},"repli cation":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"6.0","processTyp e":"mongod","version":"6.0.4","authSchemaVersion":5},{"name":"mongodb-1","disabled":false,"hostname":"<SENSITIVE>","args2_6":{ "net":{"port":27017},"replication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilit yVersion":"6.0","processType":"mongod","version":"6.0.4","authSchemaVersion":5}],"replicaSets":[{"_id":"mongodb","members":[{"_id":0,"host":"mongodb-0","priority":1,"arbi terOnly":false,"votes":1},{"_id":1,"host":"mongodb-1","priority":1,"arbiterOnly":false,"votes":1}],"protocolVersion":"1","numberArbiters":0}],"auth":{"usersWanted":[{"mec hanisms":[],"roles":[{"role":"clusterAdmin","<SENSITIVE>":"<SENSITIVE>"},{"role":"userAdminAnyDatabase","<SENSITIVE>":"<SENSITIVE>"}],"user":"<SENSITIVE>","<SENSITIVE>":"<SENSITIVE>","authenticationRestrictions":[]," scramSha256Creds":{"iterationCount":15000,"salt":"WioeMJQXT8w9Coif5Fq3gqV1OfeDi64Bvq/maw==","serverKey":"iK8SJLwCzmk95+mUePC3wrpGw29Tfx9vN+ZCKSMPKMM=","storedKey":"Z33GU9 ix2W++nlnkFBIbYP7kEATZ/6sDVQqdhEd+tT0="},"scramSha1Creds":{"iterationCount":10000,"salt":"Q9mmbNXpyLDRtYmoln1xgA==","serverKey":"AmaNP+YmbrNf23l8URaZAZKKOz0=","storedKey" :"0d8SscAfTMph+2aW416TXB1/UZw="}},{"mechanisms":[],"roles":[{"role":"readWrite","<SENSITIVE>":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha 256Creds":{"iterationCount":15000,"salt":"GOkTsWgdrct5KSSQtTHC20myEJM76v5OMEGXOA==","serverKey":"ksqF9YIWnI50+bQJhjl0/zA1a0H0UpcNnzxnFEjciV4=","storedKey":"GyxjpwCp9hTHsK 5CX2ObkIs73NP2zL1VrwbCQTLDvGE="},"scramSha1Creds":{"iterationCount":10000,"salt":"4RAcRyAxnRCQQhcHWRDA2w==","serverKey":"59KQ8PQV/rS4zxSuVca/tQbDNWw=","storedKey":"v68O4b x8u7/RNIks1WBvmXIJ+H8="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha256Creds ":{"iterationCount":15000,"salt":"vYX0jOTF0NvPPmpQm1oz/b7v1/sAnFOMWdm5Pg==","serverKey":"MWuSeUedUk33YD57g/pVw+kV89vQK8OmTibRLl2hR0U=","storedKey":"ffXuxQ5HTf0FcH2FdcNvKi gWSO/TgPdF0elXk9iYX3E="},"scramSha1Creds":{"iterationCount":10000,"salt":"zV90H0Z2XJ8sCiupCSK3PQ==","serverKey":"IwdxN4BVrGSqyLPXDhrbZKFsbtc=","storedKey":"e5XxJTwdueUyUy Jd3ioqQFuEKbc="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions ":[],"scramSha256Creds":{"iterationCount":15000,"salt":"0HwrZJa9FIMCy5r4erCq7o2gb/RSaHofCV1XMw==","serverKey":"4FTg/fstci+8W6BZE4jfyXpLJr9/f4zsuDiKrLnBcgg=","storedKey":" lThVu1E2tv14Q7H58DYYNK1jlqXaIZDCp/Omp44wR1A="},"scramSha1Creds":{"iterationCount":10000,"salt":"gDxOqOLC16/e/WvhWSGDdA==","serverKey":"q6SKd30cOY+PFQnqRFMpdgmNTFA=","stor edKey":"9swpmpNkjLofRRRprZWGrCBoolk="}},{"mechanisms":[],"roles":[{"role":"dbAdmin","db":"<SENSITIVE>"},{"role":"userAdminAnyDatabase","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha256Creds":{"iterationC ount":15000,"salt":"m40BqXb1jbVT4NIN9NjAdbTdQp84O6KtEbRjgA==","serverKey":"hztnbCDXJs0zBcwtaJsLquRtEgCHDykKj04SaQ3eLn8=","storedKey":"cftguOpTPNr4QYvL2XtrRlybPzi96CgzGoZ9 1EVZg2g="},"scramSha1Creds":{"iterationCount":10000,"salt":"SofiWm+P4s3RwvvIxflOOQ==","serverKey":"73knk0VrQPm6PWSYM5PFYwUK1lA=","storedKey":"wnYGbRIv2qPtcpv3j4r+lUX8x/4= "}},{"mechanisms":[],"roles":[{"role":"read","db":"<SENSITIVE>"},{"role":"read","db":"<SENSITIVE>"},{"role":"changestreamrole","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authentica tionRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"wzzV+4JFqwIGw1mAzRucb1oiIcYYR/gdcwZyJw==","serverKey":"Ty105G/oxXhrv9UwgIqqXHO7ZxM5LYW9T/mta7uiQYo =","storedKey":"63E/7kQy2g4O/MUd0a62q8pQNBtITkJ74dUagrRESO4="},"scramSha1Creds":{"iterationCount":10000,"salt":"R/PWGnO94tygxdNjIavdkQ==","serverKey":"Gr8vtcLPyb/pR/h8GRi vOrY6/sE=","storedKey":"kwWqu/dnDzNTDOEu03TcgeDWxPY="}}],"disabled":false,"authoritativeSet":false,"autoAuthMechanisms":["SCRAM-SHA-256"],"autoAuthMechanism":"SCRAM-SHA-2 56","deploymentAuthMechanisms":["SCRAM-SHA-256"],"autoUser":"mms-automation","key":"8tQDoV1eZdKJvpc7cA8rtu939Glj1IgsL9CNE1nf7SuZMFw8Te47PmhA9Z1NPi27cRw5+bs16kenEAPP82V7v5 1Xcv5Q9xPZKUxltKlc3t9cfq2Q7Il42DJsrjhQUhne5lKNghWLRHPSFVb8IHbuImgPcvu7mPz6VsYClu6Lno5ewW3ziIvilIW/2xvpxqG0qz4jvz5/cmtTWeNn7VJzNOYwYurWdFfvdUDL+Z+kQqcbsa95SSYA8217h6aKE2gu OwlVpK0VZBYCPACg+ID1dARawAHG7xCA92lFttymLfgu8kbUXeW6RxBsgwz5iuOXjiIrm8XpWhpWHLJNplf5YaGsqBMIbRlH3tAXGv6auqLaiGup3+kQXDJNwC7Juaa5F0FGXg+PdQPMOH4xv2SZy0zGHh988CaEtXhBVWiQ06 FhnNWyxziLCl8BGJpCbD2bsGiiUBcUHvxkCybARhguLYdnS60+tlJcMIr3rpt7MTgRuHhwki0gX1KcVmEe+tPeg57RdqcVcEEqqHYwc4Ghkk/PF/10BlsO0NiUZmJxZqow7ffSRZHtZ/VKW2og6CZp2V3BaYZmzYwHn5XFFRCD NUu8mbwvtySQVSlVVY4GbKRkgepYsWrYGc20yPH7Hzni9b8N0zCmX8HPy5icn8+jf4z7BRw=","keyfile":"/var/lib/mongodb-mms-automation/authentication/keyfile","keyfileWindows":"%SystemDriv e%\\MMSAutomation\\versions\\keyfile","autoPwd":"eX-rXNR2PB_ytwagyylk"},"tls":{"CAFilePath":"","clientCertificateMode":"OPTIONAL"},"mongoDbVersions":[{"name":"6.0.4","bui lds":[{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"rhel","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":""," gitVersion":"","architecture":"amd64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]}]}],"backupVersions":[],"monitoringVersions":[],"options":{"downl oadBase":"/var/lib/mongodb-mms-automation"}} ``` - The agent health status of the faulty members: - `kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json` ``` {"statuses":{"mongodb-0":{"IsInGoalState":true,"LastMongoUpTime":1725441832,"ExpectedToBeUp":true,"ReplicationStatus":1}},"mmsStatus":{"mongodb-0":{"name":"mongodb-0","la stGoalVersionAchieved":6,"plans":[{"automationConfigVersion":6,"started":"2024-09-02T08:56:39.289177326Z","completed":"2024-09-02T08:56:44.316657181Z","moves":[{"move":"S tart","moveDoc":"Start the process","steps":[{"step":"StartFresh","stepDoc":"Start a mongo instance (start fresh)","isWaitStep":false,"started":"2024-09-02T08:56:39.2891 92036Z","completed":"2024-09-02T08:56:43.2738461Z","result":"success"}]},{"move":"WaitAllRsMembersUp","moveDoc":"Wait until all members of this process' repl set are up", "steps":[{"step":"WaitAllRsMembersUp","stepDoc":"Wait until all members of this process' repl set are up","isWaitStep":true,"started":"2024-09-02T08:56:43.273886773Z","co mpleted":"2024-09-02T08:56:43.274018861Z","result":"success"}]},{"move":"RsInit","moveDoc":"Initialize a replica set including the current MongoDB process","steps":[{"ste p":"RsInit","stepDoc":"Initialize a replica set","isWaitStep":false,"started":"2024-09-02T08:56:43.274046073Z","completed":null,"result":"error"}]},{"move":"WaitFeatureCo mpatibilityVersionCorrect","moveDoc":"Wait for featureCompatibilityVersion to be right","steps":[{"step":"WaitFeatureCompatibilityVersionCorrect","stepDoc":"Wait for feat ureCompatibilityVersion to be right","isWaitStep":true,"started":null,"completed":null,"result":""}]}]},{"automationConfigVersion":6,"started":"2024-09-03T12:29:37.736565 486Z","completed":"2024-09-03T12:29:42.80790282Z","moves":[{"move":"EnsureAutomationCredentials","moveDoc":"Ensure the automation user exists","steps":[{"step":"EnsureAut omationCredentials","stepDoc":"Ensure the automation user exists","isWaitStep":false,"started":"2024-09-03T12:29:37.736594109Z","completed":"2024-09-03T12:29:42.806979734 Z","result":"success"}]}]}],"errorCode":0,"errorString":""}}} ``` - The verbose agent logs of the faulty members: - `kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log` ``` [2024-09-03T12:57:47.152+0000] [.debug] [src/mongoctl/processctl.go:ProcessType:1306] <mongodb-0> [12:57:47.152] Checking process type of mongodb-0.mongodb-svc.mongodb-su rplus.svc.cluster.local:27017 (local=false)... [2024-09-03T12:57:47.152+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.152] Starting RunCommand(dbName=admin, cmd=[{isMaste r 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.153] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{isMa ster 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signature":{" hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"connectionId":13,"electionId":"7fffffff000000000000008d","hosts":["mongodb-0.mong odb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","mongodb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017"],"ismaster":true,"lastWrite":{"lastWriteDate":"2024-09-03 T12:57:47Z","majorityOpTime":{"t":141,"ts":{"T":1725368267,"I":27}},"majorityWriteDate":"2024-09-03T12:57:47Z","opTime":{"t":141,"ts":{"T":1725368267,"I":27}}},"localTime ":"2024-09-03T12:57:47.153Z","logicalSessionTimeoutMinutes":30,"maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":17,"maxWriteBatchSize":100000, "me":"mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","minWireVersion":0,"ok":1,"operationTime":{"T":1725368267,"I":27},"primary":"mongodb-0.mongodb-svc.mo ngodb-<SENSITIVE>.svc.cluster.local:27017","readOnly":false,"secondary":false,"setName":"mongodb","setVersion":1,"topologyVersion":{"counter":10,"processId":"66d57dc915c38471 0a9d334d"}} [2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.153] Starting RunCommand(dbName=admin, cmd=[{getPara meter 1} {featureCompatibilityVersion 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.153] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP arameter 1} {featureCompatibilityVersion 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{" T":1725368267,"I":27},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"featureCompatibilityVersion":{"version":"6.0" },"ok":1,"operationTime":{"T":1725368267,"I":27}} [2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:FeatureCompatibilityVersion:3796] <mongodb-0> [12:57:47.153] FeatureCompatibilityVersion on 3.6+ is 6. 0 [2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:ParsedArgs:2414] <mongodb-0> [12:57:47.153] Getting parsed args [2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.154] Starting RunCommand(dbName=admin, cmd=[{isMaste r 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.154] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{isMa ster 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signature":{" hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"connectionId":13,"electionId":"7fffffff000000000000008d","hosts":["mongodb-0.mong odb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","mongodb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017"],"ismaster":true,"lastWrite":{"lastWriteDate":"2024-09-03 T12:57:47Z","majorityOpTime":{"t":141,"ts":{"T":1725368267,"I":27}},"majorityWriteDate":"2024-09-03T12:57:47Z","opTime":{"t":141,"ts":{"T":1725368267,"I":27}}},"localTime ":"2024-09-03T12:57:47.154Z","logicalSessionTimeoutMinutes":30,"maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":17,"maxWriteBatchSize":100000, "me":"mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","minWireVersion":0,"ok":1,"operationTime":{"T":1725368267,"I":27},"primary":"mongodb-0.mongodb-svc.mo ngodb-<SENSITIVE>.svc.cluster.local:27017","readOnly":false,"secondary":false,"setName":"mongodb","setVersion":1,"topologyVersion":{"counter":10,"processId":"66d57dc915c38471 0a9d334d"}} [2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.154] Starting RunCommand(dbName=admin, cmd=[{getCmdL ineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.155] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getC mdLineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signatu re":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"argv":["mongod","-f","/data/automation-mongod.conf"],"ok":1,"operationTime" :{"T":1725368267,"I":27},"parsed":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"au thorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath" :"/data","engine":"wiredTiger","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}} [2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:overwriteWithRedactedArgsFromConfig:2495] <mongodb-0> [12:57:47.155] Attempting to overwrite redacted args with real value of redacted args from config file [/data/automation-mongod.conf] for host [27017] [2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:overwriteWithRedactedArgsFromConfig:2506] <mongodb-0> [12:57:47.155] Overwrote redacted args with real value of redacted args from config file [/data/automation-mongod.conf] for host [27017] [2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.155] Starting RunCommand(dbName=admin, cmd=[{getPara meter 1} {tlsMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.156] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP arameter 1} {tlsMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28 },"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"ok":1,"operationTime":{"T":1725368267,"I":28},"tlsMode":"disabled "} [2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.156] Starting RunCommand(dbName=admin, cmd=[{getPara meter 1} {clusterAuthMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.156] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP arameter 1} {clusterAuthMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":172536826 7,"I":28},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"clusterAuthMode":"keyFile","ok":1,"operationTime":{"T":17 25368267,"I":28}} [2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.157] Starting RunCommand(dbName=admin, cmd=[{getDefa ultRWConcern 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.157] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getD efaultRWConcern 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28},"si gnature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"defaultReadConcern":{"level":"local"},"defaultReadConcernSource":"impl icit","defaultWriteConcern":{"w":"majority","wtimeout":0},"defaultWriteConcernSource":"implicit","localUpdateWallClockTime":"2024-09-03T12:57:47.157Z","ok":1,"operationTi me":{"T":1725368267,"I":28}} [2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.157] Starting RunCommand(dbName=admin, cmd=[{getCmdL ineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.158+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.158] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getC mdLineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28},"signatu re":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"argv":["mongod","-f","/data/automation-mongod.conf"],"ok":1,"operationTime" :{"T":1725368267,"I":28},"parsed":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"au thorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath" :"/data","engine":"wiredTiger","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}} [2024-09-03T12:57:47.159+0000] [.info] [src/mongoclientservice/mongoclientservice.go:func1:1619] [12:57:47.159] Testing auth with username __system db=local to mongodb-0. mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) connectMode=SingleConnect ipversion=0 tls=false [2024-09-03T12:57:47.169+0000] [.info] [src/mongoctl/processctl.go:GetKeyHashes:2080] <mongodb-0> [12:57:47.169] Able to successfully auth to mongodb-0.mongodb-svc.mongod b-<SENSITIVE>.svc.cluster.local:27017 (local=false) using desired auth key [2024-09-03T12:57:47.169+0000] [.debug] [state/stateutil/fickleutil.go:CurrentFickle:51] <mongodb-0> [12:57:47.169] Starting CurrentFickle... [2024-09-03T12:57:47.169+0000] [.debug] [src/mongoctl/replsetctl.go:ReplSetStatus:285] <mongodb-0> [12:57:47.169] Getting repl set status for process mongodb-0.mongodb-sv c.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false)... [2024-09-03T12:57:47.169+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.169] Starting RunCommand(dbName=admin, cmd=[{replSet GetStatus 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ... [2024-09-03T12:57:47.170+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.170] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{repl SetGetStatus 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"members":[{"_id":0,"name":"mongodb-0.mongodb-svc.mongodb-su rplus.svc.cluster.local:27017","health":1,"state":1,"self":true,"optime":{"ts":{"T":1725368267,"I":29},"t":141},"optimedate":"2024-09-03T12:57:47Z"},{"_id":1,"name":"mong odb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","health":1,"state":2,"self":false,"optime":{"ts":{"T":1725368266,"I":67},"t":141},"optimedate":"2024-09-03T12:5 7:46Z"}],"errmsg":"","myState":1,"optimes":{"lastCommittedOpTime":{"ts":{"T":1725368267,"I":28},"t":141},"appliedOpTime":{"ts":{"T":1725368267,"I":29},"t":141},"durableOp Time":{"ts":{"T":1725368267,"I":29},"t":141}}} ```

Nam_Nguyen4 · 2024-09-04T10:14:01.107Z

Hey,

thanks a lot, that is great! Answered there.

karolizi · 2024-09-06T11:32:05.435Z

@Nam_Nguyen4 @michael_hoeller Hi, sorry for disturbing you but I think operator still works quirky tried adding arbiter to PS replicaset to make it PSA. 2 members are fine but after adding 1 arbiter, operator goes into some crashloop with agent goal state. I checked go code for that part yet did not udnderstand this behaviour

karolizi · 2024-09-09T07:29:42.835Z

Would really appreciate the advice here @Nam_Nguyen4

karolizi · 2024-09-09T12:33:13.929Z

Updated issue @Nam_Nguyen4 MongoDB replicaset with PS or PSS setup could not have arbiter added due to crashloop for goal state · Issue #1615 · mongodb/mongodb-kubernetes-operator · GitHub

karolizi · 2024-09-12T09:29:18.867Z

@Nam_Nguyen4 any updates?