Prerequisites

Review Supported Hardware Architectures

See supported hardware architectures.

Clone the MongoDB Enterprise Kubernetes Operator Repository

Clone the MongoDB Enterprise Kubernetes Operator repository:

git clone https://github.com/mongodb/mongodb-enterprise-kubernetes.git

Set Environment Variables and GKE Zones

Set the environment variables with cluster names and the available GKE zones where you deploy the clusters, as in this example:

export MDB_GKE_PROJECT={GKE project name}
export MDB_CENTRAL_CLUSTER="mdb-central"
export MDB_CENTRAL_CLUSTER_ZONE="us-west1-a"
export MDB_CLUSTER_1="mdb-1"
export MDB_CLUSTER_1_ZONE="us-west1-b"
export MDB_CLUSTER_2="mdb-2"
export MDB_CLUSTER_2_ZONE="us-east1-b"
export MDB_CLUSTER_3="mdb-3"
export MDB_CLUSTER_3_ZONE="us-central1-a"
export MDB_CENTRAL_CLUSTER_FULL_NAME="gke_${MDB_GKE_PROJECT}_${MDB_CENTRAL_CLUSTER_ZONE}_${MDB_CENTRAL_CLUSTER}"
export MDB_CLUSTER_1_FULL_NAME="gke_${MDB_GKE_PROJECT}_${MDB_CLUSTER_1_ZONE}_${MDB_CLUSTER_1}"
export MDB_CLUSTER_2_FULL_NAME="gke_${MDB_GKE_PROJECT}_${MDB_CLUSTER_2_ZONE}_${MDB_CLUSTER_2}"
export MDB_CLUSTER_3_FULL_NAME="gke_${MDB_GKE_PROJECT}_${MDB_CLUSTER_3_ZONE}_${MDB_CLUSTER_3}"

Set up GKE Clusters

Set up GKE (Google Kubernetes Engine) clusters:

gcloud container clusters create $MDB_CENTRAL_CLUSTER \
  --zone=$MDB_CENTRAL_CLUSTER_ZONE \
  --num-nodes=5 \
  --machine-type "e2-standard-2"

gcloud container clusters create $MDB_CLUSTER_1 \
  --zone=$MDB_CLUSTER_1_ZONE \
  --num-nodes=5 \
  --machine-type "e2-standard-2"

gcloud container clusters create $MDB_CLUSTER_2 \
  --zone=$MDB_CLUSTER_2_ZONE \
  --num-nodes=5 \
  --machine-type "e2-standard-2"

gcloud container clusters create $MDB_CLUSTER_3 \
  --zone=$MDB_CLUSTER_3_ZONE \
  --num-nodes=5 \
  --machine-type "e2-standard-2"

Obtain User Authentication Credentials for Central and Member Clusters

Obtain user authentication credentials for the central and member Kubernetes clusters and save the credentials. You will later use these credentials for running kubectl commands on these clusters.

Run the following commands:

gcloud container clusters get-credentials $MDB_CENTRAL_CLUSTER \
  --zone=$MDB_CENTRAL_CLUSTER_ZONE
gcloud container clusters get-credentials $MDB_CLUSTER_1 \
  --zone=$MDB_CLUSTER_1_ZONE
gcloud container clusters get-credentials $MDB_CLUSTER_2 \
  --zone=$MDB_CLUSTER_2_ZONE
gcloud container clusters get-credentials $MDB_CLUSTER_3 \
  --zone=$MDB_CLUSTER_3_ZONE

Install Go and Helm

Install the following tools:

1. Install Go v1.17 or later.

2. Install Helm.

Understand Kubernetes Roles and Role Bindings

To use a multi-Kubernetes-cluster deployment, you must have a specific set of Kubernetes Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, and ServiceAccounts, which you can configure in any of the following ways:

• Follow the Multi-Kubernetes-Cluster Quick Start, which tells you how to use the MongoDB Plugin to automatically create the required objects and apply them to the appropriate clusters within your multi-Kubernetes-cluster deployment.

• Use Helm to configure the required Kubernetes Roles and service accounts for each member cluster:

  helm template --show-only \
    templates/database-roles.yaml \
    mongodb/enterprise-operator \
    --set namespace=mongodb | \
  kubectl apply -f - \
    --context=$MDB_CLUSTER_1_FULL_NAME \
    --namespace mongodb

  helm template --show-only \
    templates/database-roles.yaml \
    mongodb/enterprise-operator \
    --set namespace=mongodb | \
  kubectl apply -f - \
    --context=$MDB_CLUSTER_2_FULL_NAME \
    --namespace mongodb

  helm template --show-only \
    templates/database-roles.yaml \
    mongodb/enterprise-operator \
    --set namespace=mongodb | \
  kubectl apply -f - \
    --context=$MDB_CLUSTER_3_FULL_NAME \
    --namespace mongodb

• Manually create Kubernetes object .yaml files and add the required Kubernetes Roles and service accounts to your multi-Kubernetes-cluster deployment with the kubectl apply command. This may be necessary for certain highly automated workflows. MongoDB provides sample configuration files.

  For namespace-scoped resources:

  • Roles, Role Bindings, and Service Accounts for your Central Cluster

  • Roles, Role Bindings, and Service Accounts for your Member Clusters

  For cluster-scoped resources:

  • ClusterRoles, ClusterRoleBindings, and ServiceAccounts for your Central Cluster

  • ClusterRoles, ClusterRoleBindings, and ServiceAccounts for your Member Cluster

  Each file defines multiple resources. To support your deployment, you must replace the placeholder values in the following fields:

  • subjects.namespace in each RoleBinding or ClusterRoleBinding resource

  • metadata.namespace in each ServiceAccount resource

  After modifying the definitions, apply them by running the following command for each file:

  kubectl apply -f <fileName>

Set the Deployment's Scope

By default, the multi-cluster Kubernetes Operator is scoped to the namespace in which you install it. The Kubernetes Operator reconciles the MongoDBMultiCluster resource deployed in the same namespace as the Kubernetes Operator.

When you run the MongoDB kubectl plugin as part of the multi-cluster quick start, and don't modify the kubectl mongodb plugin settings, the plugin:

• Creates a default ConfigMap named mongodb-enterprise-operator-member-list that contains all the member clusters of the multi-Kubernetes-cluster deployment. This name is hard-coded and you can't change it. See Known Issues.

• Creates service accounts, Roles, and RoleBindings in the central cluster and each member cluster.

• Applies the correct permissions for service accounts.

• Uses the preceding settings to create your multi-Kubernetes-cluster deployment.

Once the Kubernetes Operator creates the multi-Kubernetes-cluster deployment, the Kubernetes Operator starts watching MongoDB resources in the mongodb namespace.

To configure the Kubernetes Operator with the correct permissions to deploy in multiple or all namespaces, run the following command and specify the namespaces that you would like the Kubernetes Operator to watch.

kubectl mongodb multicluster setup \
  --central-cluster="${MDB_CENTRAL_CLUSTER_FULL_NAME}" \
  --member-clusters="${MDB_CLUSTER_1_FULL_NAME},${MDB_CLUSTER_2_FULL_NAME},${MDB_CLUSTER_3_FULL_NAME}" \
  --member-cluster-namespace="mongodb2" \
  --central-cluster-namespace="mongodb2" \
  --cluster-scoped="true"

When you install the multi-Kubernetes-cluster deployment to multiple or all namespaces, you can configure the Kubernetes Operator to:

• Watch Resources in Multiple Namespaces

• Watch Resources in All Namespaces

WATCH_NAMESPACE: "$namespace1,$namespace2,$namespace3"

helm upgrade \
  --install \
  mongodb-enterprise-operator-multi-cluster \
  mongodb/enterprise-operator \
  --namespace mongodb \
  --set namespace=mongodb \
  --version <mongodb-kubernetes-operator-version>\
  --set operator.name=mongodb-enterprise-operator-multi-cluster \
  --set operator.createOperatorServiceAccount=false \
  --set "multiCluster.clusters={$MDB_CLUSTER_1_FULL_NAME,$MDB_CLUSTER_2_FULL_NAME,$MDB_CLUSTER_3_FULL_NAME}" \
  --set operator.watchNamespace="$namespace1,$namespace2,$namespace3"

WATCH_NAMESPACE: "*"

helm upgrade \
  --install \
  mongodb-enterprise-operator-multi-cluster \
  mongodb/enterprise-operator \
  --namespace mongodb \
  --set namespace=mongodb \
  --version <mongodb-kubernetes-operator-version>\
  --set operator.name=mongodb-enterprise-operator-multi-cluster \
  --set operator.createOperatorServiceAccount=false \
  --set "multiCluster.clusters={$MDB_CLUSTER_1_FULL_NAME,$MDB_CLUSTER_2_FULL_NAME,$MDB_CLUSTER_3_FULL_NAME}" \
  --set operator.watchNamespace="*"

Plan for External Connectivity: Should You Use a Service Mesh?

A service mesh enables inter-cluster communication between the replica set members deployed in different Kubernetes clusters. Using a service mesh greatly simplifies creating multi-Kubernetes-cluster deployments and is the recommended way of deploying MongoDB across multiple Kubernetes clusters. However, if your IT organization doesn't use a service mesh, you can deploy a replica set in a multi-Kubernetes-cluster deployment without it.

Depending on your environment, do the following:

• If you can use a service mesh, install Istio.

• If you can't use a service mesh:

  1. Review the section How Does the Kubernetes Operator Establish Connectivity?

  2. Enable external connectivity through external domains and DNS zones.

  3. Deploy a multi-Kubernetes cluster without a service mesh.

Check Connectivity Across Clusters

Follow the steps in this procedure to verify that service FQDNs are reachable across Kubernetes clusters.

In this example, you deploy a sample application defined in sample-service.yaml across two Kubernetes clusters.

kubectl create --context="${CTX_CLUSTER_1}" namespace sample
kubectl create --context="${CTX_CLUSTER_2}" namespace sample

kubectl apply --context="${CTX_CLUSTER_1}" \
   -f sample-service.yaml \
   -l service=helloworld1 \
   -n sample
kubectl apply --context="${CTX_CLUSTER_2}" \
   -f sample-service.yaml \
   -l service=helloworld2 \
   -n sample

kubectl apply --context="${CTX_CLUSTER_1}" \
   -f sample-service.yaml \
   -l version=v1 \
   -n sample

kubectl get pod --context="${CTX_CLUSTER_1}" \
   -n sample \
   -l app=helloworld

kubectl apply --context="${CTX_CLUSTER_2}" \
   -f sample-service.yaml \
   -l version=v2 \
   -n sample

kubectl get pod --context="${CTX_CLUSTER_2}" \
   -n sample \
   -l app=helloworld

kubectl run  --context="${CTX_CLUSTER_1}" \
   -n sample \
   curl --image=radial/busyboxplus:curl \
   -i --tty \
   curl -sS helloworld2.sample:5000/hello

Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8

kubectl run --context="${CTX_CLUSTER_2}" \
   -n sample \
   curl --image=radial/busyboxplus:curl \
   -i --tty \
   curl -sS helloworld1.sample:5000/hello

Hello version: v1, instance: helloworld-v1-758dd55874-6x4t8

Review the Requirements for Deploying Ops Manager

As part of the Quick Start, you deploy an Ops Manager resource on the central cluster. To learn more, see Deploy an Ops Manager Resource on the Central Cluster and Connect to Ops Manager.

Prepare for TLS-Encrypted Connections

If you plan to secure your multi-Kubernetes-cluster deployment using TLS encryption, complete the following tasks to enable internal cluster authentication and generate TLS certificates for member clusters and the MongoDB Agent:

kubectl --context $MDB_CENTRAL_CLUSTER_FULL_NAME \
--namespace=<metadata.namespace> \

curl https://raw.githubusercontent.com/mon  mongodb-enterprise-kubernetes/master/tools/multicluster/setup_tl  -o  setup_tls.sh

Choose GitOps or the kubectl MongoDB Plugin

You can choose to create and maintain the resource files needed for the MongoDBMultiCluster resources deployment in a GitOps environment.

If you use a GitOps workflow, you can't use the kubectl mongodb plugin, which automatically configures role-based access control (RBAC) and creates the kubeconfig file that allows the central cluster to communicate with its member clusters. Instead, you must manually configure or build your own automation for configuring the RBAC and kubeconfig files based on the procedure and examples in Configure Resources for GitOps.

The following prerequisite sections describe how to install the kubectl MongoDB plugin if you don't use GitOps or configure resources for GitOps if you do.

Install the kubectl MongoDB Plugin

Use the kubectl mongodb plugin to:

• Set up multi-Kubernetes-cluster Deployments

• Run automatic and manual disaster recovery

To install the kubectl mongodb plugin:

tar -zxvf kubectl-mongodb_<version>_darwin_amd64.tar.gz

mv kubectl-mongodb /usr/local/bin/kubectl-mongodb

kubectl mongodb multicluster setup
kubectl mongodb multicluster recover

Configure Resources for GitOps

If you use a GitOps workflow, you won't be able to use the kubectl mongodb plugin to automatically configure role-based access control (RBAC) or the kubeconfig file that allows the central cluster to communicate with its member clusters. Instead, you must manually configure and apply the following resource files or build your own automation based on the information below.

To configure RBAC and the kubeconfig for GitOps:

apiVersion: v1
kind: ConfigMap
data:
  cluster1: ""
  cluster2: ""
metadata:
  namespace: <namespace>
  name: mongodb-enterprise-operator-member-list
  labels:
    multi-cluster: "true"