Sync Sharded Clusters

Configure a Single mongosync Instance

To configure a single mongosync, follow the connection instructions for your cluster architecture to connect to the mongos instance in your cluster.

When you connect a single mongosync to a sharded cluster do not use the replicaSet option or the id option.

The rest of this page addresses cluster to cluster synchronization using multiple mongosync instances.

Configure Multiple mongosync Instances

The number of mongosync instances must match the number of shards on the source cluster. You must use the same version of mongosync between all instances. For a replica set source, you can only use one mongosync instance.

When you configure multiple mongosync instances to sync between sharded clusters, you must send identical API endpoint commands to each mongosync instance.

To configure multiple mongosync instances:

1. Verify cluster configuration

2. Determine the shard IDs

3. Connect the instances

db.adminCommand( { listShards: 1 } )

  shards: [
    {
      _id: 'shard01',
      host: 'shard01/localhost:27501,localhost:27502,localhost:27503',
      state: 1,
      topologyTime: Timestamp({ t: 1656612236, i: 2 })
    },
    {
      _id: 'shard02',
      host: 'shard02/localhost:27504,localhost:27505,localhost:27506',
      state: 1,
      topologyTime: Timestamp({ t: 1656612240, i: 4 })
    }
  ]

mongosync \
  --cluster0 "mongodb://user:password@cluster0host:27500" \
  --cluster1 "mongodb://user:password@cluster1host:27500" \
  --id shard01 --port 27601

 mongosync \
   --cluster0 "mongodb://user:password@cluster0host:27500" \
   --cluster1 "mongodb://user:password@cluster1host:27500" \
   --id shard02 --port 27602

Start Multiple mongosync Instances

Use curl or another HTTP client to issue the start command to each of the mongosync instances.

curl mongosync01Host:27601/api/v1/start -XPOST --data \
     '{ "source": "cluster0", "destination": "cluster1", \
        "reversible": false, "enableUserWriteBlocking": false }'
curl mongosync02Host:27602/api/v1/start -XPOST --data \
     '{ "source": "cluster0", "destination": "cluster1", \
        "reversible": false, "enableUserWriteBlocking": false }'

The start command options must be the same for all of the mongosync instances.

Check Progress

To review synchronization progress for a particular shard, use curl or another HTTP client to issue the progress command to the mongosync instance syncing that shard.

curl mongosync02Host:27602/api/v1/progress -XGET

This command checks the progress of the mongosync instance that is running on mongosync02Host and using port 27602 for synchronization. To check progress on other shards, update the host and port number then repeat the API call to each mongosync instance.

Pause a mongosync Instance

The pause command will temporarily halt the synchronization process on a single shard. It does not pause any other mongosync instances that may be running. Use curl or another HTTP client to issue the pause command to a mongosync instance.

curl mongosync01Host:27601/api/v1/pause -XPOST --data '{}'

This command pauses the mongosync instance that is running on mongosync01Host and using port 27601 for synchronization. To pause synchronization on other shards, update the host and port number then repeat the API call to each mongosync instance.

Resume Synchronization

If one or more mongosync instances are paused, you can use the resume command to resume syncing. Run a separate resume command against each paused mongosync instance to continue syncing.

Use curl or another HTTP client to issue the resume command to each mongosync instance.

curl mongosync01Host:27601/api/v1/resume -XPOST --data '{}'

This command resumes synchronization on the mongosync instance that is running on mongosync01Host and using port 27601. To resume synchronization on other shards, update the host and port number then repeat the API call to each mongosync instance.

Commit Synchronization From Multiple mongosync Instances

When you want to complete synchronization, issue the progress command and check the values for canCommit and lagTimeSeconds.

To minimize write blocking on the source cluster, you should only run the commit command when the lagTimeSeconds value is small enough for your application.

If the lagTimeSeconds value is small enough, and canCommit is true, issue the commit command to commit synchronization. Repeat the process on all of the mongosync instances.

The commit operation is blocking. The commit command will not return until commit has been called on every mongosync instance.

// Check progress
curl mongosync01Host:27601/api/v1/progress -XGET
// Commit
curl mongosync01Host:27601/api/v1/commit -XPOST --data '{}'

These commands only check progress and commit synchronization for the mongosync instance that is running on mongosync01Host and using port 27601. To synchronize all of the shards, make additional calls to progress and commit on any other mongosync instances that may be running.

Data Verification

Before transferring your application load from the source cluster to the destination, check your data to ensure that the sync was successful.

For more information, see Verify Data Transfer.

Reverse the Synchronization Direction

To reverse synchronization so that the original destination cluster acts as the source cluster:

• If you have not already done so, issue the commit command to each mongosync instance and wait until all of the commits to finish. To check if the sync process has been committed, issue the progress command to all mongosync instances and see if each response's state field contains the value COMMITTED.

• Issue the reverse command to each mongosync instance.

The reverse operation is blocking. The reverse command will not return until reverse has been called on every mongosync instance.

curl mongosync01Host:27601/api/v1/reverse -XPOST --data '{}'

This command reverses synchronization on the mongosync instance that is running on mongosync01Host and using port 27601. Make additional calls to reverse on any other mongosync instances that may be running.