Verify with Embedded Verifier

On this page

About this Task

Steps
Behavior
Learn More

mongosync includes an embedded verifier to perform a series of checks on the destination cluster to verify the sync of supported collections.

Important

The verifier doesn't check the following namespaces:

Capped collections
Collections with TTL indexes
Collections that don't use the default collation
Views

The verifier doesn't check the following collection features:

Collection metadata
Indexes

To verify the above data and metadata, script additional checks for these collections and collection features. For more information, see Verify Data Transfer.

New in version 1.9.

About this Task

Compatibility

The embedded verifier is not available in mongosync 1.8 and earlier.

For alternative verification methods, see Verify Data Transfer.

Limitations

The embedded verifier has the following limitations:

The verifier doesn't support sharded clusters. If the migration includes a sharded cluster, mongosync disables the verifier.
mongosync stores the verifier state in memory, which can result in a significant memory overhead. To run the verifier, mongosync requires approximately 10 GB of memory, plus an additional 500 MB for every 1 million documents.
The verifier cannot be resumed. If a user stops or pauses sync and then starts mongosync again for any reason, the verification process restarts from the beginning. This can cause verification to fall substantially behind the migration.
The /reverse endpoint disables the verifier. It remains disabled after additional calls to the /reverse endpoint.
If you start sync with verification enabled and buildIndexes set to never, the migration will fail if mongosync finds a TTL collection on the source cluster. This can happen after you call the /start endpoint or much later, such as where a user creates a TTL index on the source cluster while a migration is in progress.
To sync TTL collections without building indexes on the destination cluster, you must start sync with the verifier disabled.

Unsupported Verification Checks

The verifier doesn't check the following namespaces:

Capped collections
Collections with TTL indexes
Collections that don't use the default collation
Views

The verifier doesn't check the following collection features:

Collection metadata
Indexes

To verify the above data and metadata, script additional checks for these collections and collection features. For more information, see Verify Data Transfer.

Unsupported Beta Features

The verifier isn't compatible with these beta features:

Many-to-One Migrations
A->B->C Migrations
Document Filtering
Namespace Remapping
Handle Pre-Existing Data on the Destination
Clusters that run a release of MongoDB that mongosync doesn't support.

Steps

Initialize `mongosync`

Initialize the mongosync process:

./bin/mongosync \
    --logPath /var/log/mongosync \
    --cluster0 "mongodb://clusterAdmin:superSecret@clusterOne01.fancyCorp.com:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020" \
    --cluster1 "mongodb://clusterAdmin:superSecret@clusterTwo01.fancyCorp.com:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020"

Start the Sync

To start syncing data from the source cluster to the destination, use the /start endpoint.

curl localhost:27182/api/v1/start -XPOST \
    --data ' {
        "source": "cluster0",
        "destination": "cluster1",
    } '

Example output:

{"success":true}

Note, the verifier is enabled by default for replica set migrations.

Examine Progress

To examine the status of the sync, use the /progress endpoint:

curl localhost:27182/api/v1/progress -XGET

Example output:

{
   "progress":
      {
         "state":"RUNNING",
         "canCommit":true,
         "canWrite":false,
         "info":"change event application",
         "lagTimeSeconds":0,
         "collectionCopy":
            {
               "estimatedTotalBytes":694,
               "estimatedCopiedBytes":694
            },
         "directionMapping":
            {
               "Source":"cluster0: localhost:27017",
               "Destination":"cluster1: localhost:27018"
            },
         "verification":
            {
               "source":
                  {
                     "estimatedDocumentCount": 42,
                     "hashedDocumentCount": 42,
                     "lagTimeSeconds": 2,
                     "totalCollectionCount": 42,
                     "scannedCollectionCount": 10,
                     "phase": "stream hashing"
                  },
               "destination": {
                     "estimatedDocumentCount": 42,
                     "hashedDocumentCount": 42,
                     "lagTimeSeconds": 2,
                     "totalCollectionCount": 42,
                     "scannedCollectionCount": 10,
                     "phase": "stream hashing"
               }
            }
      },
   "success": true
}

Examine the verification response field for information on the status of the embedded verifier.

Behavior

Verification Checks

The embedded verifier performs a series of checks on the destination cluster. It checks all supported collections to confirm that mongosync was successful in transferring documents from the source cluster to the destination.

If the verifier encounters errors, it fails the migration with an error. If the verifier finds no errors, mongosync transitions to the COMMITTED state.

Please contact support to investigate verification issues.

Memory Requirements

Verification requires 10 GB of memory plus an additional 500 MB for every 1 million documents in the migration.

If the available memory is insufficient, the /start endpoint returns an error. If this occurs, to use mongosync with the verifier you must first increase the memory of the server and resume the migration.

If increasing server memory isn't an option, restart mongosync with the verifier disabled.

Learn More

Back

Data Transfer Verification

Use Document Counts