Filtered Sync

Filter Syntax

The start API endpoint accepts two fields that configure filtered sync: includeNamespaces and excludeNamespaces. Each field takes an array of filters that specify the databases and collections to include or exclude from sync.

Filters have the following syntax:

"includeNamespaces": [
    {
       "database": "<database-name>",
       "collections": [
          "<collection-name>"
       ]
       "databaseRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       },
       "collectionsRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       }
    }
 ],
 "excludeNamespaces": [
    {
       "database": "<database-name>",
       "collections": [
          "<collection-name>"
       ]
       "databaseRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       },
       "collectionsRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       }
    }
 ]

Filters must include either the database field or the databaseRegex field.

If you need the filter to match specific collections, you can use either the collections array to specify collections individually or define a regular expression using the collectionsRegex field.

Configure a Filter

"includeNamespaces": [
   {
      "database": "sales",
      "collectionsRegex": {
         "pattern": "^accounts_.+?$",
         "options": "ms"
      }
],
"excludeNamespaces": [
    {
        "database": "sales",
        "collections": [
            "accounts_old"
        ]
    }
]

curl -X POST "http://localhost:27182/api/v1/start" --data '
   {
      "source": "cluster0",
      "destination": "cluster1",
      "includeNamespaces": [
         {
             "database": "sales",
             "collectionsRegex": {
                "pattern": "^accounts_.+$",
                "options": "i"
             }
         }, {
            "database": "marketing"
         }
      ]
   } '

For an example configuration, see: Start mongosync with a Filter.

Replace an Existing Filter

You cannot update an existing filter. You must stop the ongoing sync process, prepare the destination cluster, and restart mongosync with a new filter.

When mongosync ran your original filter, it created databases with your data ("user databases") and the mongosync_reserved_for_internal_use system database on the destination cluster. You must remove those databases before restarting mongosync with your new filter.

Follow these steps to prepare the destination cluster for a new filter.

Adding and Renaming Collections

You can, with some restrictions, add or rename a collection during a filtered sync.

Filtering with mapReduce and $out

To use the $out aggregation stage or the mapReduce command (when set to create or replace a collection) with filtering, you must filter the whole database and not just the specified collection.

For example, consider this aggregation pipeline:

use library
db.books.aggregate( [
   { $group : { _id : "$author", titles: { $push: "$title" } } },
   { $out : "authors" }
] )

The $out stage creates the authors collection in the library database. If you want to sync the authors collection, you must specify the entire library database in your filter. The filter will not work if you only specify the authors collection.

This filter works:

"includeNamespaces": [
  {
    "database": "library"
  }
]

This filter does not work:

"includeNamespaces": [
  {
    "database": "library",
    "collections": [ "authors", "books" ]  // DOES NOT WORK WITH $OUT
  }
]

Limitations

• Filtering is not supported with reversible sync.

• The destination cluster must not contain user data prior to starting.

• The destination cluster must not contain the mongosync_reserved_for_internal_use system database prior to starting.

• You cannot modify a filter that is in use. To create a new filter, see: Replace an Existing Filter.

• You can only rename collections in certain situations. For more details see: Adding and Renaming Collections.

• If a filter includes a view but not the base collection, only the view metadata syncs to the destination cluster. To include the view documents, you must also sync the base collection.

• You cannot specify system collections or system databases in a filter.

• To use the $out aggregation stage or the mapReduce command (when set to create or replace a collection) with filtering, you must configure the filter to use the entire database. You cannot limit the filter to collections within the database.

  For more information, see Filtering with mapReduce and $out.

Examples

"includeNamespaces" : [
      {
          "database" : "sales",
          "collections": [ "EMEA", "APAC" ]
      },
      {
          "database" : "marketing"
      }
   ]

{
   "source": "cluster0",
   "destination": "cluster1",
   "includeNamespaces":
      [
         { "database" : "students", "collections": ["undergrad", "graduate", "adjuncts"] },
         { "database" : "staff" }
      ]
}

// This code works
use admin
db.runCommand( { renameCollection: "staff.employees", to: "staff.salaried" } )

// This code works
use admin
db.runCommand( { renameCollection: "students.graduate", to: "students.adjuncts" } )

// This code produces an error and mongosync stops syncing
use admin
db.runCommand( { renameCollection: "students.graduate", to: "students.notAFilteredCollection" } )

use admin
db.runCommand( { renameCollection: "staff.employees", to: "staff.onPayroll" } )