How to Index Fields in Arrays of Objects and Documents

2,100,000,000 Index Objects Limit

Atlas Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded document counts as a single object. Using the embeddedDocuments field type can result in indexing objects over this limit, which causes an index to transition to a Stale queryable state, and may result in stale results. If you create an Atlas Search index for a collection that has or will soon have more than 2.1 billion index objects, you must use the numPartitions option or shard your cluster.

The exact number of index objects can vary based on the rate of document changes and deletions. The Search Max Number of Lucene Docs metric provides the upper bound of the current number of index objects across all indexes per replica set or shard. You can approximate the expected number of index objects in a single index by doing the following:

1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object.

   total number of index objects = 1 + number of nested embedded documents

2. Multiply the number of index objects per document by the total number of documents in the collection

   total number of index objects x total number of documents in collection

Note that this approximation is a lower bound.

{
  "_id": 0,
  "name": "Springfield High",
  "mascot": "Pumas",
  "teachers": [
    {
      "first": "Jane",
      "last": "Smith",
      "classes": [
        {
          "subject": "art of science",
          "grade": "12th"
        },
        ... // 2 more embedded documents
      ]
    },
    ... // 1 more embedded document
  ],
  "clubs": {
    "stem": [
      {
        "club_name": "chess",
        "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments."
      },
      ... // 1 more embedded document
    ],
    ... // 1 more embedded document
  }
}

If your collection has large arrays that might generate 2,100,000,000 index objects, you must shard any clusters that contain indexes with the embeddedDocuments type.

Atlas Search Limitations

The following limitations apply:

• You can use embeddedDocuments only on fields with up to 5 levels of nesting. An embeddedDocuments field can't have more than 4 parent embeddedDocuments fields.

• You can't use embeddedDocuments for date or numeric faceting.

• You can't define a field inside the embeddedDocuments type as the deprecated knnVector type.

• For highlighting fields within embedded documents, you must also index the parent of the field that you want to highlight as the document type.

• You can do the following only if you index the parents of the embedded document child field as the document type:

  • Faceted search on string fields within embedded documents. You must also index the field that you want to facet on as the stringFacet type.

    Note

    When you facet on a string field inside embedded documents, Atlas Search returns facet count for only the number of matching parent documents.

    You can't facet on numeric and date fields in embedded documents.

  • Highlight fields within embedded documents. For an example, see the How to Run Atlas Search Queries Against Objects in Arrays tutorial.

  • Sort by the parent of the embedded document field. You must also index the embedded document field with string values as the token type. For child fields with number and date values, enable dynamic mapping to index those fields automatically. For an example, see Sort Example.