Improve MongoDB Search Index Performance

Dynamic vs. Static Mappings

When you create an MongoDB Search index, field mapping defaults to dynamic, which means that MongoDB Search dynamically indexes all the datatypes that can be dynamically indexed in your collection. This can lead to large indexes if you have a large number of fields in your documents, or if you have fields with long string values.

To reduce the size and performance footprint of your index, we recommend using static mappings to specify the fields that you want to index and their types.

store Option

When indexing a text field as the MongoDB Search string type, you can set the store option to false to reduce the size of your index. This prevents MongoDB Search from storing the exact document text as well as the analyzed values in the index. However, setting store to false prevents you from using the highlight option to adds fields to the result set that display search terms in their original context.

Autocomplete

The MongoDB Search autocomplete field type creates indexes that you can use with the autocomplete operator to implement search-as-you-type functionality in your application. However, the autocomplete field type can create large indexes and take longer to build than other field types.

Use the following recommendations to reduce the size and performance footprint of the autocomplete type:

• Use the autocomplete type only for specific fields in static mappings. Using a custom typeSet definition to dynamically index string fields as the autocomplete type can apply the autocomplete type too broadly and significantly increase index size and resource usage. In addition, the autocomplete type has specialized, typeahead-oriented scoring behavior suited to specific field indexing, not broad dynamic application. To learn more about similarity.type scoring with the autocomplete type, see text, phrase, queryString, and autocomplete Operators.

• Avoid using nGram as the tokenization strategy for the autocomplete type definition, as it creates more tokens for a given string than edgeGram or rightEdgeGram tokenization.

• Set a smaller range between the minGrams and maxGrams option values in the autocomplete type definition, which reduces the number of tokens created for each string.

  Generally, we recommend setting maxGrams to the character count of the longest word in the field that you want to query. If you are unsure, for English language fields, we recommend starting with maxGrams value of 10.

  Setting a larger value for minGrams can significantly reduce index size for fields in a collection with many documents. For example, setting minGrams to 2 instead of 1 creates 100,000 fewer tokens in a collection with 100,000 documents.

• Monitor your index size and adjust your autocomplete type configuration as needed. You can view MongoDB Search metrics for the following index size and tokenization metrics to help you understand the impact of your autocomplete type configuration on your index size and resource usage:

  • Search Index Size

  • Search Max nGram Fields Indexed

When indexing a string field as the autocomplete type, we recommend that you index the field as the MongoDB Search string type also for the following advantages:

• Boost the score of exact matches when using the autocomplete operator.

• Query the same field in the same query using the autocomplete operator and another operator that supports string search, such as the text operator. For an example, see the Search Across Multiple Fields example.

Embedded Documents

MongoDB Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded parent document counts as a single object. Surpassing this limit may result in stale query results.

Using the embeddedDocuments field type can result in indexing objects over this index size limit, because each indexed embedded document is counted as a single object. If you create a MongoDB Search index that has or will soon have more than 2.1 billion index objects, use the numPartitions index option to partition your index (supported only on Search Nodes deployments) or shard your cluster.

The exact number of index objects can vary based on the rate of document changes and deletions. The Search Max Number of Lucene Docs metric shows the largest number of Lucene documents in a single partition of any search index within a replica set or shard.

For example, suppose the metric displays "1,500,000,000." The index with the highest number of Lucene documents in a single partition has about 1,500,000,000 documents in that partition. If the index uses many partitions, it can contain more documents overall.

You can approximate the expected number of index objects in a single index by doing the following steps:

1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object.

   number of index objects in document = 1 + number of nested embedded documents

2. Multiply the number of index objects per document by the total number of documents in the collection

   total number of index objects = number of index objects in document x total number of documents in collection

Note that this approximation is a lower bound.

{
  "_id": 0,
  "name": "Springfield High",
  "mascot": "Pumas",
  "teachers": [
    {
      "first": "Jane",
      "last": "Smith",
      "classes": [
        {
          "subject": "art of science",
          "grade": "12th"
        },
        ... // 2 more embedded documents
      ]
    },
    ... // 1 more embedded document
  ],
  "clubs": {
    "stem": [
      {
        "club_name": "chess",
        "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments."
      },
      ... // 1 more embedded document
    ],
    ... // 1 more embedded document
  }
}

If your collection has large arrays that might generate 2,100,000,000 index objects, you must shard any clusters that contain indexes with the embeddedDocuments type.

Faceted Search

If you want to filter and facet your data using the same field, we recommend that you index the field as following MongoDB Search types:

• token

• number

• date

For an example of filtering the data by another field for faceting, see the How to Use Facets with MongoDB Search tutorial.

multi Analyzers

Using a multi analyzer to analyze the same field multiple different ways can cause large indexes, especially when analyzing fields with very long values.

Multilingual Search

You can use the MongoDB Search Language Analyzers to index many languages. For the list of languages for which MongoDB Search provides built-in analyzers, see the Language Analyzers. To index and query languages that are currently not in the list of built-in Language Analyzers, you can create a custom analyzer. For an example, see the Custom Language Analyzer Example.

Suppose you have one document for each language in your collection. Consider the following:

• You can index the fields separately in the same index using the Language Analyzers. A single index can support multiple languages in the same query.

• Alternatively, you can create an index per language, which is useful in isolating the different language documents. Note that each index is a change stream cursor and so this might be expensive to maintain.

• If you have the language documents nested inside a parent document, you can create a single index. However, your index definition payload might be large and your query might be complex.

To learn how to do a Multilingual Search, see Multilingual Search Example

Synonym Collections

Inserts and updates to a synonym source collection are fast only if the synonym source collection is small. For best performance, we recommend batching inserts and updates to synonym source collections.

A synonym mapping definition doesn't require additional disk space aside from the disk space utilized by the synonym collection in the database. However, synonym mappings create artifacts in memory and therefore, for synonym collections with many documents, MongoDB Search creates artifacts that occupy more memory.

Dynamic vs. Static Mappings

When you create an MongoDB Search index, field mapping defaults to dynamic, which means that MongoDB Search dynamically indexes all the datatypes that can be dynamically indexed in your collection. This can lead to large indexes if you have a large number of fields in your documents, or if you have fields with long string values.

To reduce the size and performance footprint of your index, we recommend using static mappings to specify the fields that you want to index and their types.

store Option

When indexing a text field as the MongoDB Search string type, you can set the store option to false to reduce the size of your index. This prevents MongoDB Search from storing the exact document text as well as the analyzed values in the index. However, setting store to false prevents you from using the highlight option to adds fields to the result set that display search terms in their original context.

Autocomplete

The MongoDB Search autocomplete field type creates indexes that you can use with the autocomplete operator to implement search-as-you-type functionality in your application. However, the autocomplete field type can create large indexes and take longer to build than other field types.

Use the following recommendations to reduce the size and performance footprint of the autocomplete type:

• Use the autocomplete type only for specific fields in static mappings. Using a custom typeSet definition to dynamically index string fields as the autocomplete type can apply the autocomplete type too broadly and significantly increase index size and resource usage. In addition, the autocomplete type has specialized, typeahead-oriented scoring behavior suited to specific field indexing, not broad dynamic application. To learn more about similarity.type scoring with the autocomplete type, see text, phrase, queryString, and autocomplete Operators.

• Avoid using nGram as the tokenization strategy for the autocomplete type definition, as it creates more tokens for a given string than edgeGram or rightEdgeGram tokenization.

• Set a smaller range between the minGrams and maxGrams option values in the autocomplete type definition, which reduces the number of tokens created for each string.

  Generally, we recommend setting maxGrams to the character count of the longest word in the field that you want to query. If you are unsure, for English language fields, we recommend starting with maxGrams value of 10.

  Setting a larger value for minGrams can significantly reduce index size for fields in a collection with many documents. For example, setting minGrams to 2 instead of 1 creates 100,000 fewer tokens in a collection with 100,000 documents.

• Monitor your index size and adjust your autocomplete type configuration as needed. You can view MongoDB Search metrics for the following index size and tokenization metrics to help you understand the impact of your autocomplete type configuration on your index size and resource usage:

  • Search Index Size

  • Search Max nGram Fields Indexed

When indexing a string field as the autocomplete type, we recommend that you index the field as the MongoDB Search string type also for the following advantages:

• Boost the score of exact matches when using the autocomplete operator.

• Query the same field in the same query using the autocomplete operator and another operator that supports string search, such as the text operator. For an example, see the Search Across Multiple Fields example.

Embedded Documents

MongoDB Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded parent document counts as a single object. Surpassing this limit may result in stale query results.

Using the embeddedDocuments field type can result in indexing objects over this index size limit, because each indexed embedded document is counted as a single object. If you create a MongoDB Search index that has or will soon have more than 2.1 billion index objects, use the numPartitions index option to partition your index (supported only on Search Nodes deployments) or shard your cluster.

The exact number of index objects can vary based on the rate of document changes and deletions. The Search Max Number of Lucene Docs metric shows the largest number of Lucene documents in a single partition of any search index within a replica set or shard.

For example, suppose the metric displays "1,500,000,000." The index with the highest number of Lucene documents in a single partition has about 1,500,000,000 documents in that partition. If the index uses many partitions, it can contain more documents overall.

You can approximate the expected number of index objects in a single index by doing the following steps:

1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object.

   number of index objects in document = 1 + number of nested embedded documents

2. Multiply the number of index objects per document by the total number of documents in the collection

   total number of index objects = number of index objects in document x total number of documents in collection

Note that this approximation is a lower bound.

{
  "_id": 0,
  "name": "Springfield High",
  "mascot": "Pumas",
  "teachers": [
    {
      "first": "Jane",
      "last": "Smith",
      "classes": [
        {
          "subject": "art of science",
          "grade": "12th"
        },
        ... // 2 more embedded documents
      ]
    },
    ... // 1 more embedded document
  ],
  "clubs": {
    "stem": [
      {
        "club_name": "chess",
        "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments."
      },
      ... // 1 more embedded document
    ],
    ... // 1 more embedded document
  }
}

If your collection has large arrays that might generate 2,100,000,000 index objects, you must shard any clusters that contain indexes with the embeddedDocuments type.

Faceted Search

If you want to filter and facet your data using the same field, we recommend that you index the field as following MongoDB Search types:

• token

• number

• date

For an example of filtering the data by another field for faceting, see the How to Use Facets with MongoDB Search tutorial.

multi Analyzers

Using a multi analyzer to analyze the same field multiple different ways can cause large indexes, especially when analyzing fields with very long values.

Multilingual Search

You can use the MongoDB Search Language Analyzers to index many languages. For the list of languages for which MongoDB Search provides built-in analyzers, see the Language Analyzers. To index and query languages that are currently not in the list of built-in Language Analyzers, you can create a custom analyzer. For an example, see the Custom Language Analyzer Example.

Suppose you have one document for each language in your collection. Consider the following:

• You can index the fields separately in the same index using the Language Analyzers. A single index can support multiple languages in the same query.

• Alternatively, you can create an index per language, which is useful in isolating the different language documents. Note that each index is a change stream cursor and so this might be expensive to maintain.

• If you have the language documents nested inside a parent document, you can create a single index. However, your index definition payload might be large and your query might be complex.

To learn how to do a Multilingual Search, see Multilingual Search Example

Synonym Collections

Inserts and updates to a synonym source collection are fast only if the synonym source collection is small. For best performance, we recommend batching inserts and updates to synonym source collections.

A synonym mapping definition doesn't require additional disk space aside from the disk space utilized by the synonym collection in the database. However, synonym mappings create artifacts in memory and therefore, for synonym collections with many documents, MongoDB Search creates artifacts that occupy more memory.

Dynamic vs. Static Mappings

When you create an MongoDB Search index, field mapping defaults to dynamic, which means that MongoDB Search dynamically indexes all the datatypes that can be dynamically indexed in your collection. This can lead to large indexes if you have a large number of fields in your documents, or if you have fields with long string values.

To reduce the size and performance footprint of your index, we recommend using static mappings to specify the fields that you want to index and their types.

store Option

When indexing a text field as the MongoDB Search string type, you can set the store option to false to reduce the size of your index. This prevents MongoDB Search from storing the exact document text as well as the analyzed values in the index. However, setting store to false prevents you from using the highlight option to adds fields to the result set that display search terms in their original context.

Autocomplete

The MongoDB Search autocomplete field type creates indexes that you can use with the autocomplete operator to implement search-as-you-type functionality in your application. However, the autocomplete field type can create large indexes and take longer to build than other field types.

Use the following recommendations to reduce the size and performance footprint of the autocomplete type:

• Use the autocomplete type only for specific fields in static mappings. Using a custom typeSet definition to dynamically index string fields as the autocomplete type can apply the autocomplete type too broadly and significantly increase index size and resource usage. In addition, the autocomplete type has specialized, typeahead-oriented scoring behavior suited to specific field indexing, not broad dynamic application. To learn more about similarity.type scoring with the autocomplete type, see text, phrase, queryString, and autocomplete Operators.

• Avoid using nGram as the tokenization strategy for the autocomplete type definition, as it creates more tokens for a given string than edgeGram or rightEdgeGram tokenization.

• Set a smaller range between the minGrams and maxGrams option values in the autocomplete type definition, which reduces the number of tokens created for each string.

  Generally, we recommend setting maxGrams to the character count of the longest word in the field that you want to query. If you are unsure, for English language fields, we recommend starting with maxGrams value of 10.

  Setting a larger value for minGrams can significantly reduce index size for fields in a collection with many documents. For example, setting minGrams to 2 instead of 1 creates 100,000 fewer tokens in a collection with 100,000 documents.

When indexing a string field as the autocomplete type, we recommend that you index the field as the MongoDB Search string type also for the following advantages:

• Boost the score of exact matches when using the autocomplete operator.

• Query the same field in the same query using the autocomplete operator and another operator that supports string search, such as the text operator. For an example, see the Search Across Multiple Fields example.

Embedded Documents

MongoDB Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded parent document counts as a single object. Surpassing this limit may result in stale query results.

Using the embeddedDocuments field type can result in indexing objects over this index size limit, because each indexed embedded document is counted as a single object. If you create a MongoDB Search index that has or will soon have more than 2.1 billion index objects, use the numPartitions index option to partition your index (supported only on Search Nodes deployments) or shard your cluster.

The exact number of index objects can vary based on the rate of document changes and deletions. The Search Max Number of Lucene Docs metric shows the largest number of Lucene documents in a single partition of any search index within a replica set or shard.

For example, suppose the metric displays "1,500,000,000." The index with the highest number of Lucene documents in a single partition has about 1,500,000,000 documents in that partition. If the index uses many partitions, it can contain more documents overall.

You can approximate the expected number of index objects in a single index by doing the following steps:

1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object.

   number of index objects in document = 1 + number of nested embedded documents

2. Multiply the number of index objects per document by the total number of documents in the collection

   total number of index objects = number of index objects in document x total number of documents in collection

Note that this approximation is a lower bound.

{
  "_id": 0,
  "name": "Springfield High",
  "mascot": "Pumas",
  "teachers": [
    {
      "first": "Jane",
      "last": "Smith",
      "classes": [
        {
          "subject": "art of science",
          "grade": "12th"
        },
        ... // 2 more embedded documents
      ]
    },
    ... // 1 more embedded document
  ],
  "clubs": {
    "stem": [
      {
        "club_name": "chess",
        "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments."
      },
      ... // 1 more embedded document
    ],
    ... // 1 more embedded document
  }
}

If your collection has large arrays that might generate 2,100,000,000 index objects, you must shard any clusters that contain indexes with the embeddedDocuments type.

Faceted Search

If you want to filter and facet your data using the same field, we recommend that you index the field as following MongoDB Search types:

• token

• number

• date

For an example of filtering the data by another field for faceting, see the How to Use Facets with MongoDB Search tutorial.

multi Analyzers

Using a multi analyzer to analyze the same field multiple different ways can cause large indexes, especially when analyzing fields with very long values.

Multilingual Search

You can use the MongoDB Search Language Analyzers to index many languages. For the list of languages for which MongoDB Search provides built-in analyzers, see the Language Analyzers. To index and query languages that are currently not in the list of built-in Language Analyzers, you can create a custom analyzer. For an example, see the Custom Language Analyzer Example.

Suppose you have one document for each language in your collection. Consider the following:

• You can index the fields separately in the same index using the Language Analyzers. A single index can support multiple languages in the same query.

• Alternatively, you can create an index per language, which is useful in isolating the different language documents. Note that each index is a change stream cursor and so this might be expensive to maintain.

• If you have the language documents nested inside a parent document, you can create a single index. However, your index definition payload might be large and your query might be complex.

To learn how to do a Multilingual Search, see Multilingual Search Example

Synonym Collections

Inserts and updates to a synonym source collection are fast only if the synonym source collection is small. For best performance, we recommend batching inserts and updates to synonym source collections.

A synonym mapping definition doesn't require additional disk space aside from the disk space utilized by the synonym collection in the database. However, synonym mappings create artifacts in memory and therefore, for synonym collections with many documents, MongoDB Search creates artifacts that occupy more memory.