An Introduction to Indexes for MongoDB Atlas Search

Nic Raboy7 min read • Published Jan 17, 2022 • Updated Feb 03, 2023

Atlas

Rate this tutorial

Imagine reading a long book like "A Song of Fire and Ice," "The Lord of the Rings," or "Harry Potter." Now imagine that there was a specific detail in one of those books that you needed to revisit. You wouldn't want to search every page in those long books to find what you were looking for. Instead, you'd want to use some sort of book index to help you quickly locate what you were looking for. This same concept of indexing content within a book can be carried to MongoDB Atlas Search with search indexes.

Atlas Search makes it easy to build fast, relevant, full-text search on top of your data in the cloud. It's fully integrated, fully managed, and available with every MongoDB Atlas cluster running MongoDB version 4.2 or higher.

Correctly defining your indexes is important because they are responsible for making sure that you're receiving relevant results when using Atlas Search. There is no one-size-fits-all solution and different indexes will bring you different benefits.

In this tutorial, we're going to get a gentle introduction to creating indexes that will be valuable for various full-text search use cases.

Before we get too invested in this introduction, it's important to note that Atlas Search uses Apache Lucene. This means that search indexes are not unique to Atlas Search and if you're already comfortable with Apache Lucene, your existing knowledge of indexing will transfer. However, the tutorial could act as a solid refresher regardless.

Understanding the Data Model for the Documents in the Example

Before we start creating indexes, we should probably define what our data model will be for the example. In an effort to cover various indexing scenarios, the data model will be complex.

Take the following for example:

1 {
2     "_id": "cea29beb0b6f7b9187666cbed2f070b3",
3     "name": "Pikachu",
4     "pokedex_entry": {
5         "red": "When several of these Pokemon gather, their electricity could build and cause lightning storms.",
6         "yellow": "It keeps its tail raised to monitor its surroundings. If you yank its tail, it will try to bite you."
7     },
8     "moves": [
9         {
10             "name": "Thunder Shock",
11             "description": "A move that may cause paralysis."
12         },
13         {
14             "name": "Thunder Wave",
15             "description": "An electrical attack that may paralyze the foe."
16         }
17     ],
18     "location": {
19         "type": "Point",
20         "coordinates": [-127, 37]
21     }
22 }

The above example document is around Pokemon, but Atlas Search can be used on whatever documents are part of your application.

Example documents like the one above allow us to use text search, geo search, and potentially others. For each of these different search scenarios, the index might change.

When we create an index for Atlas Search, it is created at the collection level.

Statically Mapping Fields in a Document or Dynamically Mapping Fields as the Schema Evolves

There are two ways to map fields within a document when creating an index:

Dynamic Mappings
Static Mappings

If your document schema is still changing or your use case doesn't allow for it to be rigidly defined, you might want to choose to dynamically map your document fields. A dynamic mapping will automatically assign fields when new data is inserted.

Take the following for example:

1 {
2     "mappings": {
3         "dynamic": true
4     }
5 }

The above JSON represents a valid index. When you add it to a collection, you are essentially mapping every field that exists in the documents and any field that might exist in the future.

We can do a simple search using this index like the following:

1 db.pokemon.aggregate([
2     {
3         "$search": {
4             "text": {
5                 "query": "thunder",
6                 "path": ["moves.name"]
7             }
8         }
9     }
10 ]);

We didn't explicitly define the fields for this index, but attempting to search for "thunder" within the moves array will give us matching results based on our example data.

To be clear, dynamic mappings can be applied at the document level or the field level. At the document level, a dynamic mapping automatically indexes all common data types. At both levels, it automatically indexes all new and existing data.

While convenient, having a dynamic mapping index on all fields of a document comes at a cost. These indexes will take up more disk space and may be less performant.

The alternative is to use a static mapping, in which case you specify the fields to map and what type of fields they are. Take the following for example:

1 {
2     "mappings": {
3         "dynamic": false,
4         "fields": {
5             "name": {
6                 "type": "string"
7             }
8         }
9     }
10 }

In the above example, the only field within our document that is being indexed is the name field.

The following search query would return results:

1 db.pokemon.aggregate([
2     {
3         "$search": {
4             "text": {
5                 "query": "pikachu",
6                 "path": ["name"]
7             }
8         }
9     }
10 ]);

If we try to search on any other field within our document, we won't end up with results because those fields are not statically mapped nor is the document schema dynamically mapped.

There is, however, a way to get the best of both worlds if we need it.

Take the following which uses static and dynamic mappings:

1 {
2     "mappings": {
3         "dynamic": false,
4         "fields": {
5             "name": {
6                 "type": "string"
7             },
8             "pokedex_entry": {
9                 "type": "document",
10                 "dynamic": true
11             }
12         }
13     }
14 }

In the above example, we are still using a static mapping for the name field. However, we are using a dynamic mapping on the pokedex_entry field. The pokedex_entry field is an object so any field within that object will get the dynamic mapping treatment. This means all sub-fields are automatically mapped, as well as any new fields that might exist in the future. This could be useful if you want to specify what top level fields to map, but map all fields within a particular object as well.

Take the following search query as an example:

1 db.pokemon.aggregate([
2     {
3         "$search": {
4             "text": {
5                 "query": "pokemon",
6                 "path": ["name", "pokedex_entry.red"]
7             }
8         }
9     }
10 ]);

The above search will return results if "pokemon" appears in the name field or the red field within the pokedex_entry object.

When using a static mapping, you need to specify a type for the field or have dynamic set to true on the field. If you only specify a type, dynamic defaults to false. If you only specify dynamic as true, then Atlas Search can automatically default certain field types (e.g., string, date, number).

Atlas Search Indexes for Complex Fields within a Document

With the basic dynamic versus static mapping discussion out of the way for MongoDB Atlas Search indexes, now we can focus on more complicated or specific scenarios.

Let's first take a look at what our fully mapped index would look like for the document in our example:

1 {
2     "mappings": {
3         "dynamic": false,
4         "fields": {
5             "name": {
6                 "type": "string"
7             },
8             "moves": {
9                 "type": "document",
10                 "fields": {
11                     "name": {
12                         "type": "string"
13                     },
14                     "description": {
15                         "type": "string"
16                     }
17                 }
18             },
19             "pokedex_entry": {
20                 "type": "document",
21                 "fields": {
22                     "red": {
23                         "type": "string"
24                     },
25                     "yellow": {
26                         "type": "string"
27                     }
28                 }
29             },
30             "location": {
31                 "type": "geo"
32             }
33         }
34     }
35 }

In the above example, we are using a static mapping for every field within our documents. An interesting thing to note is the moves array and the pokedex_entry object in the example document. Even though one is an array and the other is an object, the index is a document for both. While writing searches isn't the focus of this tutorial, searching an array and object would be similar using dot notation.

Had any of the fields been nested deeper within the document, the same approach would be applied. For example, we could have something like this:

1 {
2     "mappings": {
3         "dynamic": false,
4         "fields": {
5             "pokedex_entry": {
6                 "type": "document",
7                 "fields": {
8                     "gameboy": {
9                         "type": "document",
10                         "fields": {
11                             "red": {
12                                 "type": "string"
13                             },
14                             "yellow": {
15                                 "type": "string"
16                             }
17                         }
18                     }
19                 }
20             }
21         }
22     }
23 }

In the above example, the pokedex_entry field was changed slightly to have another level of objects. Probably not a realistic way to model data for this dataset, but it should get the point across about mapping deeper nested fields.

Changing the Options for Specific Mapped Fields

Up until now, each of the indexes have only had their types defined in the mapping. The default options are currently being applied to every field. Options are a way to refine the index further based on your data to ultimately get more relevant search results. Let's play around with some of the options within the mappings of our index.

Most of the fields in our example use the string data type, so there's so much more we can do using options. Let's see what some of those are.

1 {
2     "mappings": {
3         "dynamic": false,
4         "fields": {
5             "name": {
6                 "type": "string",
7                 "searchAnalyzer": "lucene.spanish",
8                 "ignoreAbove": 3000
9             }
10         }
11     }
12 }

In the above example, we are specifying that we want to use a language analyzer on the name field instead of the default standard analyzer. We're also saying that the name field should not be indexed if the field value is greater than 3000 characters.

The 3000 characters is just a random number for this example, but adding a limit, depending on your use case, could improve performance or the index size.

In a future tutorial, we're going to explore the finer details in regards to what the search analyzers are and what they can accomplish.

These are just some of the available options for the string data type. Each data type will have its own set of options. If you want to use the default for any particular option, it does not need to be explicitly added to the mapped field.

You can learn more about the data types and their indexing options in the official documentation.

Conclusion

You just received what was hopefully a gentle introduction to creating indexes to be used in Atlas Search. To use Atlas Search, you will need at least one index on your collection, even if it is a default dynamic index. However, if you know your schema and are able to create static mappings, it is usually the better way to go to fine-tune relevancy and performance.

To learn more about Atlas Search indexes and the various data types, options, and analyzers available, check out the official documentation.

To learn how to build more on Atlas Search, check out my other tutorials: Building an Autocomplete Form Element with Atlas Search and JavaScript and Visually Showing Atlas Search Highlights with JavaScript and HTML.

Have a question or feedback about this tutorial? Head to the MongoDB Community Forums and let's chat!

Rate this tutorial

Quickstart

Building AI and RAG Apps With MongoDB, Anyscale and PyMongo

Jul 17, 2024 | 7 min read

Tutorial

Semantic search with Jina Embeddings v2 and MongoDB Atlas

Dec 05, 2023 | 12 min read

Tutorial

Streaming Data from MongoDB to BigQuery Using Confluent Connectors

Jul 11, 2023 | 4 min read

Article

Easy Deployment of MEAN Stack with MongoDB Atlas, Cloud Run, and HashiCorp Terraform

Jan 30, 2023 | 4 min read

Understanding the Data Model for the Documents in the Example
Statically Mapping Fields in a Document or Dynamically Mapping Fields as the Schema Evolves
Atlas Search Indexes for Complex Fields within a Document
Changing the Options for Specific Mapped Fields
Conclusion