How to Index Token Fields

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

1
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
2
    --header "Accept: application/vnd.atlas.2025-03-12+json" \
3
    --header "Content-Type: application/json" \
4
    -X POST "https://cloud.mongodb.com/api/atlas/v2/groups/{groupId}/clusters/{clusterName}/search/indexes" \
5
    -d '{ "collectionName": "movies",
6
        "database": "sample_mflix",
7
        "name": "default",
8
        "type": "search",
9
        "definition": {
10
            "mappings": { 
11
              "dynamic": false, 
12
              "fields": { 
13
                "genres": [
14
                  {
15
                    "type": "string"
16
                  },
17
                  {
18
                    "type": "token"
19
                  }
20
                ]
21
              }
22
            }
23
        }
24
    }'

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
2
    --header "Accept: application/vnd.atlas.2025-03-12+json" \
3
    --header "Content-Type: application/json" \
4
    -X POST "https://cloud.mongodb.com/api/atlas/v2/groups/{groupId}/clusters/{clusterName}/search/indexes" \
5
    -d '{ "collectionName": "movies",
6
        "database": "sample_mflix",
7
        "name": "default",
8
        "type": "search",
9
        "definition": {
10
            "mappings": { 
11
              "dynamic": false, 
12
              "fields": { 
13
                "title": { 
14
                  "type": "token",
15
                  "normalizer": "lowercase" 
16
                } 
17
              }
18
            }
19
        }
20
    }'

{"collectionName":"comments","database":"sample_mflix","indexID":"687a8d52a3b4982b8c228383","latestDefinition":{"mappings":{"dynamic":true,"fields":{"title":{"normalizer":"lowercase","type":"token"}}}},"latestDefinitionVersion":{"version":0},"name":"default_15","queryable":false,"status":"PENDING","type":"search"}

1
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
2
    --header "Accept: application/vnd.atlas.2025-03-12+json" \
3
    --header "Content-Type: application/json" \
4
    -X POST "https://cloud.mongodb.com/api/atlas/v2/groups/{groupId}/clusters/{clusterName}/search/indexes" \
5
    -d '{ "collectionName": "movies",
6
        "database": "sample_mflix",
7
        "name": "default",
8
        "type": "search",
9
        "definition": {
10
            "mappings": { 
11
              "dynamic": false, 
12
              "fields": { 
13
                "genres": [
14
                  {
15
                    "type": "string"
16
                  },
17
                  {
18
                    "type": "token"
19
                  }
20
                ]
21
              }
22
            }
23
        }
24
    }'

{"collectionName":"movies","database":"sample_mflix","indexID":"687a8ce25532743c25aa2d34","latestDefinition":{"mappings":{"dynamic":true,"fields":{"genres":[{"type":"string"},{"type":"token"}]}}},"latestDefinitionVersion":{"version":0},"name":"default_20","queryable":false,"status":"PENDING","type":"search"}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

1
{
2
  "name": "default",
3
  "collectionName": "<collection>",
4
  "database": "<database>",
5
  "definition": {
6
    "mappings": {
7
      "dynamic": true|false,
8
      "fields": {
9
        "<field-name>": {
10
          "type": "token",
11
          "normalizer": "lowercase|none"
12
        }
13
      }
14
    }
15
  }
16
}

atlas clusters search indexes create --clusterName <cluster-name> --file <path-to-file>

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
{
2
  "name": "default",
3
  "collectionName": "movies",
4
  "database": "sample_mflix",
5
  "definition": {
6
    "mappings": {
7
      "dynamic": false,
8
      "fields": {
9
        "title": {
10
          "type": "token",
11
          "normalizer": "lowercase"
12
        }
13
      }
14
    }
15
  }
16
}

1
{
2
  "name": "default",
3
  "collectionName": "movies",
4
  "database": "sample_mflix",
5
  "definition": {
6
    "mappings": {
7
      "dynamic": false,
8
      "fields": {
9
        "genres": [
10
          {
11
            "type": "string"
12
          },
13
          {
14
            "type": "token"
15
          }
16
        ]
17
      }
18
    }
19
  }
20
}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
{
2
  "mappings": { 
3
    "dynamic": false, 
4
    "fields": { 
5
        "title": {
6
            "type": "token",
7
            "normalizer": "lowercase"
8
        }
9
    } 
10
  }
11
}

1
{
2
  "mappings": { 
3
    "dynamic": false, 
4
    "fields": { 
5
        "genres": [
6
          {
7
            "type": "string"
8
          },
9
          {
10
            "type": "token"
11
          }
12
        ]
13
    } 
14
  }
15
}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

 use <database>

switched to db <database>

db.<collection>.createSearchIndex(
  "default",
    {
      "mappings": {
        "dynamic": true|false,
        "fields": {
        "<field-name>": {
            "type": "token",
            "normalizer": "lowercase|none"
        }
      }
    }
  }
)

default

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
db.movies.createSearchIndex(
2
  "default",
3
  {
4
    "mappings": { 
5
      "dynamic": false,
6
      "fields": {
7
        "title": {
8
          "type": "token",
9
          "normalizer": "lowercase"
10
        } 
11
      }
12
    }
13
  }
14
)

1
db.movies.createSearchIndex(
2
  "default",
3
  {
4
    "mappings": { 
5
      "dynamic": false,
6
      "fields": {
7
        "genres": [
8
          {
9
            "type": "string"
10
          },
11
          {
12
            "type": "token"
13
          }
14
        ]
15
      }
16
    }
17
  }
18
)

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

Choose your preferred configuration method in the Atlas UI and then select the database and collection.

1
{
2
  "mappings": {
3
    "dynamic": true|false,
4
    "fields": {
5
      "<field-name>": {
6
        "type": "token",
7
        "normalizer": "lowercase|none"
8
      }
9
    }
10
  }
11
}

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
{
2
  "mappings": {
3
    "dynamic": false,
4
    "fields": {
5
      "title": {
6
        "type": "token",
7
        "normalizer": "lowercase"
8
      }
9
    }
10
  }
11
}

1
{
2
  "mappings": {
3
    "dynamic": false,
4
    "fields": {
5
      "genres": [
6
        {
7
          "type": "string"
8
        },
9
        {
10
          "type": "token"
11
        }
12
      ]
13
    }
14
  }
15
}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

1
# Create a new directory for the project
2
mkdir atlas-search-project && cd atlas-search-project

1
#include <mongoc/mongoc.h>
2
#include <stdlib.h>
3
4
int main (void)
5
{
6
    mongoc_client_t *client = NULL;
7
    mongoc_collection_t *collection = NULL;
8
    mongoc_database_t *database = NULL;
9
    bson_error_t error;
10
    bson_t cmd = BSON_INITIALIZER;
11
    bool ok = true;
12
13
    mongoc_init();
14
15
    // Connect to your Atlas deployment
16
    client = mongoc_client_new("<connectionString>");
17
    if (!client) {
18
        fprintf(stderr, "Failed to create a MongoDB client.\n");
19
        ok = false;
20
        goto cleanup;
21
    }
22
23
    // Access your database and collection
24
    database = mongoc_client_get_database(client, "<databaseName>");
25
    collection = mongoc_database_get_collection(database, "<collectionName>");
26
27
    // Specify the command and the new index
28
    const char *cmd_str = BSON_STR({
29
        "createSearchIndexes" : "<collectionName>",
30
        "indexes" : [ {
31
            "name" : "default",
32
            "definition" : {
33
                "mappings": { 
34
                    "dynamic": true|false, 
35
                    "fields": { 
36
                        "<fieldName>" : {
37
                            "type": "token",
38
                            "normalizer": "lowercase|none"
39
                        }
40
                    } 
41
                },
42
            }
43
	    } ]
44
	});
45
46
    // Convert your command to BSON
47
    if (!bson_init_from_json(&cmd, cmd_str, -1, &error)) {
48
        fprintf(stderr, "Failed to parse command: %s\n", error.message);
49
        ok = false;
50
        goto cleanup;
51
    }
52
53
    // Create the MongoDB Search index by running the command
54
    if (!mongoc_collection_command_simple (collection, &cmd, NULL, NULL, &error)) {
55
        fprintf(stderr, "Failed to run createSearchIndexes: %s\n", error.message);
56
        ok = false;
57
        goto cleanup;
58
    }
59
    printf ("Index created!\n");
60
61
cleanup:
62
   mongoc_collection_destroy (collection);
63
   mongoc_client_destroy (client);
64
   mongoc_database_destroy (database);
65
   bson_destroy (&cmd);
66
   mongoc_cleanup ();
67
   return ok ? EXIT_SUCCESS : EXIT_FAILURE;
68
}

mongodb://localhost:<port-number>/?directConnection=true

1
cmake_minimum_required(VERSION 3.11)
2
project(atlas-search-index LANGUAGES C)
3
add_executable (index.out create_index.c)
4
find_package(mongoc <version> REQUIRED)
5
target_link_libraries(index.out mongoc::mongoc)

cmake -S. -Bcmake-build
cmake --build cmake-build --target index.out
./cmake-build/index.out

Index created!

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
#include <mongoc/mongoc.h>
2
#include <stdlib.h>
3
4
int main (void)
5
{
6
    mongoc_client_t *client = NULL;
7
    mongoc_collection_t *collection = NULL;
8
    mongoc_database_t *database = NULL;
9
    bson_error_t error;
10
    bson_t cmd = BSON_INITIALIZER;
11
    bool ok = true;
12
13
    mongoc_init();
14
15
    // Connect to your Atlas deployment
16
    client = mongoc_client_new("<connectionString>");
17
    if (!client) {
18
        fprintf(stderr, "Failed to create a MongoDB client.\n");
19
        ok = false;
20
        goto cleanup;
21
    }
22
23
    // Access your database and collection
24
    database = mongoc_client_get_database(client, "sample_mflix");
25
    collection = mongoc_database_get_collection(database, "movies");
26
27
    // Specify the command and the new index
28
    const char *cmd_str = BSON_STR({
29
        "createSearchIndexes" : "movies",
30
        "indexes" : [ {
31
            "name" : "default",
32
            "definition" : {
33
                "mappings": { 
34
                    "dynamic": false, 
35
                    "fields": { 
36
                        "title" : {
37
                            "type": "token",
38
                            "normalizer": "lowercase"
39
                        }
40
                    } 
41
                },
42
            }
43
	    } ]
44
	});
45
46
    // Convert your command to BSON
47
    if (!bson_init_from_json(&cmd, cmd_str, -1, &error)) {
48
        fprintf(stderr, "Failed to parse command: %s\n", error.message);
49
        ok = false;
50
        goto cleanup;
51
    }
52
53
    // Create the MongoDB Search index by running the command
54
    if (!mongoc_collection_command_simple (collection, &cmd, NULL, NULL, &error)) {
55
        fprintf(stderr, "Failed to run createSearchIndexes: %s\n", error.message);
56
        ok = false;
57
        goto cleanup;
58
    }
59
    printf ("Index created!\n");
60
61
cleanup:
62
   mongoc_collection_destroy (collection);
63
   mongoc_client_destroy (client);
64
   mongoc_database_destroy (database);
65
   bson_destroy (&cmd);
66
   mongoc_cleanup ();
67
   return ok ? EXIT_SUCCESS : EXIT_FAILURE;
68
}

1
#include <mongoc/mongoc.h>
2
#include <stdlib.h>
3
4
int main (void)
5
{
6
    mongoc_client_t *client = NULL;
7
    mongoc_collection_t *collection = NULL;
8
    mongoc_database_t *database = NULL;
9
    bson_error_t error;
10
    bson_t cmd = BSON_INITIALIZER;
11
    bool ok = true;
12
13
    mongoc_init();
14
15
    // Connect to your Atlas deployment
16
    client = mongoc_client_new("<connectionString>");
17
    if (!client) {
18
        fprintf(stderr, "Failed to create a MongoDB client.\n");
19
        ok = false;
20
        goto cleanup;
21
    }
22
23
    // Access your database and collection
24
    database = mongoc_client_get_database(client, "sample_mflix");
25
    collection = mongoc_database_get_collection(database, "movies");
26
27
    // Specify the command and the new index
28
    const char *cmd_str = BSON_STR({
29
        "createSearchIndexes" : "movies",
30
        "indexes" : [ {
31
            "name" : "default",
32
            "definition" : {
33
                "mappings": { 
34
                    "dynamic": false, 
35
                    "fields": { 
36
                        "genres" : [
37
                            {
38
                                "type": "string"
39
                            },
40
                            {
41
                                "type": "token"
42
                            }
43
                        ]
44
                    } 
45
                },
46
            }
47
	    } ]
48
	});
49
50
    // Convert your command to BSON
51
    if (!bson_init_from_json(&cmd, cmd_str, -1, &error)) {
52
        fprintf(stderr, "Failed to parse command: %s\n", error.message);
53
        ok = false;
54
        goto cleanup;
55
    }
56
57
    // Create the MongoDB Search index by running the command
58
    if (!mongoc_collection_command_simple (collection, &cmd, NULL, NULL, &error)) {
59
        fprintf(stderr, "Failed to run createSearchIndexes: %s\n", error.message);
60
        ok = false;
61
        goto cleanup;
62
    }
63
    printf ("Index created!\n");
64
65
cleanup:
66
   mongoc_collection_destroy (collection);
67
   mongoc_client_destroy (client);
68
   mongoc_database_destroy (database);
69
   bson_destroy (&cmd);
70
   mongoc_cleanup ();
71
   return ok ? EXIT_SUCCESS : EXIT_FAILURE;
72
}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

# Create a new directory and initialize the project
mkdir atlas-search-project && cd atlas-search-project
dotnet new console
# Add the MongoDB .NET/C# Driver to your project
dotnet add package MongoDB.Driver

1
using MongoDB.Bson;
2
using MongoDB.Driver;
3
4
// connect to your Atlas deployment
5
var uri = "<connection-string>";
6
7
var client = new MongoClient(uri);
8
9
var db = client.GetDatabase("<database>");
10
var collection = db.GetCollection<BsonDocument>("<collection>");
11
12
// define your MongoDB Search index
13
var index =  new CreateSearchIndexModel(
14
  "default", new BsonDocument
15
  {
16
    { "mappings", new BsonDocument
17
      {
18
        { "dynamic", true|false },
19
        { "fields", new BsonDocument
20
          {
21
            { "<field-name>", new BsonDocument
22
              {
23
                { "type", "token" },
24
                { "normalizer", "lowercase|none" }
25
              }
26
            }
27
          }
28
        }
29
      }
30
    }
31
  });
32
33
var result = collection.SearchIndexes.CreateOne(index);
34
Console.WriteLine($"New index name: {result}");

mongodb://localhost:<port-number>/?directConnection=true

dotnet run Program.cs

New index name: default

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
using MongoDB.Bson;
2
using MongoDB.Driver;
3
4
// connect to your Atlas deployment
5
var uri = "<connection-string>";
6
7
var client = new MongoClient(uri);
8
9
var db = client.GetDatabase("sample_mflix");
10
var collection = db.GetCollection<BsonDocument>("movies");
11
12
// define your MongoDB Search index
13
var index =  new CreateSearchIndexModel(
14
  "default", new BsonDocument
15
  {
16
    { "mappings", new BsonDocument
17
      {
18
        { "dynamic", false },
19
        { "fields", new BsonDocument
20
          {
21
            { "title", new BsonDocument
22
              {
23
                { "type", "token" },
24
                { "normalizer", "lowercase" }
25
              }
26
            }
27
          }
28
        }
29
      }
30
    }
31
  });
32
33
var result = collection.SearchIndexes.CreateOne(index);
34
Console.WriteLine($"New index name: {result}");

1
using MongoDB.Bson;
2
using MongoDB.Driver;
3
4
// connect to your Atlas deployment
5
var uri = "<connection-string>";
6
7
var client = new MongoClient(uri);
8
9
var db = client.GetDatabase("sample_mflix");
10
var collection = db.GetCollection<BsonDocument>("movies");
11
12
// define your MongoDB Search index
13
var index = new CreateSearchIndexModel(
14
  "default", new BsonDocument
15
  {
16
    { "mappings", new BsonDocument
17
      {
18
        { "dynamic", false },
19
        { "fields", new BsonDocument
20
          {
21
            { "genres", new BsonArray
22
              {
23
                new BsonDocument
24
                {
25
                  { "type", "string" }
26
                },
27
                new BsonDocument
28
                {
29
                  { "type", "token" }
30
                }
31
              }
32
            }
33
          }
34
        }
35
      }
36
    }
37
  });
38
39
var result = collection.SearchIndexes.CreateOne(index);
40
Console.WriteLine($"New index name: {result}");

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

# Create a new directory for the project
mkdir atlas-search-project && cd atlas-search-project

1
#include <iostream>
2
3
#include <bsoncxx/builder/basic/document.hpp>
4
#include <bsoncxx/json.hpp>
5
#include <mongocxx/client.hpp>
6
#include <mongocxx/instance.hpp>
7
#include <mongocxx/uri.hpp>
8
9
using bsoncxx::builder::basic::kvp;
10
using bsoncxx::builder::basic::make_document;
11
12
13
int main(){
14
15
    mongocxx::instance instance;
16
    mongocxx::uri uri("<connection string>");
17
    mongocxx::client client(uri);
18
19
    auto db = client["<database>"];
20
    auto collection = db["<collection>"];
21
22
    // instantiate a ``mongocxx::search_index_view`` on your collection
23
    auto siv = collection.search_indexes();
24
25
    {
26
        auto name = "default";
27
        auto definition = make_document(
28
            kvp("mappings", make_document(
29
                kvp("dynamic", true|false),
30
                kvp("fields", make_document(
31
                    kvp("<field-name>", make_document(
32
                        kvp("type", "token"),
33
                        kvp("normalizer", "lowercase|none")
34
                    ))
35
                ))
36
            ))
37
        );
38
39
        auto model = mongocxx::search_index_model(name, definition.view());
40
41
        // Create the search index
42
        auto result = siv.create_one(model);
43
        std::cout << "New index name: " << result << std::endl;
44
    }
45
}

mongodb://localhost:<port-number>/?directConnection=true

c++ --std=c++17 CreateIndex.cpp $(pkg-config --cflags --libs libmongocxx) -o ./app.out
./app.out

New index name: default

dyld[54430]: Library not loaded: @rpath/libmongocxx._noabi.dylib

c++ --std=c++17 CreateIndex.cpp -Wl,-rpath,/usr/local/lib/ $(pkg-config --cflags --libs libmongocxx) -o ./app.out
./app.out

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
#include <iostream>
2
3
#include <bsoncxx/builder/basic/document.hpp>
4
#include <bsoncxx/json.hpp>
5
#include <mongocxx/client.hpp>
6
#include <mongocxx/instance.hpp>
7
#include <mongocxx/uri.hpp>
8
9
using bsoncxx::builder::basic::kvp;
10
using bsoncxx::builder::basic::make_document;
11
12
13
int main(){
14
15
    mongocxx::instance instance;
16
    mongocxx::uri uri("<connection string>");
17
    mongocxx::client client(uri);
18
19
    auto db = client["sample_mflix"];
20
    auto collection = db["movies"];
21
22
    // instantiate a ``mongocxx::search_index_view`` on your collection
23
    auto siv = collection.search_indexes();
24
25
    {
26
        auto name = "default";
27
        auto definition = make_document(
28
            kvp("mappings", make_document(
29
                kvp("dynamic", false),
30
                kvp("fields", make_document(
31
                    kvp("title", make_document(
32
                        kvp("type", "token"),
33
                        kvp("normalizer", "lowercase")
34
                    ))
35
                ))
36
            ))
37
        );
38
39
        auto model = mongocxx::search_index_model(name, definition.view());
40
41
        // Create the search index
42
        auto result = siv.create_one(model);
43
        std::cout << "New index name: " << result << std::endl;
44
    }
45
}

1
#include <iostream>
2
3
#include <bsoncxx/builder/basic/document.hpp>
4
#include <bsoncxx/json.hpp>
5
#include <mongocxx/client.hpp>
6
#include <mongocxx/instance.hpp>
7
#include <mongocxx/uri.hpp>
8
9
using bsoncxx::builder::basic::kvp;
10
using bsoncxx::builder::basic::make_document;
11
12
13
int main(){
14
15
    mongocxx::instance instance;
16
    mongocxx::uri uri("<connection string>");
17
    mongocxx::client client(uri);
18
19
    auto db = client["sample_mflix"];
20
    auto collection = db["movies"];
21
22
    // instantiate a ``mongocxx::search_index_view`` on your collection
23
    auto siv = collection.search_indexes();
24
25
    {
26
        auto name = "default";
27
        auto definition = make_document(
28
            kvp("mappings", make_document(
29
                kvp("dynamic", false),
30
                kvp("fields", make_document(
31
                    kvp("genres", [](bsoncxx::builder::basic::sub_array array) {
32
                        array.append(make_document(
33
                            kvp("type", "string")
34
                        ));
35
                        array.append(make_document(
36
                            kvp("type", "token")
37
                        ));
38
                    })
39
                ))
40
            ))
41
        );
42
43
        auto model = mongocxx::search_index_model(name, definition.view());
44
45
        // Create the search index
46
        auto result = siv.create_one(model);
47
        std::cout << "New index name: " << result << std::endl;
48
    }
49
}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

# Create a new directory and initialize the project
mkdir atlas-search-project && cd atlas-search-project
go mod init atlas-search-project
# Add the MongoDB Go Driver to your project
go get go.mongodb.org/mongo-driver/v2/mongo

1
package main
2
3
import (
4
	"context"
5
	"fmt"
6
	"log"
7
8
	"go.mongodb.org/mongo-driver/v2/bson"
9
	"go.mongodb.org/mongo-driver/v2/mongo"
10
	"go.mongodb.org/mongo-driver/v2/mongo/options"
11
)
12
13
func main() {
14
	ctx := context.Background()
15
	uri := "<connection-string>"
16
	client, err := mongo.Connect(options.Client().
17
		ApplyURI(uri))
18
	if err != nil {
19
		panic(err)
20
	}
21
22
	defer func() {
23
		if err := client.Disconnect(context.TODO()); err != nil {
24
			panic(err)
25
		}
26
	}()
27
28
	coll := client.Database("<database>").Collection("<collection>")
29
	const indexName = "default"
30
	opts := options.SearchIndexes().SetName(indexName).SetType("search")
31
32
	// Define the MongoDB Search index for the token field
33
	searchIndexModel := mongo.SearchIndexModel{
34
		Definition: bson.D{
35
			{"mappings", bson.D{
36
				{"dynamic", false},
37
				{"fields", bson.D{
38
					{"<field-name>", bson.D{
39
						{"type", "token"},
40
						{"normalizer", "lowercase|none"}, // Optional: specify a normalizer
41
					}},
42
				}},
43
			}},
44
		},
45
		Options: opts,
46
	}
47
48
	// Create the index
49
	searchIndexName, err := coll.SearchIndexes().CreateOne(ctx, searchIndexModel)
50
	if err != nil {
51
		log.Fatalf("Failed to create the Atlas Search index: %v", err)
52
	}
53
54
	fmt.Println("Name of index created: " + searchIndexName)
55
}

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

mongodb://localhost:<port-number>/?directConnection=true

go run create_index.go

New index name: default

Configure token Field Properties

The MongoDB Search token type takes the following parameters:

Try an Example for the token Type

The following index definition examples use collections in the sample data. After you load the sample data, you can use these examples to index fields as the token type.

To learn how to set up and run the following example code, see Define the Index for the token Type.

1
package main
2
3
import (
4
	"context"
5
	"fmt"
6
	"log"
7
8
	"go.mongodb.org/mongo-driver/v2/bson"
9
	"go.mongodb.org/mongo-driver/v2/mongo"
10
	"go.mongodb.org/mongo-driver/v2/mongo/options"
11
)
12
13
func main() {
14
	ctx := context.Background()
15
	uri := "<connection-string>"
16
	client, err := mongo.Connect(options.Client().
17
		ApplyURI(uri))
18
	if err != nil {
19
		panic(err)
20
	}
21
22
	defer func() {
23
		if err := client.Disconnect(context.TODO()); err != nil {
24
			panic(err)
25
		}
26
	}()
27
28
	coll := client.Database("sample_mflix").Collection("movies")
29
	const indexName = "default"
30
	opts := options.SearchIndexes().SetName(indexName).SetType("search")
31
32
	// Define the MongoDB Search index for the token field
33
	searchIndexModel := mongo.SearchIndexModel{
34
		Definition: bson.D{
35
			{"mappings", bson.D{
36
				{"dynamic", false},
37
				{"fields", bson.D{
38
					{"title", bson.D{
39
						{"type", "token"},
40
					}},
41
				}},
42
			}},
43
		},
44
		Options: opts,
45
	}
46
47
	// Create the index
48
	searchIndexName, err := coll.SearchIndexes().CreateOne(ctx, searchIndexModel)
49
	if err != nil {
50
		log.Fatalf("Failed to create the Atlas Search index: %v", err)
51
	}
52
53
	fmt.Println("Name of index created: " + searchIndexName)
54
}

1
package main
2
3
import (
4
	"context"
5
	"log"
6
7
	"go.mongodb.org/mongo-driver/v2/bson"
8
	"go.mongodb.org/mongo-driver/v2/mongo"
9
	"go.mongodb.org/mongo-driver/v2/mongo/options"
10
)
11
12
func main() {
13
	ctx := context.Background()
14
15
	// Retrieves your Atlas connection string
16
	const uri = "<connection-string>"
17
18
	// Connect to your Atlas cluster
19
	clientOptions := options.Client().ApplyURI(uri)
20
	client, err := mongo.Connect(clientOptions)
21
	if err != nil {
22
		log.Fatalf("Failed to connect to the server: %v", err)
23
	}
24
	defer func() {
25
		if err := client.Disconnect(ctx); err != nil {
26
			log.Fatalf("Failed to disconnect: %v", err)
27
		}
28
	}()
29
30
	// Set the namespace
31
	coll := client.Database("sample_mflix").Collection("movies")
32
33
	// Define your search index with key-value pairs
34
	searchIndexModel := mongo.SearchIndexModel{
35
		Definition: bson.D{
36
			{"mappings", bson.D{
37
				{"dynamic", false},
38
				{"fields", bson.D{
39
					{"genres", bson.A{
40
						bson.D{
41
							{"type", "string"},
42
						},
43
						bson.D{
44
							{"type", "token"},
45
						},
46
					}},
47
				}},
48
			}},
49
		},
50
		Options: opts,
51
	}
52
53
	// Create the index
54
	result, err := coll.SearchIndexes().CreateOne(ctx, searchIndexModel)
55
	if err != nil {
56
		log.Fatalf("Failed to create search index: %v", err)
57
	}
58
	log.Printf("New index name: %v", result)
59
}

token Type Limitations

When you index a field as the token type, you must index that field as string type also to query the text value using operators such as text, phrase, etc. For the following operators, you don't need to index the field as string type also to query the text value in the field:

• equals

• in

• range

Review the Behavior of the token Type

When you index a field as token type, MongoDB Search indexes the terms in the string as a single token (searchable term) and stores them in a columnar storage for efficient filtering or sort operations. You can use a normalizer to transform the token. By default, the normalizer is set to none and so MongoDB Search indexes strings in their original form.

The major difference between the MongoDB Search string and token types is that MongoDB Search creates one or more tokens for fields indexed as string type whereas MongoDB Search creates only a single token for fields indexed as the token type.

If a string being indexed as a token field type exceeds 8181 characters, MongoDB Search truncates it to 8181 characters before indexing.

Define the Index for the token Type

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>mongodb-driver-bom</artifactId>
            <version>5.5.1</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

dependencies {
    implementation(platform("org.mongodb:mongodb-driver-bom:5.5.1"))
}

<dependencies>
    <dependency>    
      <groupId>org.mongodb</groupId>    
      <artifactId>mongodb-driver-sync</artifactId>    
      <version>4.11.1</version>    
    </dependency>
</dependencies>

dependencies {
   // MongoDB Java Sync Driver v4.11.0 or later
   implementation 'org.mongodb:mongodb-driver-sync'
}

1
import com.mongodb.client.MongoClient;
2
import com.mongodb.client.MongoClients;
3
import com.mongodb.client.MongoCollection;
4
import com.mongodb.client.MongoDatabase;
5
import org.bson.Document;
6
7
public class CreateIndex {
8
    public static void main(String[] args) {
9
        // connect to your Atlas cluster
10
        String uri = "<connection-string>";
11
12
        try (MongoClient mongoClient = MongoClients.create(uri)) {
13
            // set namespace
14
            MongoDatabase database = mongoClient.getDatabase("<database>");
15
            MongoCollection<Document> collection = database.getCollection("<collection>");
16
17
            Document index = new Document("mappings",
18
                    new Document("dynamic", true|false)
19
                        .append("fields", new Document("<field-name>", 
20
                                              new Document("type", "token")
21
                                                  .append("normalizer", "lowercase|none")
22
                                                  )));
23
24
            String result = collection.createSearchIndex(index);
25
            System.out.println("New index name: " + result);
26
        }
27
    }
28
}