Token Filters

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type asciiConverter in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select asciiFolding from the dropdown.

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add to create the custom analyzer.

10. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.first_name field.

11. Select page_updated_by.first_name from the Field Name dropdown and String from the Data Type dropdown.

12. In the properties section for the data type, select asciiConverter from the Index Analyzer and Search Analyzer dropdowns.

13. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "page_updated_by": {
        "type": "document",
        "dynamic": false,
        "fields": {
          "first_name": {
            "type": "string",
            "analyzer": "asciiConverter"
          }
        }
      }
    }
  },
  "analyzers": [
    {
      "name": "asciiConverter",
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "asciiFolding"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type dmsAnalyzer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select daitchMokotoffSoundex from the dropdown and select the value shown in the following table for the originalTokens field:

   Field	Value
   originalTokens	include

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add to create the custom analyzer.

10. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.last_name field.

11. Select page_updated_by.last_name from the Field Name dropdown and String from the Data Type dropdown.

12. In the properties section for the data type, select dmsAnalyzer from the Index Analyzer and Search Analyzer dropdowns.

13. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "page_updated_by": {
        "type": "document",
        "dynamic": false,
        "fields": {
          "last_name": {
            "type": "string",
            "analyzer": "dmsAnalyzer"
          }
        }
      }
    }
  },
  "analyzers": [
    {
      "name": "dmsAnalyzer",
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "daitchMokotoffSoundex",
          "originalTokens": "include"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type titleAutocomplete in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select icuFolding from the dropdown and click Add token filter to add the token filter to your custom analyzer.

8. Click Add token filter to add another token filter.

9. Select edgeGram from the dropdown and type the value shown in the following table for the fields:

   Field	Value
   minGram	4
   maxGram	7

10. Click Add token filter to add the token filter to your custom analyzer.

11. Click Add to create the custom analyzer.

12. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the title field.

13. Select title from the Field Name dropdown and String from the Data Type dropdown.

14. In the properties section for the data type, select titleAutocomplete from the Index Analyzer and Search Analyzer dropdowns.

15. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "analyzer": "titleAutocomplete",
  "mappings": {
    "dynamic": false,
    "fields": {
      "title": {
        "type": "string",
        "analyzer": "titleAutocomplete"
      }
    }
  },
  "analyzers": [
    {
      "name": "titleAutocomplete",
      "charFilters": [],
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "icuFolding"
        },
        {
          "type": "edgeGram",
          "minGram": 4,
          "maxGram": 7
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type englishPossessiveStemmer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select englishPossessive from the dropdown.

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add to create the custom analyzer.

10. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the title field.

11. Select title from the Field Name dropdown and String from the Data Type dropdown.

12. In the properties section for the data type, select englishPossessiveStemmer from the Index Analyzer and Search Analyzer dropdowns.

13. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "mappings": {
    "fields": {
      "title": {
        "type": "string",
        "analyzer": "englishPossessiveStemmer"
      }
    }
  },
  "analyzers": [
    {
      "name": "englishPossessiveStemmer",
      "charFilters": [],
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "englishPossessive"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type wordDelimiterGraphFlatten in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select whitespace from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select wordDelimiterGraph from the dropdown and configure the following fields for the token filter.

   1. Select the following fields:

      Field	Value
      delimiterOptions.generateWordParts	true
      delimiterOptions.preserveOriginal	true

   2. Type SIGN_IN in the protectedWords.words field.

   3. Select protectedWords.ignoreCase.

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add token filter to add another token filter.

10. Select flattenGraph from the dropdown.

11. Click Add token filter to add the token filter to your custom analyzer.

12. Click Add to create the custom analyzer.

13. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

14. Select message from the Field Name dropdown and String from the Data Type dropdown.

15. In the properties section for the data type, select wordDelimiterGraphFlatten from the Index Analyzer and Search Analyzer dropdowns.

16. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "mappings": {
    "fields": {
      "message": {
        "type": "string",
        "analyzer": "wordDelimiterGraphFlatten"
      }
    }
  },
  "analyzers": [
    {
      "name": "wordDelimiterGraphFlatten",
      "charFilters": [],
      "tokenizer": {
        "type": "whitespace"
      },
      "tokenFilters": [
        {
          "type": "wordDelimiterGraph",
          "delimiterOptions" : {
            "generateWordParts" : true,
            "preserveOriginal" : true
          },
          "protectedWords": {
            "words": [
              "SIGN_IN"
            ],
            "ignoreCase": false
          }
        },
        {
          "type": "flattenGraph"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type diacriticFolder in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select keyword from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select icuFolding from the dropdown.

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add to create the custom analyzer.

10. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the text.sv_FI nested field.

11. Select text.sv_FI nested from the Field Name dropdown and String from the Data Type dropdown.

12. In the properties section for the data type, select diacriticFolder from the Index Analyzer and Search Analyzer dropdowns.

13. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "analyzer": "diacriticFolder",
  "mappings": {
    "fields": {
      "text": {
        "type": "document",
        "fields": {
          "sv_FI": {
            "analyzer": "diacriticFolder",
            "type": "string"
          }
        }
      }
    }
  },
  "analyzers": [
    {
      "name": "diacriticFolder",
      "charFilters": [],
      "tokenizer": {
        "type": "keyword"
      },
      "tokenFilters": [
        {
          "type": "icuFolding"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type textNormalizer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select whitespace from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select icuNormalizer from the dropdown and select nfkc from the normalizationForm dropdown.

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add to create the custom analyzer.

10. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

11. Select message from the Field Name dropdown and String from the Data Type dropdown.

12. In the properties section for the data type, select textNormalizer from the Index Analyzer and Search Analyzer dropdowns.

13. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "analyzer": "textNormalizer",
  "mappings": {
    "fields": {
      "message": {
        "type": "string",
        "analyzer": "textNormalizer"
      }
    }
  },
  "analyzers": [
    {
      "name": "textNormalizer",
      "charFilters": [],
      "tokenizer": {
        "type": "whitespace"
      },
      "tokenFilters": [
        {
          "type": "icuNormalizer",
          "normalizationForm": "nfkc"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type kStemmer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select lowercase from the dropdown and click Add token filter to add the token filter to your custom analyzer.

8. Click Add token filter to add another token filter.

9. Select kStemming from the dropdown.

10. Click Add token filter to add the token filter to your custom analyzer.

11. Click Add to create the custom analyzer.

12. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the text.en_US nested field.

13. Select text.en_US nested from the Field Name dropdown and String from the Data Type dropdown.

14. In the properties section for the data type, select kStemmer from the Index Analyzer and Search Analyzer dropdowns.

15. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "analyzer": "kStemmer",
  "mappings": {
    "dynamic": true
  },
  "analyzers": [
    {
      "name": "kStemmer",
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "lowercase"
        },
        {
          "type": "kStemming"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type longOnly in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select icuFolding from the dropdown and click Add token filter to add the token filter to your custom analyzer.

8. Click Add token filter to add another token filter.

9. Select length from the dropdown and configure the following field for the token filter:

   Field	Value
   min	20

10. Click Add token filter to add the token filter to your custom analyzer.

11. Click Add to create the custom analyzer.

12. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the text.sv.FI nested field.

13. Select text.sv.FI nested from the Field Name dropdown and String from the Data Type dropdown.

14. In the properties section for the data type, select longOnly from the Index Analyzer and Search Analyzer dropdowns.

15. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "mappings": {
    "fields": {
      "text": {
        "type": "document",
        "dynamic": true,
        "fields": {
          "sv_FI": {
            "type": "string",
            "analyzer": "longOnly"
          }
        }
      }
    }
  },
  "analyzers": [
    {
      "name": "longOnly",
      "charFilters": [],
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "icuFolding"
        },
        {
          "type": "length",
          "min": 20
        }
      ]
    }
  ]
}

The following example index definition indexes the title field in the minutes collection as type autocomplete with the nGram tokenization strategy. It applies a custom analyzer named keywordLowerer on the title field. The custom analyzer specifies the following:

1. Apply keyword tokenizer to create a single token for a string or array of strings.

2. Apply the lowercase token filter to convert token text to lowercase.

{
  "mappings": {
    "fields": {
      "title": {
        "analyzer": "keywordLowerer",
        "tokenization": "nGram",
        "type": "autocomplete"
      }
    }
  },
  "analyzers": [
    {
      "name": "keywordLowerer",
      "charFilters": [],
      "tokenizer": {
        "type": "keyword"
      },
      "tokenFilters": [
        {
          "type": "lowercase"
        }
      ]
    }
  ]
}

The following query searches the title field using the autocomplete operator for the characters standup.

db.minutes.aggregate([
  {
    "$search": {
      "index": "default",
      "autocomplete": {
        "query": "standup",
        "path": "title"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "title": 1
    }
  }
])

[ { _id: 4, title: 'The daily huddle on tHe StandUpApp2' } ]

Atlas Search returns the document with _id: 4 in the results because the document contains the query term standup. Atlas Search creates tokens for the title field using the keyword tokenizer, lowercase token filter, and the nGram tokenization strategy for the autocomplete type. Specifically, Atlas Search uses the keyword tokenizer to tokenize the entire string as a single token, which supports only exact matches on the entire string, and then it uses the lowercase token filter to convert the tokens to lowercase. For the document in the results, Atlas Search creates the following token using the custom analyzer:

After applying the custom analyzer, Atlas Search creates further tokens of n-grams because Atlas Search indexes the title field as the autocomplete type as specified in the index definition. Atlas Search uses the tokens of n-grams, which includes a token for standup, to match the document to the query term standup.

The following index definition indexes the message field in the minutes collection using a custom analyzer named lowerCaser. The custom analyzer specifies the following:

1. Apply standard tokenizer to create tokens based on word break rules.

2. Apply the following filters on the tokens:

   • icuNormalizer to normalize the tokens using a standard Unicode Normalization Mode.

   • lowercase token filter to convert token text to lowercase.

{
  "mappings": {
    "fields": {
      "message": {
        "type": "string",
        "analyzer": "lowerCaser"
      }
    }
  },
  "analyzers": [
    {
      "name": "lowerCaser",
      "charFilters": [],
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "icuNormalizer",
          "normalizationForm": "nfkd"
        },
        {
          "type": "lowercase"
        }
      ]
    }
  ]
}

The following query searches the message field for the term sign-in.

db.minutes.aggregate([
  {
    "$search": {
      "index": "default",
      "text": {
        "query": "sign-in",
        "path": "message"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "message": 1
    }
  }
])

[
  { _id: 1, message: 'try to siGn-In' },
  { _id: 3, message: 'try to sign-in' },
  { _id: 2, message: 'do not forget to SIGN-IN. See ① for details.' }
]

Atlas Search returns the documents with _id: 1, _id: 3, and _id: 2 in the results for the query term sign-in because the icuNormalizer tokenizer first creates separate tokens by splitting the text, including the hyphenated word, but retains the original letter case in the document and then the lowercase token filter converts the tokens to lowercase. Atlas Search also analyzes the query term using the index analyzer (or if specified, using the searchAnalyzer) to split the query term and match it to the document.

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type titleAutocomplete in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select englishPossessive from the dropdown and click Add token filter to add the token filter to your custom analyzer.

8. Click Add token filter to add another token filter.

9. Select nGram from the dropdown and configure the following fields for the token filter:

   Field	Value
   minGram	4
   maxGram	7

10. Click Add token filter to add the token filter to your custom analyzer.

11. Click Add to create the custom analyzer.

12. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the title field.

13. Select title from the Field Name dropdown and String from the Data Type dropdown.

14. In the properties section for the data type, select titleAutocomplete from the Index Analyzer and Search Analyzer dropdowns.

15. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
 "mappings": {
    "fields": {
      "title": {
        "type": "string",
        "analyzer": "titleAutocomplete"
      }
    }
  },
  "analyzers": [
    {
      "name": "titleAutocomplete",
      "charFilters": [],
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "englishPossessive"
        },
        {
          "type": "nGram",
          "minGram": 4,
          "maxGram": 7
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type porterStemmer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select lowercase from the dropdown and click Add token filter to add the token filter to your custom analyzer.

8. Click Add token filter to add another token filter.

9. Select porterStemming from the dropdown.

10. Click Add token filter to add the token filter to your custom analyzer.

11. Click Add to create the custom analyzer.

12. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the title field.

13. Select title from the Field Name dropdown and String from the Data Type dropdown.

14. In the properties section for the data type, select porterStemmer from the Index Analyzer and Search Analyzer dropdowns.

15. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "mappings": {
    "fields": {
      "title": {
        "type": "string",
        "analyzer": "porterStemmer"
      }
    }
  },
  "analyzers": [
    {
      "name": "porterStemmer",
      "tokenizer": {
        "type": "standard"
      },
      "tokenFilters": [
        {
          "type": "lowercase"
        },
        {
          "type": "porterStemming"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type emailRedact in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select keyword from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select lowercase from the dropdown and click Add token filter to add the token filter to your custom analyzer.

8. Click Add token filter to add another token filter.

9. Select regex from the dropdown and configure the following for the token filter:

   1. Type ^([a-z0-9_\\.-]+)@([\\da-z\\.-]+)\\.([a-z\\.]{2,5})$ in the pattern field.

   2. Type redacted in the replacement field.

   3. Select all from the matches dropdown.

10. Click Add token filter to add the token filter to your custom analyzer.

11. Click Add to create the custom analyzer.

12. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.email nested field.

13. Select page_updated_by.email nested from the Field Name dropdown and String from the Data Type dropdown.

14. In the properties section for the data type, select emailRedact from the Index Analyzer and Search Analyzer dropdowns.

15. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "analyzer": "lucene.standard",
  "mappings": {
    "dynamic": false,
    "fields": {
      "page_updated_by": {
        "type": "document",
        "fields": {
          "email": {
            "type": "string",
            "analyzer": "emailRedact"
          }
        }
      }
    }
  },
  "analyzers": [
    {
      "charFilters": [],
      "name": "emailRedact",
      "tokenizer": {
        "type": "keyword"
      },
      "tokenFilters": [
        {
          "type": "lowercase"
        },
        {
          "matches": "all",
          "pattern": "^([a-z0-9_\\.-]+)@([\\da-z\\.-]+)\\.([a-z\\.]{2,5})$",
          "replacement": "redacted",
          "type": "regex"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type keywordReverse in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select keyword from the dropdown.

6. Expand Token Filters and click Add token filter.

7. Select reverse from the dropdown.

8. Click Add token filter to add the token filter to your custom analyzer.

9. Click Add to create the custom analyzer.

10. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.email nested field.

11. Select page_updated_by.email nested from the Field Name dropdown and String from the Data Type dropdown.

12. In the properties section for the data type, select keywordReverse from the Index Analyzer and Search Analyzer dropdowns.

13. Click Add, then Save Changes.

Replace the default index definition with the following example:

{
  "analyzer": "lucene.keyword",
  "mappings": {
    "dynamic":
  },
  "analyzers": [
    {
      "name": "keywordReverse",
      "charFilters": [],
      "tokenizer": {
        "type": "keyword"
      },
      "tokenFilters": [
        {
          "type": "reverse"
        }
      ]
    }
  ]
}

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type emailAutocompleteIndex in the Analyzer Name field.

4. Expand Character Filters and click Add character filter.

5. Select mapping from the dropdown and click Add mapping.

6. Enter the following key and value:

   Key	Value
   @	AT

7. Click Add character filter to add the character filter to your custom analyzer.

8. Expand Tokenizer if it's collapsed.

9. Select whitespace from the dropdown and enter 15 in the maxTokenLength field.

10. Expand Token Filters and click Add token filter.

11. Select shingle from the dropdown and configure the following fields.

    Field	Field Value
    minShingleSize	2
    minShingleSize	3

12. Click Add token filter to add another token filter.

13. Click Add token filter to add another token filter.

14. Select edgeGram from the dropdown and configure the following fields for the token filter:

    Field	Field Value
    minGram	2
    maxGram	15

15. Click Add token filter to add the token filter to your custom analyzer.

16. Click Add to add the custom analyzer to your index.

17. In the Custom Analyzers section, click Add Custom Analyzer.

18. Select the Create Your Own radio button and click Next.

19. Type emailAutocompleteSearch in the Analyzer Name field.

20. Expand Character Filters and click Add character filter.

21. Select mapping from the dropdown and click Add mapping.

22. Enter the following key and value:

    Key	Value
    @	AT

23. Click Add character filter to add the character filter to your custom analyzer.

24. Expand Tokenizer if it's collapsed.

25. Select whitespace from the dropdown and enter 15 in the maxTokenLength field.

26. Click Add to add the custom analyzer to your index.

27. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.email nested field.

28. Select page_updated_by.email nested from the Field Name dropdown and String from the Data Type dropdown.

29. In the properties section for the data type, select emailAutocompleteIndex from the Index Analyzer dropdown and emailAutocompleteSearch from the Search Analyzer dropdown.

30. Click Add, then Save Changes.