Ksenia Samokhvalova

Senior Product Designer

Amelia Short

Product Designer II

There are cases when it helps to have synonyms set up to work with your Atlas Search index. For example, if the search in your application needs to work with addresses, it might help to set up a list of common synonyms for postal abbreviations, so one could type in “blvd” instead of “boulevard” and still find all places with “boulevard” in the address.

This tutorial will show you how to set up your Atlas Search index to recognize US postal abbreviations.

## Prerequisites

To be successful with this tutorial, you will need:
* Python, to use a script that scrapes [a list of street suffix abbreviations](https://pe.usps.com/text/pub28/28apc_002.htm) helpfully compiled by the United States Postal Service (USPS). This tutorial was written using Python 3.10.15, but you could try it on earlier versions of 3, if you’d like.
* A MongoDB Atlas cluster. Follow the [Get Started with Atlas](https://docs.atlas.mongodb.com/getting-started/) guide to create your account and a MongoDB cluster. For this tutorial, you can use your [free-forever MongoDB Atlas cluster!](https://mongodb.prakticum-team.ru/cloud/atlas/register) Keep a note of your database username, password, and [connection string](https://docs.atlas.mongodb.com/tutorial/connect-to-your-cluster/#connect-to-your-atlas-cluster) as you will need those later.
* [Rosetta](https://appletoolbox.com/what-is-rosetta-2/), if you’re on a MacOS with an M1 chip. This will allow you to run MongoDB tools like mongoimport and mongosh. 
* mongosh for running commands in the MongoDB shell. If you don’t already have it, [install mongosh](https://mongodb.prakticum-team.ru/docs/mongodb-shell/install/).
* A copy of mongoimport. If you have MongoDB installed on your workstation, then you may already have mongoimport installed. If not, follow the instructions on the MongoDB website to [install mongoimport](https://mongodb.prakticum-team.ru/docs/database-tools/mongoimport/#installation). 
* We're going to be using a sample\_restaurants dataset in this tutorial since it contains address data. For instructions on how to load sample data, see the [documentation.](https://docs.atlas.mongodb.com/sample-data/available-sample-datasets/) Also, you can [see all available sample datasets](https://mongodb.prakticum-team.ru/docs/atlas/sample-data/#std-label-available-sample-datasets).

The examples shown here were all written on a MacOS but should run on any unix-type system. If you're running on Windows, we recommend running the example commands inside the [Windows Subsystem for Linux.](https://docs.microsoft.com/en-us/windows/wsl/install-win10)

## A bit about synonyms in Atlas Search
To learn about synonyms in Atlas Search, we suggest you start by checking out our [documentation](https://mongodb.prakticum-team.ru/docs/atlas/atlas-search/synonyms/). Synonyms allow you to index and search your collection for words that have the same or nearly the same meaning, or, in the case of our tutorial, you can search using different ways to write out an address and still get the results you expect. To set up and use synonyms in Atlas Search, you will need to:

1. Create a collection in the same database as the collection you’re indexing  containing the synonyms. Note that every document in the synonyms collection must have [<span class="colour" style="color:rgb(17, 85, 204)"><u>a specific format</u></span>](https://mongodb.prakticum-team.ru/docs/atlas/atlas-search/synonyms/#std-label-synonyms-coll-spec).
2. Reference your synonyms collection in your [search index definition](https://mongodb.prakticum-team.ru/docs/atlas/atlas-search/index-definitions/#std-label-ref-index-definitions) via a synonym mapping.
3. Reference your synonym mapping in the $search command with the [$text operator](https://mongodb.prakticum-team.ru/docs/atlas/atlas-search/text/#std-label-text-ref). 

We will walk you through these steps in the tutorial, but first, let’s start with creating the JSON documents that will form our synonyms collection.

## Scrape the USPS postal abbreviations page

We will use [the list of official street suffix abbreviations](https://pe.usps.com/text/pub28/28apc_002.htm) and [a list of secondary unit designators](https://pe.usps.com/text/pub28/28apc_003.htm) from the USPS website to create a JSON document for each set of the synonyms.

All documents in the synonyms collection must have a [specific format](https://mongodb.prakticum-team.ru/docs/atlas/atlas-search/synonyms/#format-of-synonyms-source-collection-documents)that specifies the type of synonyms—equivalent or explicit. Explicit synonyms have a one-way mapping. For example, if “boat” is explicitly mapped to “sail,” we’d be saying that if someone searches “boat,” we want to return all documents that include “sail” and “boat.” However, if we search the word “sail,” we would not get any documents that have the word “boat.” In the case of postal abbreviations, however, one can use all abbreviations interchangeably, so we will use the “equivalent” type of synonym in the mappingType field.

Here is a sample document in the synonyms collection for all the possible abbreviations of “avenue”:
```
“Avenue”: 

{

"mappingType":"equivalent",

"synonyms":["AVENUE","AV","AVEN","AVENU","AVN","AVNUE","AVE"]

}
```
We wrote the web scraping code for you in Python, and you can run it with the following commands to create a document for each synonym group:
```
git clone https://github.com/mongodb-developer/Postal-Abbreviations-Synonyms-Atlas-Search-Tutorial/ 

cd Postal-Abbreviations-Synonyms-Atlas-Search-Tutorial

python3 main.py
```
To see details of the Python code, read the rest of the section.

In order to scrape the USPS postal website, we will need to import the following packages/libraries and install them using PIP: [requests](https://pypi.org/project/requests/), [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/), and [pandas](https://pandas.pydata.org/). We’ll also want to import [json](https://docs.python.org/3/library/json.html) and [re](https://docs.python.org/3/library/re.html) for formatting our data when we’re ready:
```
import json

import requests

from bs4 import BeautifulSoup

import pandas as pd

import re
```
Let’s start with the Street Suffix Abbreviations page. We want to create objects that represent both the URL and the page itself:
```
# Create a URL object

streetsUrl = 'https://pe.usps.com/text/pub28/28apc_002.htm'

# Create object page

headers = {

    "User-Agent": 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Mobile Safari/537.36'}

streetsPage = requests.get(streetsUrl, headers=headers)
```
Next, we want to get the information on the page. We’ll start by parsing the HTML, and then get the table by its id:
```
# Obtain page's information

streetsSoup = BeautifulSoup(streetsPage.text, 'html.parser')
```

```
# Get the table by its id

streetsTable = streetsSoup.find('table', {'id': 'ep533076'})
```
Now that we have the table, we’re going to want to transform it into a [dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), and then format it in a way that’s useful for us:
```
# Transform the table into a list of dataframes

streetsDf = pd.read_html(str(streetsTable))
```
One thing to take note of is that in the table provided on USPS’s website, one primary name is usually mapped to multiple commonly used names.
![Table with USPS names and abbreviations](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/usps_abbreviations_e2a3892370.png)
This means we need to dynamically group together commonly used names by their corresponding primary name and compile that into a list:
```
# Group together all "Commonly Used Street Suffix or Abbreviation" entries

streetsGroup = streetsDf[0].groupby(0)[1].apply(list)
```
Once our names are all grouped together, we can loop through them and export them as individual JSON files.
```
for x in range(streetsGroup.size):


    dictionary = {

        "mappingType": "equivalent",

        "synonyms": streetsGroup[x]

    }


    # export the JSON into a file

    with open(streetsGroup.index.values[x] + ".json", "w") as outfile:

        json.dump(dictionary, outfile)
```
Now, let’s do the same thing for the Secondary Unit Designators page:

Just as before, we’ll start with getting the page and transforming it to a dataframe:
```
# Create a URL object

unitsUrl = 'https://pe.usps.com/text/pub28/28apc_003.htm'


unitsPage = requests.get(unitsUrl, headers=headers)


# Obtain page's information

unitsSoup = BeautifulSoup(unitsPage.text, 'html.parser')


# Get the table by its id

unitsTable = unitsSoup.find('table', {'id': 'ep538257'})


# Transform the table into a list of dataframes

unitsDf = pd.read_html(str(unitsTable))
```
If we look at the table more closely, we can see that one of the values is blank. While it makes sense that the USPS would include this in the table, it’s not something that we want in our synonyms list.
![Table with USPS descriptions and abbreviations](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/usps_description_abbreviation_1777e4c265.png)
To take care of that, we’ll simply remove all rows that have blank values:
```
unitsDf[0] = unitsDf[0].dropna()
```
Next, we’ll take our new dataframe and turn it into a list:
```
# Create a 2D list that we will use for our synonyms

unitsList = unitsDf[0][[0, 2]].values.tolist()
```
You may have noticed that some of the values in the table have asterisks in them. Let’s quickly get rid of them so they won’t be included in our synonym mappings:
```
# Remove all non-alphanumeric characters

unitsList = [[re.sub("[^ \w]"," ",x).strip().lower() for x in y] for y in unitsList]
```
Now we can now loop through them and export them as individual JSON files just as we did before. The one thing to note is that we want to restrict the range on which we’re iterating to include only the relevant data we want:
```
# Restrict the range to only retrieve the results we want

for x in range(1, len(unitsList) - 1):


    dictionary = {

        "mappingType": "equivalent",

        "synonyms": unitsList[x]

    }


    # export the JSON into a file

    with open(unitsList[x][0] + ".json", "w") as outfile:

        json.dump(dictionary, outfile)
```
## Create a synonyms collection with JSON schema validation
Now that we created the JSON documents for abbreviations, let’s load them all into a collection in the sample\_restaurants database. If you haven’t already created a MongoDB cluster, now is a good time to do that and load the sample data in.

The first step is to connect to your Atlas cluster. We will use mongosh to do it. If you don’t have mongosh installed, follow the [instructions](https://mongodb.prakticum-team.ru/docs/mongodb-shell/install/).

To connect to your Atlas cluster, you will need a [connection string](https://mongodb.prakticum-team.ru/docs/guides/atlas/connection-string/). Choose the “Connect with the MongoDB Shell” option and follow the instructions. Note that you will need to connect with a [database user](https://mongodb.prakticum-team.ru/docs/atlas/security-add-mongodb-users/) that has permissions to modify the database, since we would be creating a collection in the sample\_restaurant database. The command you need to enter in the terminal will look something like:
```
mongosh "mongodb+srv://cluster0.XXXXX.mongodb.net/sample_restaurant" --apiVersion 1 --username <USERNAME>
```
When prompted for the password, enter the database user’s password.

We created our synonym JSON documents in the right format already, but let’s make sure that if we decide to add more documents to this collection, they will also have the correct format. To do that, we will create a synonyms collection with a validator that uses [$jsonSchema](https://mongodb.prakticum-team.ru/docs/v5.0/reference/operator/query/jsonSchema/). The commands below will create a collection with the name “postal\_synonyms” in the sample\_restaurants database and ensure that only documents with correct format are inserted into the collection.
```
use('sample_restaurants')

db.createCollection("postal_synonyms", { validator: { $jsonSchema: { "bsonType": "object", "required": ["mappingType", "synonyms"], "properties": { "mappingType": { "type": "string", "enum": ["equivalent", "explicit"], "description": "must be a either equivalent or explicit" }, "synonyms": { "bsonType": "array", "items": { "type": "string" }, "description": "must be an Array with each item a string and is required" }, "input": { "type": "array", "items": { "type": "string" }, "description": "must be an Array and is each item is a string" } }, "anyOf": [{ "not": { "properties": { "mappingType": { "enum": ["explicit"] } }, "required": ["mappingType"] } }, { "required": ["input"] }] } } })

```
## Import the JSON files into the synonyms collection
We will use mongoimport to import all the JSON files we created.

You will need a [connection string](https://mongodb.prakticum-team.ru/docs/guides/atlas/connection-string/) for your Atlas cluster to use in the mongoimport command. If you don’t already have mongoimport installed, use the [instructions](https://mongodb.prakticum-team.ru/docs/database-tools/mongoimport/#installation) in the MongoDB documentation.

In the terminal, navigate to the folder where all the JSON files for postal abbreviation synonyms were created.
```
cat *.json | mongoimport --uri 'mongodb+srv://<USERNAME>:<PASSWORD>@cluster0.pwh9dzy.mongodb.net/sample_restaurants?retryWrites=true&w=majority' --collection='postal_synonyms'

```
If you liked mongoimport, check out this [very helpful mongoimport guide](https://mongodb.prakticum-team.ru/developer/products/mongodb/mongoimport-guide/).

Take a look at the synonyms collections you just created in Atlas. You should see around 229 documents there.

![Document with postal synonyms in Atlas UI
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/document_with_postal_synonyms_atlas_ui_4b65e482b1.png)

## Create a search index with synonyms mapping in JSON Editor

Now that we created the synonyms collection in our sample\_restaurants database, let’s put it to use.

Let’s start by creating a search index. Navigate to the Search tab in your Atlas cluster and click the “CREATE INDEX” button.

Since the Visual Index builder doesn’t support synonym mappings yet, we will choose JSON Editor and click Next:

![The Create Search Index UI in Atlas](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_search_index_ui_atlas_fc3e960409.png)
In the JSON Editor, pick restaurants collection in the sample\_restaurants database and enter the following into the index definition. Here, the source collection name refers to the name of the collection with all the postal abbreviation synonyms, which we named “postal\_synonyms.”
```
{

  "mappings": {

    "dynamic": true

  },

  "synonyms": [

    {

      "analyzer": "lucene.standard",

      "name": "synonym_mapping",

      "source": {

        "collection": "postal_synonyms"

      }

    }

    ]

}

```
![The Create Search Index JSON Editor UI in Atlas
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_search_index_json_editor_ui_atlas_7b7d85d542.png)
We are indexing the restaurants collection and creating a synonym mapping with the name “synonym\_mapping” that references the synonyms collection “postal\_synonyms.”

Click on Next and then on Create Search Index, and wait for the search index to build.

Once the index is active, we’re ready to test it out.

## Test that synonyms are working (aggregation pipeline in Atlas or Compass)

Now that we have an active search index, we’re ready to test that our synonyms are working. Let’s head to the Aggregation pipeline in the Collections tab to test different calls to $search. You can also [use Compass](https://mongodb.prakticum-team.ru/docs/atlas/compass-connection/), the MongoDB GUI, if you prefer.

Choose $search from the list of pipeline stages. The UI gives us a helpful placeholder for the $search command’s arguments.

Let’s look for all restaurants that are located on a boulevard. We will search in the “address.street” field, so the arguments to the $search stage will look like this:
```
{

  index: 'default',

  text: {

    query: 'boulevard',

    path: 'address.street'

  }

}

```
Let’s add a $count stage after the $search stage to see how many restaurants with an address that contains “boulevard” we found:
![Screenshot showing results of an aggregation pipeline with two stages and "boulevard" query term.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/results_aggregation_pipeline_8366b8a888.png)
As expected, we found a lot of restaurants with the word “boulevard” in the address. But what if we don’t want to have users type “boulevard” in the search bar? What would happen if we put in “blvd,” for example?

```
{

  index: 'default',

  text: {

    query: blvd,

    path: 'address.street'

  }

}
```

![Screenshot showing results of an aggregation pipeline with two stages and "blvd" query term.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/aggregation_pipline_two_stages_9d5398cb57.png)
Looks like it found us restaurants with addresses that have “blvd” in them. What about the addresses with “boulevard,” though? Those did not get picked up by the search. 

And what if we weren’t sure how to spell “boulevard” and just searched for “boul”? [USPS’s website](https://pe.usps.com/text/pub28/28apc_002.htm) tells us it’s an acceptable abbreviation for boulevard, but our $search finds nothing.
![Screenshot showing results of an aggregation pipeline with two stages and "boul" query term.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/results_aggregation_pipeline_boul_528fb41e77.png)
This is where our synonyms come in! We need to add a synonyms option to the text operator in the $search command and reference the synonym mapping’s name:
```
{

  index: 'default',

  text: {

    query: 'blvd',

    path: 'address.street',

    synonyms:'synonym_mapping'

  }

}
```

![Screenshot showing results of an aggregation pipeline with two stages, "boul" query term, and synonyms enabled.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/boul_query_term_f05944e2b0.png)
And there you have it! We found all the restaurants on boulevards, regardless of which way the address was abbreviated, all thanks to our synonyms.

## Conclusion

Synonyms is just one of many features [Atlas Search](https://mongodb.prakticum-team.ru/atlas/search) offers to give you all the necessary search functionality in your application. All of these features are available right now on [MongoDB Atlas](https://mongodb.prakticum-team.ru/cloud/atlas/register). We just showed you how to add support for common postal abbreviations to your Atlas Search index—what can you do with Atlas Search next? Try it now on your free-forever [MongoDB Atlas](https://mongodb.prakticum-team.ru/cloud/atlas/register) cluster and head over to [community forums](https://mongodb.prakticum-team.ru/community/forums/) if you have any questions!

This tutorial will show you how to set up your Atlas Search index to recognize US postal abbreviations. 

Ksenia Samokhvalova

Content by Ksenia Samokhvalova

Add US Postal Abbreviations to Your Atlas Search in 5 Minutes