Brian Leonard

Solutions Architect


# Semantic Search Made Easy With LangChain and MongoDB

Enabling semantic search on user-specific data is a multi-step process that includes loading, transforming, embedding and storing data before it can be queried. 

![](https://python.langchain.com/v0.1/assets/images/data_connection-95ff2033a8faa5f3ba41376c0f6dd32a.jpg)

That graphic is from the team over at [LangChain](https://python.langchain.com/docs/modules/data_connection/), whose goal is to provide a set of utilities to greatly simplify this process. 

In this tutorial, we'll walk through each of these steps, using MongoDB Atlas as our Store. Specifically, we'll use the [AT&T](https://en.wikipedia.org/wiki/AT%26T) and [Bank of America](https://en.wikipedia.org/wiki/Bank_of_America) Wikipedia pages as our data source. We'll then use libraries from LangChain to Load, Transform, Embed and Store: 

![](https://raw.githubusercontent.com/wbleonard/atlas-langchain/main/images/flow-store.png)

Once the source is store is stored in MongoDB, we can retrieve the data that interests us:

![](https://raw.githubusercontent.com/wbleonard/atlas-langchain/main/images/flow-retrieve.png)


## Prerequisites
* [MongoDB Atlas Subscription](https://mongodb.prakticum-team.ru/proxy/cloud.mongodb.com/) (Free Tier is fine)
* Open AI [API key](https://platform.openai.com/account/api-keys)

## Quick Start Steps
1. Get the code:
```zsh
git clone https://github.com/wbleonard/atlas-langchain.git
```
2. Update [params.py](params.py) with your MongoDB connection string and Open AI [API key](https://platform.openai.com/account/api-keys).
3. Create a new Python environment
```zsh
python3 -m venv env
```
4. Activate the new Python environment
```zsh
source env/bin/activate
```

5. Install the requirements
```zsh
pip3 install -r requirements.txt
```
6. Load, Transform, Embed and Store
```zsh
python3 vectorize.py
```

7. Retrieve
```zsh
python3 query.py -q "Who started AT&T?"
```

## The Details
### Load -> Transform -> Embed -> Store 
#### Step 1: Load
There's no lacking for sources of data: Slack, YouTube, Git, Excel, Reddit, Twitter, etc., and [LangChain provides a growing list](https://python.langchain.com/v0.2/api_reference/community/document_loaders.html) of integrations that includes this list and many more.

For this exercise, we're going to use the [WebBaseLoader](https://python.langchain.com/v0.2/api_reference/community/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) to load the Wikipedia pages for [AT&T](https://en.wikipedia.org/wiki/AT%26T) and [Bank of America](https://en.wikipedia.org/wiki/Bank_of_America). 

```python
from langchain_community.document_loaders import WebBaseLoader

# Step 1: Load
loaders = [
 WebBaseLoader("https://en.wikipedia.org/wiki/AT%26T"),
 WebBaseLoader("https://en.wikipedia.org/wiki/Bank_of_America")
]

docs = []

for loader in loaders:
    for doc in loader.lazy_load():
        docs.append(doc)

```

 #### Step 2: Transform (Split)
 Now that we have a bunch of text loaded, it needs to be split into smaller chunks so we can tease out the relevant portion based on our search query. For this example we'll use the recommended [RecursiveCharacterTextSplitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter). As I have it configured, it attempts to split on paragraphs (`"\n\n"`), then sentences(`"(?<=\. )"`), then words (`" "`) using a chunk size of 1000 characters. So if a paragraph doesn't fit into 1000 characters, it will truncate at the next word it can fit to keep the chunk size under 1000 chacters. You can tune the `chunk_size` to your liking. Smaller numbers will lead to more documents, and vice-versa.

```python
# Step 2: Transform (Split)
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0, separators=[
                                              "\n\n", "\n", r"(?<=\. )", " "], length_function=len)
docs = text_splitter.split_documents(docs)
```

#### Step 3: Embed
[Embedding](https://python.langchain.com/docs/modules/data_connection/text_embedding/) is where you use an LLM to create a vector representation text. There are many options to choose from, such as [OpenAI](https://openai.com/) and [Hugging Face](https://huggingface.co/), and LangChang provides a standard interface for interacting with all of them. 

For this exercise we're going to use the popular [OpenAI embedding](https://api.python.langchain.com/en/latest/embeddings/langchain_openai.embeddings.base.OpenAIEmbeddings.html#langchain_openai.embeddings.base.OpenAIEmbeddings). Before proceeding, you'll need an [API key](https://platform.openai.com/account/api-keys) for the OpenAI platform, which you will set in [params.py](params.py).

We're simply going to load the embedder in this step. The real power comes when we store the embeddings in Step 4. 

```python
# Step 3: Embed
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(openai_api_key=params.OPENAI_API_KEY)
```

#### Step 4: Store
You'll need a vector database to store the embeddings, and lucky for you MongoDB fits that bill. Even luckier for you, the folks at LangChain have a [MongoDB Atlas module](https://python.langchain.com/v0.2/docs/integrations/vectorstores/mongodb_atlas/) that will do all the heavy lifting for you! Don't forget to add your MongoDB Atlas connection string to [params.py](params.py).

```python
# Step 4: Store
from pymongo import MongoClient
from langchain_mongodb.vectorstores import MongoDBAtlasVectorSearch

client = MongoClient(params.MONGODB_CONN_STRING)
collection = client[params.DB_NAME][params.COLL_NAME]

# Insert the documents in MongoDB Atlas with their embedding
docsearch = MongoDBAtlasVectorSearch.from_documents(
    docs, embeddings, collection=collection, index_name=index_name
)
```

#### Step 5: Index the Vector Embeddings
The final step before we can query the data is to create a [search index on the stored embeddings](https://mongodb.prakticum-team.ru/docs/atlas/atlas-search/field-types/knn-vector/). 

If you're on Atlas dedicated compute, Langchain can do this for you. 

```python
# Step 5: Create Vector Search Index
# THIS ONLY WORKS ON DEDICATED CLUSTERS (M10+)
docsearch.create_vector_search_index(dimensions=1536, update=True)
```
If you are on shared compute (M0, M2 or M5), in the Atlas console, create a Atlas Vector Search `langchain_vsearch_index` with the following definition: 

```JSON
{
    "fields": [
    {
      "type": "vector",
      "path": "embedding",
      "numDimensions": 1536,
      "similarity": "cosine"
    }
  ]
}
```

![](https://raw.githubusercontent.com/wbleonard/atlas-langchain/main/images/create-search-index.png)

![](https://raw.githubusercontent.com/wbleonard/atlas-langchain/main/images/create-search-index2.png)

![](https://raw.githubusercontent.com/wbleonard/atlas-langchain/main/images/create-search-index3.png)

You'll find the complete script in [vectorize.py](vectorize.py), which needs to be run only once or when new data sources are added.

```zsh
python3 vectorize.py
```

### Retrieve 
We could now run a search, using methods like [similirity_search](https://python.langchain.com/v0.2/api_reference/mongodb/vectorstores/langchain_mongodb.vectorstores.MongoDBAtlasVectorSearch.html#langchain_mongodb.vectorstores.MongoDBAtlasVectorSearch.similarity_search) or [max_marginal_relevance_search](https://python.langchain.com/v0.2/api_reference/mongodb/vectorstores/langchain_mongodb.vectorstores.MongoDBAtlasVectorSearch.html#langchain_mongodb.vectorstores.MongoDBAtlasVectorSearch.max_marginal_relevance_search) and that would return the relevant slice of data, which in our case would be an entire paragraph. However, we can continue to harness the power of the LLM to [contextually compress](https://python.langchain.com/v0.2/api_reference/langchain/retrievers/langchain.retrievers.contextual_compression.ContextualCompressionRetriever.html#contextualcompressionretriever) the response so that it more directly tries to answer our question. 

```python
from pymongo import MongoClient
from langchain_mongodb.vectorstores import MongoDBAtlasVectorSearch
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

# Initialize MongoDB python client
client = MongoClient(params.MONGODB_CONN_STRING)
collection = client[params.DB_NAME][params.COLL_NAME]

# initialize vector store
vectorStore = MongoDBAtlasVectorSearch(
    collection, OpenAIEmbeddings(openai_api_key=params.OPENAI_API_KEY), index_name=params.INDEX_NAME
)
# perform a search between the embedding of the query and the embeddings of the documents
print("\nQuery Response:")
print("---------------")
docs = vectorStore.max_marginal_relevance_search(query, K=1)
#docs = vectorStore.similarity_search(query, K=1)

print(docs[0].metadata['title'])
print(docs[0].page_content)

# Contextual Compression
llm = OpenAI(openai_api_key=params.OPENAI_API_KEY, temperature=0)
compressor = LLMChainExtractor.from_llm(llm)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorStore.as_retriever()
)
```

```zsh
python3 query.py -q "Who started AT&T?"

Your question:
-------------
Who started AT&T?

AI Response:
-----------
AT&T - Wikipedia
"AT&T was founded as Bell Telephone Company by Alexander Graham Bell, Thomas Watson and Gardiner Greene Hubbard after Bell's patenting of the telephone in 1875."[25] "On December 30, 1899, AT&T acquired the assets of its parent American Bell Telephone, becoming the new parent company."[28]
```

## Resources
* [MongoDB Atlas](https://mongodb.prakticum-team.ru/proxy/cloud.mongodb.com/)
* [Open AI API key](https://platform.openai.com/account/api-keys)
* [LangChain](https://python.langchain.com)
  * [WebBaseLoader](https://python.langchain.com/v0.2/api_reference/community/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html)
  * [RecursiveCharacterTextSplitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter)
  * [MongoDB Atlas module](https://python.langchain.com/v0.2/docs/integrations/vectorstores/mongodb_atlas/)
  * [Contextual Compression. ](https://python.langchain.com/v0.2/api_reference/langchain/retrievers/langchain.retrievers.contextual_compression.ContextualCompressionRetriever.html#contextualcompressionretriever)
  * [MongoDBAtlasVectorSearch API](https://python.langchain.com/v0.2/api_reference/mongodb/index.html)




Dive into semantic search with our tutorial on integrating LangChain and MongoDB. Simplify loading, transforming, embedding, and storing data.

Simplify Semantic Search With LangChain and MongoDB

Atlas offers the ability to pause clusters that aren't currently needed. Learn to automatically pause clusters that go unused for any period of time.

Auto Pausing Inactive Clusters

# Integrating MongoDB Change Streams with Socket.IO

## Introduction

The [Socket.IO Getting Started guide](https://socket.io/get-started/chat) provides a nice introduction to [Socket.IO](https://socket.io/). The guide bundles the server and client into a single application where messages submitted via an HTML input form are received and displayed on the page. 

Since MongoDB supports an exceptional [eventing framework](https://mongodb.prakticum-team.ru/docs/manual/changeStreams/) of its own, this tutorial will demonstrate how to propagate events emitted from MongoDB through to Socket.IO. To keep things consistent, I will try to mirror the Socket.IO Getting Started guide as much as possible. 

![Architecture](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/architecture.png)

Let's get started...

## The web framework

As with the Socket.IO Getting Started guide, we're going to set up a simple HTML webpage, however, in our example, it's only going to display a list of messages -- there will be no input form.

First let's create a `package.json` manifest file that describes our project. I recommend you place it in a dedicated empty directory (I'll call mine `mongo-socket-chat-example`).


```JSON
{
 "name": "monngo-socket-chat-example",
 "version": "0.0.1",
 "description": "my first mongo socket.io app",
 "dependencies": {}
}
```

Then use `npm` to install `express`:
```zsh
npm install express@4
```

One express is installed, we can set up an `index.js` file that will set up our application.


```JavaScript
const express = require('express');
const app = express();
const http = require('http');
const server = http.createServer(app);

app.get('/', (req, res) => {
 res.send('<h1>Hello world</h1>');
});

server.listen(3000, () => {
 console.log('listening on *:3000');
});
```


This means that:



* Express initializes `app` to be a function handler that you can supply to an HTTP server (as seen in line 4).
* We define a route handler / that gets called when we hit our website home.
* We make the HTTP server listen on port 3000.

If you run `node index.js` you should see the following:

![Running the Web Framework](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/run1.png)

And if you point your browser to [`http://localhost:3000`](http://localhost:3000):

![Hello World](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/result1.png)

## Serving HTML

So far in `index.js` we are calling `res.send` and passing it a string of HTML. Our code would look very confusing if we just placed our entire application’s HTML there, so  we're going to create an `index.html` file and serve that instead.

Let’s refactor our route handler to use `sendFile` instead.

```JavaScript
app.get('/', (req, res) => {

 res.sendFile(__dirname + '/index.html');

});
```

Here's a simple `index.html` file to display a list of messages with some styling included:

```HTML
<!DOCTYPE html>
<html>
  <head>
    <title>Socket.IO chat</title>
    <style>
      body { margin: 0; padding-bottom: 3rem; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; }

      #messages { list-style-type: none; margin: 0; padding: 0; }
      #messages > li { padding: 0.5rem 1rem; }
      #messages > li:nth-child(odd) { background: #efefef; }
    </style>
  </head>
  <body>
    <h1>Hello world from file</h1>
    <ul id="messages"></ul>
</body>
</html>

```

If you restart the process (by hitting Control+C and running `node index.js` again) and refresh the page it should look like this:

![Hello World Refactored](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/result2.png)

## Integrating MongoDB

Since you're here, I'm going to assume you already have access to a MongoDB cluster. If you do not, just follow these [Create a MongoDB Cluster](https://mongodb.prakticum-team.ru/docs/drivers/node/current/quick-start/#create-a-mongodb-cluster) instructions.

   > Note, a cluster (replica set) is required because we'll be using [change streams](https://mongodb.prakticum-team.ru/docs/manual/changeStreams/), which require an [oplog](https://mongodb.prakticum-team.ru/docs/manual/core/replica-set-oplog/). There's no need to worry, however, as it is easy enough to [configure a single node replica set](https://mongodb.prakticum-team.ru/docs/manual/tutorial/convert-standalone-to-replica-set/). 


## Create the Database and Collection

For this example I'm going to create a `chat` database with a `messages` collection along with an initial record that I will later use to validate connectivity to MongoDB from my client application: 

![Initial Chat Message](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/message.png)

## Install dotenv

To avoid storing MongoDB credentials in our application code, we'll use [dotenv](https://www.npmjs.com/package/dotenv) to read the MongoDB connection string from our environment. As with the `express` framework, use npm to install `dotenv`:

```zsh
npm install dotenv --save
```

Create a `.env` file with the following `MONGODB_CONNECTION_STRING` variable:

```JavaScript
MONGODB_CONNECTION_STRING='<Your MongoDB Connection String>' 
```

Then add the following to your `index.js`:

```JavaScript
require('dotenv').config()
console.log(process.env.MONGODB_CONNECTION_STRING) // remove this after you've confirmed it working
```

If you restart the process (by hitting Control+C and running `node index.js` again) you can verify that your environment is working properly:

![Server Connection to MongoDB](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/run3.png)



## Install the Node.js driver:

Use npm once again to install the Node.js driver: 

```zsh
npm install mongodb@4.5
```
## Connect to MongoDB

Add the following code to your `index.js`:

```JavaScript
const { MongoClient } = require("mongodb");

const client = new MongoClient(process.env.MONGODB_CONNECTION_STRING);  // remove this after you've confirmed it working

async function run() {

  try {

    await client.connect();
    const database = client.db('chat');
    const messages = database.collection('messages');

    // Query for our test message:
    const query = { message: 'Hello from MongoDB' };
    const message = await messages.findOne(query);
    console.log(message);

  } catch {

    // Ensures that the client will close when you error
    await client.close();
  }
}

run().catch(console.dir);
```
Restart your application and you should see the following

![MongoDB Connection Validation](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/run4.png)

For further information, this [MongoDB Node.js Quick Start](https://mongodb.prakticum-team.ru/docs/drivers/node/current/quick-start/) provides an excellent introduction to incorporating MongoDB into your Node.js applications.  

## Watching for Changes
We want to be alerted any time a new message is inserted into the database. For the purpose of this tutorial we'll also [watch](https://mongodb.prakticum-team.ru/docs/drivers/node/current/usage-examples/changeStream/) for message updates. Replace the three lines of query test code in `index.js` with the following:

```JavaScript
       // open a Change Stream on the "messages" collection
       changeStream = messages.watch();

       // set up a listener when change events are emitted
       changeStream.on("change", next => {
           // process any change event
           switch (next.operationType) {
               case 'insert':
                   console.log(next.fullDocument.message);
                   break;
               case 'update':
                   console.log(next.updateDescription.updatedFields.message);
           }
       });
```

Then edit and/or insert some messages: 

![Change Stream Success](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/run5.png)

## Integrating Socket.IO
Socket.IO is composed of two parts:



* A server that integrates with (or mounts on) the Node.JS HTTP Server [socket.io](https://github.com/socketio/socket.io)
* A client library that loads on the browser side [socket.io-client](https://github.com/socketio/socket.io-client)

During development, `socket.io` serves the client automatically for us, as we’ll see, so for now we only have to install one module:

```zsh
npm install socket.io
```

That will install the module and add the dependency to `package.json`. Now let’s edit `index.js` to add it:

```JavaScript
const { Server } = require("socket.io");
const io = new Server(server);
```

Now in `index.html` add the following snippet before the `</body>` (end body tag):

```HTML
<script src="/socket.io/socket.io.js">&lt;/script>
<script>
   var socket = io();
</script>
```

## Broadcasting

The next goal is for us to emit the event from the server to the rest of the users.

In order to send an event to everyone, Socket.IO gives us the `io.emit()` method, which looks as follows:

```JavaScript
io.emit('<event name>', '<event data>')
```

So, augment our change stream code with the following: 

```JavaScript
      switch (next.operationType) {
                case 'insert':
                    io.emit('chat message', next.fullDocument.message);
                    console.log(next.fullDocument.message);
                    break;

                case 'update':
                    io.emit('chat message', next.updateDescription.updatedFields.message);
                    console.log(next.updateDescription.updatedFields.message);
            }
```

And on the client side when we capture a 'chat message' event we’ll include it in the page. The _total_ client-side JavaScript code now amounts to:

```HTML
<!DOCTYPE html>
<html>
 <head>
   <title>Socket.IO chat</title>
   <style>
     body { margin: 0; padding-bottom: 3rem; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; }

     #messages { list-style-type: none; margin: 0; padding: 0; }
     #messages > li { padding: 0.5rem 1rem; }
     #messages > li:nth-child(odd) { background: #efefef; }
   </style>
 </head>
 <body>
   <ul id="messages"></ul>

   <script src="/socket.io/socket.io.js"></script>
   <script>
       var socket = io();
       var messages = document.getElementById('messages');

       socket.on('chat message', function(msg) {
           var item = document.createElement('li');
           item.textContent = msg;
           messages.appendChild(item);
           window.scrollTo(0, document.body.scrollHeight);
       });
   </script>
</body>
</html>
```


And that completes our chat application, in about 80 lines of code! This is what it looks like on the web client when messages are inserted or updated in our `chat.messages` collection in MongoDB:

![Final Result](https://raw.githubusercontent.com/wbleonard/mongo-socket-chat-example/main/images/result-final.png)



Interested in integrating MongoDB eventing with Socket.io? This tutorial builds in the Socket.io getting started guide to incorporate MongoDB.

Integrating MongoDB Change Streams With Socket.IO

# MongoDB Network Compression: A Win-Win

An under-advertised feature of MongoDB is its ability to compress data between the client and the server. The CRM company Close has a really [nice article](https://making.close.com/posts/mongodb-network-compression) on how compression reduced their network traffic from about 140 Mbps to 65 Mpbs. As Close notes, with cloud data transfer costs ranging from $0.01 per GB and up, you can get a nice little savings with a simple configuration change. 

![mongodb-network-compression-chart](https://raw.githubusercontent.com/wbleonard/mongodb-network-compression/main/img/mongodb-network-compression-chart.webp)

MongoDB supports the following compressors:

* [snappy](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-snappy)
* [zlib](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-zlib) (Available starting in MongoDB 3.6)
* [zstd](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-zlib) (Available starting in MongoDB 4.2)

Enabling compression from the [client](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html) simply involves installing the desired compression library and then passing the compressor as an argument when you connect to MongoDB. For example:

```PYTHON
client = MongoClient('mongodb://localhost', compressors='zstd')
```


This article provides two tuneable Python scripts, [read-from-mongo.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/read-from-mongo.py) and [write-to-mongo.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/write-to-mongo.py), that you can use to see the impact of network compression yourself. 

## Setup

### Client Configuration

Edit [params.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/params.py) and at a minimum, set your connection string. Other tunables include the amount of bytes to read and insert (default 10 MB) and the batch size to read (100 records) and insert (1 MB):

``` PYTHON
# Read to Mongo
target_read_database        = 'sample_airbnb'
target_read_collection      = 'listingsAndReviews'
megabytes_to_read           = 10
batch_size                  = 100   # Batch size in records (for reads)

# Write to Mongo
drop_collection             = True  # Drop collection on run
target_write_database       = 'test'
target_write_collection     = 'network-compression-test'
megabytes_to_insert         = 10
batch_size_mb               = 1     # Batch size of bulk insert in megabytes
```
### Compression Library
The [snappy](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-snappy) compression in Python requires the `python-snappy` package.

```pip3 install python-snappy```

The [zstd](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-zlib) compression requires the zstandard package

```pip3 install zstandard```

The [zlib](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-zlib) compression is native to Python.

### Sample Data
My  [read-from-mongo.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/read-from-mongo.py) script uses the [Sample AirBnB Listings Dataset](https://docs.atlas.mongodb.com/sample-data/sample-airbnb/#std-label-sample-airbnb) but ANY dataset will suffice for this test. 

The [write-to-mongo.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/write-to-mongo.py) script generates sample data using the Python package 
[Faker](https://faker.readthedocs.io/en/master/).

```pip3 install faker ```

## Execution
### Read from Mongo
The cloud providers notably charge for data egress, so anything that reduces network traffic out is a win. 

Let's first run the script without network compression (the default):

```ZSH
✗ python3 read-from-mongo.py

MongoDB Network Compression Test
Network Compression: Off
Now: 2021-11-03 12:24:00.904843

Collection to read from: sample_airbnb.listingsAndReviews
Bytes to read: 10 MB
Bulk read size: 100 records

1 megabytes read at 307.7 kilobytes/second
2 megabytes read at 317.6 kilobytes/second
3 megabytes read at 323.5 kilobytes/second
4 megabytes read at 318.0 kilobytes/second
5 megabytes read at 327.1 kilobytes/second
6 megabytes read at 325.3 kilobytes/second
7 megabytes read at 326.0 kilobytes/second
8 megabytes read at 324.0 kilobytes/second
9 megabytes read at 322.7 kilobytes/second
10 megabytes read at 321.0 kilobytes/second

 8600 records read in 31 seconds (276.0 records/second)

 MongoDB Server Reported Megabytes Out: 188.278 MB
 ```

_You've obviously noticed the reported Megabytes out (188 MB) are more than 18 times our test size of 10 MBs. There are several reasons for this, including other workloads running on the server, data replication to secondary nodes, and the TCP packet being larger than just the data. Focus on the delta between the other tests runs._

The script accepts an optional compression argument, that must be either `snappy`, `zlib` or `zstd`. Let's run the test again using `snappy`, which is known to be fast, while sacrificing some compression:

```ZSH
✗ python3 read-from-mongo.py -c "snappy"

MongoDB Network Compression Test
Network Compression: snappy
Now: 2021-11-03 12:24:41.602969

Collection to read from: sample_airbnb.listingsAndReviews
Bytes to read: 10 MB
Bulk read size: 100 records

1 megabytes read at 500.8 kilobytes/second
2 megabytes read at 493.8 kilobytes/second
3 megabytes read at 486.7 kilobytes/second
4 megabytes read at 480.7 kilobytes/second
5 megabytes read at 480.1 kilobytes/second
6 megabytes read at 477.6 kilobytes/second
7 megabytes read at 488.4 kilobytes/second
8 megabytes read at 482.3 kilobytes/second
9 megabytes read at 482.4 kilobytes/second
10 megabytes read at 477.6 kilobytes/second

 8600 records read in 21 seconds (410.7 records/second)

 MongoDB Server Reported Megabytes Out: 126.55 MB
```
With `snappy` compression, our reported bytes out were about `62 MBs` fewer. That's a `33%` savings. But wait, the `10 MBs` of data was read in `10` fewer seconds. That's also a `33%` performance boost!

Let's try this again using `zlib`, which can achieve better compression, but at the expense of performance.  

_[zlib](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/glossary/#std-term-zlib) compression supports an optional compression level. For this test I've set it to `9` (max compression)._

```ZSH
✗ python3 read-from-mongo.py -c "zlib"

MongoDB Network Compression Test
Network Compression: zlib
Now: 2021-11-03 12:25:07.493369

Collection to read from: sample_airbnb.listingsAndReviews
Bytes to read: 10 MB
Bulk read size: 100 records

1 megabytes read at 362.0 kilobytes/second
2 megabytes read at 373.4 kilobytes/second
3 megabytes read at 394.8 kilobytes/second
4 megabytes read at 393.3 kilobytes/second
5 megabytes read at 398.1 kilobytes/second
6 megabytes read at 397.4 kilobytes/second
7 megabytes read at 402.9 kilobytes/second
8 megabytes read at 397.7 kilobytes/second
9 megabytes read at 402.7 kilobytes/second
10 megabytes read at 401.6 kilobytes/second

 8600 records read in 25 seconds (345.4 records/second)

 MongoDB Server Reported Megabytes Out: 67.705 MB
 ```
 With `zlib` compression configured at its maximum compression level, we were able to achieve a `64%` reduction in network egress, although it took 4 seconds longer. However, that's still a `19%` performance improvement over using no compression at all.

 Let's run a final test using `zstd`, which is advertised to bring together the speed of `snappy` with the compression efficiency of `zlib`:

 ```ZSH
 ✗ python3 read-from-mongo.py -c "zstd"

MongoDB Network Compression Test
Network Compression: zstd
Now: 2021-11-03 12:25:40.075553

Collection to read from: sample_airbnb.listingsAndReviews
Bytes to read: 10 MB
Bulk read size: 100 records

1 megabytes read at 886.1 kilobytes/second
2 megabytes read at 798.1 kilobytes/second
3 megabytes read at 772.2 kilobytes/second
4 megabytes read at 735.7 kilobytes/second
5 megabytes read at 734.4 kilobytes/second
6 megabytes read at 714.8 kilobytes/second
7 megabytes read at 709.4 kilobytes/second
8 megabytes read at 698.5 kilobytes/second
9 megabytes read at 701.9 kilobytes/second
10 megabytes read at 693.9 kilobytes/second

 8600 records read in 14 seconds (596.6 records/second)

 MongoDB Server Reported Megabytes Out: 61.254 MB
 ```
And sure enough, `zstd` lives up to its reputation, achieving `68%` percent improvement in compression along with a `55%` improvement in performance!

### Write to Mongo

The cloud providers often don't charge us for data ingress. However, given the substantial performance improvements with read workloads, what can be expected from write workloads?

The [write-to-mongo.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/write-to-mongo.py) script writes a randomly generated document to the database and collection configured in [params.py](https://github.com/wbleonard/mongodb-network-compression/blob/main/params.py), the default being `test.network_compression_test`.

As before, let's run the test without compression:

```ZSH
python3 write-to-mongo.py

MongoDB Network Compression Test
Network Compression: Off
Now: 2021-11-03 12:47:03.658036

Bytes to insert: 10 MB
Bulk insert batch size: 1 MB

1 megabytes inserted at 614.3 kilobytes/second
2 megabytes inserted at 639.3 kilobytes/second
3 megabytes inserted at 652.0 kilobytes/second
4 megabytes inserted at 631.0 kilobytes/second
5 megabytes inserted at 640.4 kilobytes/second
6 megabytes inserted at 645.3 kilobytes/second
7 megabytes inserted at 649.9 kilobytes/second
8 megabytes inserted at 652.7 kilobytes/second
9 megabytes inserted at 654.9 kilobytes/second
10 megabytes inserted at 657.2 kilobytes/second

 27778 records inserted in 15.0 seconds

 MongoDB Server Reported Megabytes In: 21.647 MB
```

So it took `15` seconds to write `27,778` records. Let's run the same test with `zstd` compression:

```ZSH
✗ python3 write-to-mongo.py -c 'zstd'

MongoDB Network Compression Test
Network Compression: zstd
Now: 2021-11-03 12:48:16.485174

Bytes to insert: 10 MB
Bulk insert batch size: 1 MB

1 megabytes inserted at 599.4 kilobytes/second
2 megabytes inserted at 645.4 kilobytes/second
3 megabytes inserted at 645.8 kilobytes/second
4 megabytes inserted at 660.1 kilobytes/second
5 megabytes inserted at 669.5 kilobytes/second
6 megabytes inserted at 665.3 kilobytes/second
7 megabytes inserted at 671.0 kilobytes/second
8 megabytes inserted at 675.2 kilobytes/second
9 megabytes inserted at 675.8 kilobytes/second
10 megabytes inserted at 676.7 kilobytes/second

 27778 records inserted in 15.0 seconds

 MongoDB Server Reported Megabytes In: 8.179 MB
 ```
Our reported megabytes in are reduced by `62%`. However, our write performance remained identical. Personally, I think most of this is due to the time it takes the [Faker](https://faker.readthedocs.io/en/master/) library to generate the sample data. But having gained compression without a performance impact it is still a win.
## Measurement

There are a couple of options for measuring network traffic. This script is using the [db.serverStatus()](https://mongodb.prakticum-team.ru/proxy/docs.mongodb.com/manual/reference/method/db.serverStatus/) `physicalBytesOut` and `physicalBytesIn`, reporting on the delta between the reading at the start and end of the test run. As mentioned previously, our measurements are corrupted by other network traffic occuring on the server, but my tests have shown a consistent improvement when run. Visually, my results achieved appear as follows:

![megabytes out](https://github.com/wbleonard/mongodb-network-compression/raw/main/img/megabytes_read_graph.png)
![seconds to read](https://github.com/wbleonard/mongodb-network-compression/raw/main/img/seconds_to_read_graph.png)



Another option would be using a network analysis tool like [Wireshark](https://www.wireshark.org/). But that's beyond the scope of this article for now.

Bottom line, compression reduces network traffic by more than 60%, which is in line with the improvement seen by Close. More importantly, compression also had a dramatic improvement on read performance. That's a Win-Win.




MongoDB can compress data between the client and server. This tutorial shows how to enable network compression with a script you can run to see results.

MongoDB Network Compression: A Win-Win

In this article, learn how a scheduled trigger can be used to easily incorporate automation into your environment.

Brian Leonard

Content by Brian Leonard

Simplify Semantic Search With LangChain and MongoDB

Auto Pausing Inactive Clusters

Integrating MongoDB Change Streams With Socket.IO

MongoDB Network Compression: A Win-Win

Atlas Cluster Automation Using Scheduled Triggers