Community

Srinivas Mutyala

Database Architect

MongoDB is celebrated for its high performance and scalability, making it a popular choice among NoSQL databases. However, to fully leverage its potential, fine-tuning your MongoDB deployment is essential. This guide outlines various strategies and best practices for enhancing MongoDB performance, covering everything from identifying bottlenecks to optimizing queries and hardware.

# Understanding your workload
Before diving into performance tuning, it's crucial to understand your workload. MongoDB's performance can vary significantly based on whether your application is read-heavy, write-heavy, or a balanced mix. Utilize tools like MongoDB's [Atlas Profiler](https://mongodb.prakticum-team.ru/docs/atlas/tutorial/profile-database/) or the open-source ```mongostat``` to analyze your database operations and gain insights into your workload.

# Indexing for performance
Effective indexing is one of the most impactful ways to enhance query performance in MongoDB. Here are key practices:

 - **Create relevant indexes**:  Tailor indexes to match your application's query patterns. Use the [explain() method to understand query behavior and optimize accordingly.

```db.collection.find({ field: value }).explain("executionStats")```

You can also get this information from MongoDB Compass with sophisticated output as shown below.

![MongoDB Compass interface showing the 'addrInfo' collection within the 'IndexDB' database. The query { state: 'st1' } is entered into the query bar. The interface displays multiple documents with fields such as _id, state, city, pin, landmark, and lastModifiedOn. Buttons for 'Add Data', 'Export Data', 'Update', and 'Delete' are visible below the query bar, along with options for explaining the query, resetting, and finding documents. The left sidebar lists other collections and databases.](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltda048ed558850827/668d521daba9d75558c00b58/Screenshot_2024-07-09_at_16.05.54.png)

![MongoDB Compass Explain Plan interface showing a visual tree of the query execution plan. The diagram includes two steps: FETCH and IXSCAN. The IXSCAN step indicates an index scan on the 'state_1_city_1' index, with 1 document returned and examined, and no multikey index. The FETCH step shows the same document returned and execution time as 0 ms. The Query Performance Summary on the right highlights 1 document returned, 1 document examined, 0 ms execution time, not sorted in memory, and 1 index key examined](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3303e642c01a8a8b/668d534eaba9d7dd5fc00b83/Screenshot_2024-07-09_at_16.11.44.png)

- **Avoid over-indexing**: While indexes improve query speed, they can hinder write operations and consume additional disk space. Regularly review and remove unused or unnecessary indexes.

```db.collection.dropIndex("indexName")```

- **Use compound indexes**: For queries involving multiple fields, compound indexes can significantly boost performance.

```db.collection.createIndex({ field1: 1, field2: -1 })```

# Optimising query patterns
Optimizing your query patterns is crucial for reducing execution time and resource usage:

 - **Projection**: Use projection to limit the fields returned by your queries, minimizing data transfer and processing load. Also, it’s better to exclude _id with 0 (false) if it’s not a field pertaining to the application — i.e., an auto-generated field by MongoDB.
```db.collection.find({ field: value }, { field1: 1, field2: 1 })```
 -  **Aggregation framework**: Leverage MongoDB's [aggregation framework](https://mongodb.prakticum-team.ru/developer/products/mongodb/introduction-aggregation-framework/) for complex data processing. Ensure aggregations utilize indexed fields where possible.

```db.collection.aggregate([ { $match: { field: value } }, { $group: { _id: "$field", total: { $sum: "$amount" } } } ])```

 - **Avoid $where**: The ```$where``` operator can be slow and resource-intensive. Use it sparingly and only when necessary. Instead, the use of $expr with aggregation operators that do not use JavaScript (i.e., non-$function and non-$accumulator operators) is faster than $where because it does not execute JavaScript and is preferable, when possible.
However, if you must create custom expressions, $function is preferred over $where.

# Hardware considerations
The hardware on which MongoDB runs plays a crucial role in its performance:

 - **RAM**: MongoDB relies heavily on RAM to store working sets. If your dataset exceeds your available RAM, consider upgrading your memory.
 - **Storage**: Utilize SSDs for storage to enhance I/O throughput and data access speeds.
- **Network**: Ensure your network bandwidth and latency are sufficient, especially in distributed deployments.

# Replication and sharding

Replication and sharding
MongoDB supports replication and sharding to improve availability and scalability:

 - **Replication**: This ensures data redundancy and high availability. Configure read preference settings to effectively route read operations across replicas.
```rs.initiate()```

Following are the available read methods with MongoDB which you can configure at the application level.

 - **primary**: Reads from the primary only
 - **primaryPreferred**: Reads from the primary if available, otherwise from a secondary
 - **secondary**: Reads from a secondary only
 - **secondaryPreferred**: Reads from a secondary if available, otherwise from the primary
 - **nearest**: Reads from the nearest node based on network latency and operational health

**Example: Setting read preferences in application code (Node.js)**
```javascript
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb://host1,host2,host3/?readPreference=secondaryPreferred";
MongoClient.connect(uri, function(err, client) {
  const db = client.db('test');
  // Perform operations 
});
```

- **Sharding**:  This distributes data across multiple servers and is crucial for managing large datasets and high throughput operations. Choose a shard key that evenly distributes data and query load.
```sh.enableSharding("mydatabase") sh.shardCollection("mydatabase.mycollection", { shardKey: 1 })```

Choosing a shard key in MongoDB can significantly impact performance depending on whether your workload is read-heavy or write-heavy. Here are some guidelines for selecting a shard key based on your workload:

## **Read-heavy workloads** ##
Shard key selection: Choose a shard key that evenly distributes read operations across shards.

 **Considerations**: 
Use a high-cardinality field that ensures even distribution of reads.
Avoid shard keys that can cause hot spots where most reads target a single shard.

**Example**: Use a user ID if user-related queries are common.

```sh.shardCollection("mydatabase.mycollection", { userID: 1 })```

## **Write-heavy workloads** ##
Shard key selection: Choose a shard key that balances the write load across shards.

 **Considerations**: 
Use a field that changes frequently and ensures even write distribution.
Avoid monotonically increasing keys (e.g., timestamps) as they can lead to a single shard being a bottleneck.

**Example**: Use a hashed shard key to distribute writes evenly if you can not get a unique shard key.

```sh.shardCollection("mydatabase.mycollection", { hashedField: "hashed" })```

**Additional considerations**: 
Monitor and adjust: Continuously monitor the performance and adjust shard keys if needed.

**Indexing**: Ensure indexes are aligned with the shard key for optimal query performance.
By selecting the appropriate shard key and considering the nature of your workload, you can optimize your MongoDB deployment for both read and write operations.

# Performance monitoring and maintenance
Regular monitoring and maintenance are vital for sustained performance:

- **Monitoring tools**: Utilize MongoDB Atlas, mongostat, and mongotop to monitor database performance and resource usage.

``` mongostat --host <host> mongotop --host <host> ```

- **Routine maintenance**:  Regularly compact collections, repair databases, and rebalance shards to ensure optimal performance.
```db.repairDatabase()```

# Read/write concerns
The choice of write concern can influence both the performance and the durability of the data.

## Performance
A lower write concern (e.g., w: 0) can enhance performance by reducing the latency of the write operation. However, it risks data durability.

**Impact on latency**

Lower write concern (e.g., w: 0):

**Latency reduction**:
- The client does not wait for any acknowledgment from the server.
- The operation is sent to the server and considered complete from the client's perspective.
- There is no network round-trip latency as there is no need for the server to respond.

**Trade-off**:
- There's an increased risk of data loss since the client receives no confirmation of write success.
- It's suitable for non-critical data or scenarios where high write throughput is needed with minimal latency.

Higher write concern (e.g., w: 1 or w: "majority"):

**Latency increase**
- The client waits for acknowledgement from the server.
- For w: 1, waits for acknowledgment from the primary node.
- For w: "majority", waits for acknowledgment from the majority of replica set members.
- Network round-trip latency and server processing time add to the overall latency.
- Enhanced data durability and consistency.
- Ensures the write operation is replicated and acknowledged.

```db.collection.insertOne({ field: "value" }, { writeConcern: { w: 1 } })```

# Read preferences

The choice of read preference can influence both the performance and the availability of the data.

**Performance**:  Distributing read operations to secondary members can enhance performance by reducing the load on the primary.
To successfully distribute read operations to secondary members and thereby enhance performance, you need to set the read preference in MongoDB. Here are examples of how to configure read preferences:

## MongoDB Shell

```db.getMongo().setReadPref("secondaryPreferred")```

**Connection URI**

```mongodb://host1,host2,host3/?readPreference=secondaryPreferred```

** Application code example (NodeJS) **

```javascript
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb://host1,host2,host3/?readPreference=secondaryPreferred";
MongoClient.connect(uri, function(err, client) {
  const db = client.db('test');
  // Perform operations
})
```

By setting the read preference to secondaryPreferred, you direct read operations to secondary members when they are available, reducing the load on the primary node and enhancing overall performance.

**Checks to identify the common reasons for performance issues**:

 - Run mongotop and mongostat, and check which namespace is causing the issue.
- System level - check for primary replication. Is there any lag, and how is the opLog window?
- Application level — check for any batch loads at the application level.
- Any slow queries (with currentOp())?
- Are there proper indexes?
- Sharded cluster — are the majority of the queries using the shard key? 
- WT cache? Any evicts?
- Do you see write contention?
- Open files ( ulimit -a ) - 65000
- Check whether the mongod process alone causes server load or any other processes. 

- *top* or *htop*: Monitor CPU and memory usage of *mongod*  and other processes.
- *ps* and *grep*: Run ```ps aux | grep mongod``` to view *mongod* resource usage.
- *iostat*: Use ```iostat -x 1 10``` to check disk I/O metrics.
- *vmstat*: Run ```vmstat 1 10``` for overall system performance snapshots.

![write contention in MongoDB. The left side shows a single update operation (db.col.updateOne()) resulting in a new document version (v1 to v2) using MVCC (Multi-Version Concurrency Control). The right side depicts multiple conflicting document updates (v2-1, v2-2, v2-3) with crosses indicating failed operations due to write contention. Arrows and pointers show the process of updates and conflicts. The diagram emphasizes avoiding write contention by revising the schema design and highlights the use of the WT (WiredTiger) storage engine's optimistic concurrency protocol.]( https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt592c2d7725933aee/668d642a032545c51888574b/Screenshot_2024-07-09_at_17.24.00.png)

Write contention in MongoDB can be identified by the following indicators:

**High locking percentages**: Use mongostat to monitor lock percentages. High values indicate contention.
**Slow write operations**: Check for slow write operations using db.currentOp() which may indicate contention.
**Frequent write conflicts**:  Review logs for messages about write conflicts or rejections.

**Increased latency**: Observe increased latency in write-heavy operations or applications.

**Example command to monitor lock percentages**: ```mongostat --host <hostname>```

Designing the schema properly, such as using appropriate indexes and avoiding hotspots with distributed writes, can help mitigate write contention.

# Conclusion
Achieving optimal MongoDB performance involves a comprehensive approach, including query optimization, proper indexing, sufficient hardware resources, and continuous monitoring. By implementing the strategies outlined in this guide, you can significantly enhance the efficiency and responsiveness of your MongoDB deployment, ensuring it meets the demands of your applications.

Questions? Comments? Head to the [MongoDB Developer Community](https://mongodb.prakticum-team.ru/community/forums/) next.


Read this guide to learn various strategies and best practices for enhancing MongoDB performance

Comprehensive Guide to Optimising MongoDB Performance

Someone somewhere might be wondering why we get superior performance with MongoDB over RDBMS databases. What is the secret behind it? I too had this question until I learned about the internal workings of MongoDB, especially data modeling, advanced index methods, and finally, how the WiredTiger storage engine works.

I wanted to share my learnings and experiences to reveal the secret of it so that it might be helpful to you, too.

## Data modeling: embedded structure (no JOINs)

MongoDB uses a document-oriented data model, storing data in JSON-like BSON documents. This allows for efficient storage and retrieval of complex data structures. 

MongoDB's model can lead to simpler and more performant queries compared to the normalization requirements of RDBMS.

![Relational database tables with an arrow pointing to a document model](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0dd37875492fd34d/65cbc21cbf8ac4a06fc5be6d/tables-to-documents.png)

The initial phase of enhancing performance involves comprehending the query behaviors of your application. This understanding enables you to tailor your data model and choose suitable indexes to align with these patterns effectively.

Always remember MongoDB's optimized document size (which is 16 MB) so you can avoid embedding images, audio, and video files in the same collection, as depicted in the image below.  

![Illustration saying that embedding is better than referencing](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt785c0c6dc266b59a/65cbc9cc3fe0594863b661d9/Screenshot_2024-02-13_194011.png)

Customizing your data model to match the query patterns of your application leads to streamlined queries, heightened throughput for insert and update operations, and better workload distribution across a sharded cluster.

While MongoDB offers a flexible schema, overlooking schema design is not advisable. Although you can adjust your schema as needed, adhering to schema design best practices from the outset of your project can prevent the need for extensive refactoring down the line.

A major advantage of BSON documents is that you have the flexibility to model your data any way your application needs. The inclusion of arrays and subdocuments within documents provides significant versatility in modeling intricate data relationships. But you can also model flat, tabular, and columnar structures, simple key-value pairs, text, geospatial and time-series data, or the nodes and edges of connected graph data structures. The ideal schema design for your application will depend on its specific query patterns.

### How is embedding within collections in MongoDB different from storing in multiple tables in RDBMS?

An example of a best practice for an address/contact book involves separating groups and portraits information in a different collection because as they can go big due to n-n relations and image size, respectively. They may hit a 16 MB optimized document size. 

![Splitting groups and portraits into different collections and using referencing](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9031bfe41e9e043e/65cbdd9a31489681e4d72297/Screenshot_2024-02-13_202203.png)

Embedding data in a single collection in MongoDB (or minimizing the number of collections, at least) versus storing in multiple tables in RDBMS offers huge performance improvements due to the data locality which will reduce the data seeks, as shown in the picture below. 

![Diagram representing how data locality plays a part in faster data reads](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt67ff52152c70f879/65cbdfb3fce01c26d9fc83ad/Screenshot_2024-02-13_213018.png)

Data locality is the major reason why MongoDB data seeks are faster. 

**Difference: tabular vs document** 
|                             | Tabular                       | MongoDB         |
| --------------------------- | ----------------------------- | --------------- |
| Steps to create the model   | 1 - define schema.  2 - develop app and queries |  1 - identifying the queries  2- define schema            |
| Initial schema              |    3rd normal form. One possible solution   | Many possible solutions   |
| Final schema                | Likely denormalized       |  Few changes |
| Schema evolution            | Difficult and not optimal. Likely downtime         | Easy.    No downtime             |
| Performance                 | Mediocre                  | Optimized   |

## WiredTiger’s cache and compression
WiredTiger is an open-source, high-performance storage engine for MongoDB. WiredTiger provides features such as document-level concurrency control, compression, and support for both in-memory and on-disk storage.

**Cache:**

WiredTiger cache architecture: WiredTiger utilizes a sophisticated caching mechanism to efficiently manage data in memory. The cache is used to store frequently accessed data, reducing the need to read from disk and improving overall performance.

Memory management: The cache dynamically manages memory usage based on the workload. It employs techniques such as eviction (removing less frequently used data from the cache) and promotion (moving frequently used data to the cache) to optimize memory utilization.

Configuration: WiredTiger allows users to configure the size of the cache based on their system's available memory and workload characteristics. Properly sizing the cache is crucial for achieving optimal performance.

Durability: WiredTiger ensures durability by flushing modified data from the cache to disk. This process helps maintain data consistency in case of a system failure.

**Compression**:

Data compression: WiredTiger supports data compression to reduce the amount of storage space required. Compressing data can lead to significant disk space savings and improved I/O performance.

Configurable compression: Users can configure compression options based on their requirements. WiredTiger supports different compression algorithms, allowing users to choose the one that best suits their workload and performance goals.

Trade-offs: While compression reduces storage costs and can improve read/write performance, it may introduce additional CPU overhead during compression and decompression processes. Users need to carefully consider the trade-offs and select compression settings that align with their application's needs.

Compatibility: WiredTiger's compression features are transparent to applications and don't require any changes to the application code. The engine handles compression and decompression internally.

Overall, WiredTiger's cache and compression features contribute to its efficiency and performance characteristics. By optimizing memory usage and providing configurable compression options, WiredTiger aims to meet the diverse needs of MongoDB users in terms of both speed and storage efficiency.

Few RDBMS systems also employ caching, but the performance benefits may vary based on the database system and configuration. 

### Advanced indexing capabilities 

MongoDB, being a NoSQL database, offers advanced indexing capabilities to optimize query performance and support efficient data retrieval. Here are some of MongoDB's advanced indexing features:

**Compound indexes**

MongoDB allows you to create compound indexes on multiple fields. A compound index is an index on multiple fields in a specific order. This can be useful for queries that involve multiple criteria.

The order of fields in a compound index is crucial. MongoDB can use the index efficiently for queries that match the index fields from left to right.

**Multikey indexes**

MongoDB supports indexing on arrays. When you index an array field, MongoDB creates separate index entries for each element of the array.

Multikey indexes are helpful when working with documents that contain arrays, and you need to query based on elements within those arrays.

**Text indexes**

MongoDB provides text indexes to support full-text search. Text indexes tokenize and stem words, allowing for more flexible and language-aware text searches.

Text indexes are suitable for scenarios where users need to perform text search operations on large amounts of textual data.

**Geospatial indexes**

MongoDB supports geospatial indexes to optimize queries that involve geospatial data. These indexes can efficiently handle queries related to location-based information.

Geospatial indexes support 2D and 3D indexing, allowing for the representation of both flat and spherical geometries.

**Wildcard indexes**

MongoDB supports wildcard indexes, enabling you to create indexes that cover only a subset of fields in a document. This can be useful when you have specific query patterns and want to optimize for those patterns without indexing every field.

**Partial indexes**

Partial indexes allow you to index only the documents that satisfy a specified filter expression. This can be beneficial when you have a large collection but want to create an index for a subset of documents that meet specific criteria.

**Hashed indexes**

Hashed indexes are useful for sharding scenarios. MongoDB automatically hashes the indexed field's values and distributes the data across the shards, providing a more even distribution of data and queries.

**TTL (time-to-live) indexes**

TTL indexes allow you to automatically expire documents from a collection after a certain amount of time. This is helpful for managing data that has a natural expiration, such as session information or log entries.

These advanced indexing capabilities in MongoDB provide developers with powerful tools to optimize query performance for a wide range of scenarios and data structures. Properly leveraging these features can significantly enhance the efficiency and responsiveness of MongoDB databases.

![Visualisation of index capabilities](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte1e79d07eab8ae74/65cc8efcfce01c2084fc85aa/Screenshot_2024-02-14_095532.png)

In conclusion, the superior performance of MongoDB over traditional RDBMS databases stems from its adept handling of data modeling, advanced indexing methods, and the efficiency of the WiredTiger storage engine. By tailoring your data model to match application query patterns, leveraging MongoDB's optimized document structure, and harnessing advanced indexing capabilities, you can achieve enhanced throughput and more effective workload distribution.

Remember, while MongoDB offers flexibility in schema design, it's crucial not to overlook the importance of schema design best practices from the outset of your project. This proactive approach can save you from potential refactoring efforts down the line.

For further exploration and discussion on MongoDB and database optimization strategies, consider joining our [Developer Community](https://mongodb.prakticum-team.ru/community/forums/). There, you can engage with fellow developers, share insights, and stay updated on the latest developments in database technology.

Keep optimizing and innovating with MongoDB to unlock the full potential of your applications. 


Guest Author Srinivas Mutyala discusses the reasons for MongoDB's improved performance over traditional RDMBS.

Srinivas Mutyala

Content by Srinivas Mutyala

Comprehensive Guide to Optimising MongoDB Performance

MongoDB's Performance over RDBMS