Production Notes for Self-Managed Deployments

WiredTiger

The WiredTiger storage engine is multithreaded and can take advantage of additional CPU cores. Specifically, the total number of active threads (i.e. concurrent operations) relative to the number of available CPUs can impact performance:

• Throughput increases as the number of concurrent active operations increases up to the number of CPUs.

• Throughput decreases as the number of concurrent active operations exceeds the number of CPUs by some threshold amount.

The threshold depends on your application. You can determine the optimum number of concurrent active operations for your application by experimenting and measuring throughput. The output from mongostat provides statistics on the number of active reads/writes in the (ar|aw) column.

With WiredTiger, MongoDB utilizes both the WiredTiger internal cache and the filesystem cache.

The default WiredTiger internal cache size is the larger of either:

• 50% of (RAM - 1 GB), or

• 256 MB.

For example, on a system with a total of 4GB of RAM the WiredTiger cache uses 1.5GB of RAM (0.5 * (4 GB - 1 GB) = 1.5 GB). Conversely, on a system with a total of 1.25 GB of RAM WiredTiger allocates 256 MB to the WiredTiger cache because that is more than half of the total RAM minus one gigabyte (0.5 * (1.25 GB - 1 GB) = 128 MB < 256 MB).

By default, WiredTiger uses Snappy block compression for all collections and prefix compression for all indexes. Compression defaults are configurable at a global level and can also be set on a per-collection and per-index basis during collection and index creation.

Different representations are used for data in the WiredTiger internal cache versus the on-disk format:

• Data in the filesystem cache is the same as the on-disk format, including benefits of any compression for data files. The filesystem cache is used by the operating system to reduce disk I/O.

• Indexes loaded in the WiredTiger internal cache have a different data representation to the on-disk format, but can still take advantage of index prefix compression to reduce RAM usage. Index prefix compression deduplicates common prefixes from indexed fields.

• Collection data in the WiredTiger internal cache is uncompressed and uses a different representation from the on-disk format. Block compression can provide significant on-disk storage savings, but data must be uncompressed to be manipulated by the server.

With the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes.

To adjust the size of the WiredTiger internal cache, see storage.wiredTiger.engineConfig.cacheSizeGB and --wiredTigerCacheSizeGB. Avoid increasing the WiredTiger internal cache size above its default value.

The default WiredTiger internal cache size value assumes that there is a single mongod instance per machine. If a single machine contains multiple MongoDB instances, then you should decrease the setting to accommodate the other mongod instances.

If you run mongod in a container (for example, lxc, cgroups, Docker, etc.) that does not have access to all of the RAM available in a system, you must set storage.wiredTiger.engineConfig.cacheSizeGB to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container. See memLimitMB.

To view statistics on the cache and eviction rate, see the wiredTiger.cache field returned from the serverStatus command.

Compression and Encryption

When using encryption, CPUs equipped with AES-NI instruction-set extensions show significant performance advantages. If you are using MongoDB Enterprise with the Encrypted Storage Engine, choose a CPU that supports AES-NI for better performance.

Configuring NUMA on Windows

On Windows, memory interleaving must be enabled through the machine's BIOS. Consult your system documentation for details.

Configuring NUMA on Linux

On Linux, you must disable zone reclaim and also ensure that your mongod and mongos instances are started by numactl, which is generally configured through your platform's init system. You must perform both of these operations to properly disable NUMA for use with MongoDB.

1. Disable zone reclaim with one of the following commands:

   echo 0 | sudo tee /proc/sys/vm/zone_reclaim_mode

   sudo sysctl -w vm.zone_reclaim_mode=0

2. Ensure that mongod and mongos are started by numactl. This is generally configured through your platform's init system. Run the following command to determine which init system is in use on your platform:

   ps --no-headers -o comm 1

   • If "systemd", your platform uses the systemd init system, and you must follow the steps in the systemd tab below to edit your MongoDB service file(s).

   • If "init", your platform uses the SysV Init system, and you do not need to perform this step. The default MongoDB init script for SysV Init includes the necessary steps to start MongoDB instances via numactl by default.

   • If you manage your own init scripts (i.e. you are not using either of these init systems), you must follow the steps in the Custom init scripts tab below to edit your custom init script(s).

     
     
   systemd

   Custom init scripts

   You must use numactl to start each of your mongod instances, including all config servers, mongos instances, and clients. Edit the default systemd service file for each as follows:

   1. Copy the default MongoDB service file:

      sudo cp /lib/systemd/system/mongod.service /etc/systemd/system/

   2. Edit the /etc/systemd/system/mongod.service file, and update the ExecStart statement to begin with:

      /usr/bin/numactl --interleave=all

      Example

      If your existing ExecStart statement reads:

      ExecStart=/usr/bin/mongod --config /etc/mongod.conf

      Update that statement to read:

      ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongod.conf

   3. Apply the change to systemd:

      sudo systemctl daemon-reload

   4. Restart any running mongod instances:

      sudo systemctl stop mongod
      sudo systemctl start mongod

   5. If applicable, repeat these steps for any mongos instances.

   You must use numactl to start each of your mongod instances, including all config servers, mongos instances, and clients.

   1. Install numactl for your platform if not already installed. Refer to the documentation for your operating system for information on installing the numactl package.

   2. Configure each of your custom init scripts to start each MongoDB instance via numactl:

      numactl --interleave=all <path> <options>

      Where <path> is the path to the program you are starting and <options> are any optional arguments to pass to that program.

      Example

      numactl --interleave=all /usr/local/bin/mongod -f /etc/mongod.conf

For more information, see the Documentation for /proc/sys/vm/*.

Swap

MongoDB performs best where swapping can be avoided or kept to a minimum, as retrieving data from swap will always be slower than accessing data in RAM. However, if the system hosting MongoDB runs out of RAM, swapping can prevent the Linux OOM Killer from terminating the mongod process.

Generally, you should choose one of the following swap strategies:

1. Assign swap space on your system, and configure the kernel to only permit swapping under high memory load, or

2. Do not assign swap space on your system, and configure the kernel to disable swapping entirely

See Set vm.swappiness for instructions on configuring swap on your Linux system following these guidelines.

RAID

For optimal performance in terms of the storage layer, use disks backed by RAID-10. RAID-5 and RAID-6 do not typically provide sufficient performance to support a MongoDB deployment.

Remote Filesystems (NFS)

With the WiredTiger storage engine, WiredTiger objects may be stored on remote file systems if the remote file system conforms to ISO/IEC 9945-1:1996 (POSIX.1). Because remote file systems are often slower than local file systems, using a remote file system for storage may degrade performance.

If you decide to use NFS, add the following NFS options to your /etc/fstab file:

• bg

• hard

• nolock

• noatime

• nointr

Depending on your kernel version, some of these values may already be set as the default. Consult your platform's documentation for more information.

Separate Components onto Different Storage Devices

For improved performance, consider separating your database's data, journal, and logs onto different storage devices, based on your application's access and write pattern. Mount the components as separate filesystems and use symbolic links to map each component's path to the device storing it.

For the WiredTiger storage engine, you can also store the indexes on a different storage device. See storage.wiredTiger.engineConfig.directoryForIndexes.

Scheduling

Kernel and File Systems

When running MongoDB in production on Linux, you should use Linux kernel version 2.6.36 or later, with either the XFS or EXT4 filesystem. If possible, use XFS as it generally performs better with MongoDB.

With the WiredTiger storage engine, using XFS is strongly recommended for data bearing nodes to avoid performance issues that may occur when using EXT4 with WiredTiger.

• In general, if you use the XFS file system, use at least version 2.6.25 of the Linux Kernel.

• If you use the EXT4 file system, use at least version 2.6.28 of the Linux Kernel.

• On Red Hat Enterprise Linux and CentOS, use at least version 2.6.18-194 of the Linux kernel.

System C Library

MongoDB uses the GNU C Library (glibc) on Linux. Generally, each Linux distro provides its own vetted version of this library. For best results, use the latest update available for this system-provided version. You can check whether you have the latest version installed by using your system's package manager. For example:

• On RHEL / CentOS, the following command updates the system-provided GNU C Library:

  sudo yum update glibc

• On Ubuntu / Debian, the following command updates the system-provided GNU C Library:

  sudo apt-get install libc6

fsync() on Directories

Set vm.swappiness to 1 or 0

“Swappiness” is a Linux kernel setting that influences the behavior of the Virtual Memory manager. The vm.swappiness setting ranges from 0 to 100: the higher the value, the more strongly it prefers swapping memory pages to disk over dropping pages from RAM.

• A setting of 0 disables swapping entirely [4].

• A setting of 1 permits the kernel to swap only to avoid out-of-memory problems.

• A setting of 60 tells the kernel to swap to disk often, and is the default value on many Linux distributions.

• A setting of 100 tells the kernel to swap aggressively to disk.

MongoDB performs best where swapping can be avoided or kept to a minimum. As such you should set vm.swappiness to either 1 or 0 depending on your application needs and cluster configuration.

• To check the current swappiness setting on your system, run:

  cat /proc/sys/vm/swappiness

• To change swappiness on your system:

  1. Edit the /etc/sysctl.conf file and add the following line:

     vm.swappiness = 1

  2. Run the following command to apply the setting:

     sudo sysctl -p

Recommended Configuration

For all MongoDB deployments:

• Use the Network Time Protocol (NTP) to synchronize time among your hosts. This is especially important in sharded clusters.

For the WiredTiger storage engines, consider the following recommendations:

• Turn off atime for the storage volume containing the database files.

• Adjust the ulimit settings for your platform according to the recommendations in the ulimit reference. Low ulimit values will negatively affect MongoDB when under heavy use and can lead to failed connections to MongoDB processes and loss of service.

  Note

  If the ulimit value for number of open files is under 64000, MongoDB generates a startup warning.

• Disable Transparent Huge Pages. MongoDB performs better with normal (4096 bytes) virtual memory pages. See Transparent Huge Pages Settings.

• Disable NUMA in your BIOS. If that is not possible, see MongoDB on NUMA Hardware.

• Configure SELinux for MongoDB if you are not using the default MongoDB directory paths or ports.

  See: Configure SELinux for MongoDB and Configure SELinux for MongoDB Enterprise for the required configuration.

  Note

  If you are using SELinux, any MongoDB operation that requires server-side JavaScript will result in segfault errors. Disable Server-Side Execution of JavaScript describes how to disable execution of server-side JavaScript.

For the WiredTiger storage engine:

• Set the readahead setting between 8 and 32 regardless of storage media type (spinning disk, SSD, etc.).

  Higher readahead commonly benefits sequential I/O operations. Since MongoDB disk access patterns are generally random, using higher readahead settings provides limited benefit or potential performance degradation. As such, for optimal MongoDB performance, set readahead between 8 and 32, unless testing shows a measurable, repeatable, and reliable benefit in a higher readahead value. MongoDB commercial support can provide advice and guidance on alternate readahead configurations.

MongoDB and TLS/SSL Libraries

On Linux platforms, you may observe one of the following statements in the MongoDB log:

<path to TLS/SSL libs>/libssl.so.<version>: no version information available (required by /usr/bin/mongod)
<path to TLS/SSL libs>/libcrypto.so.<version>: no version information available (required by /usr/bin/mongod)

These warnings indicate that the system's TLS/SSL libraries are different from the TLS/SSL libraries that the mongod was compiled against. Typically these messages do not require intervention; however, you can use the following operations to determine the symbol versions that mongod expects:

objdump -T <path to mongod>/mongod | grep " SSL_"
objdump -T <path to mongod>/mongod | grep " CRYPTO_"

These operations will return output that resembles one the of the following lines:

0000000000000000      DF *UND*       0000000000000000  libssl.so.10 SSL_write
0000000000000000      DF *UND*       0000000000000000  OPENSSL_1.0.0 SSL_write

The last two strings in this output are the symbol version and symbol name. Compare these values with the values returned by the following operations to detect symbol version mismatches:

objdump -T <path to TLS/SSL libs>/libssl.so.1*
objdump -T <path to TLS/SSL libs>/libcrypto.so.1*

This procedure is neither exact nor exhaustive: many symbols used by mongod from the libcrypto library do not begin with CRYPTO_.

AWS EC2

There are two performance configurations to consider:

• Reproducible performance for performance testing or benchmarking, and

• Raw maximum performance

To tune performance on EC2 for either configuration, you should:

• Enable AWS Enhanced Networking for your instance. Not all instance types support Enhanced Networking.

  To learn more about Enhanced Networking, see to the AWS documentation.

If you are concerned more about reproducible performance on EC2, you should also:

• Use provisioned IOPS for the storage, with separate devices for journal and data. Do not use the ephemeral (SSD) storage available on most instance types as their performance changes moment to moment. (The i series is a notable exception, but very expensive.)

• Disable DVFS and CPU power saving modes.

  Tip

  See also:

  Amazon documentation on Processor State Control

• Disable hyperthreading.

  Tip

  See also:

  Amazon blog post on disabling Hyper-Threading.

• Use numactl to bind memory locality to a single socket.

Azure

Use Premium Storage. Microsoft Azure offers two general types of storage: Standard storage, and Premium storage. MongoDB on Azure has better performance when using Premium storage than it does with Standard storage.

The TCP idle timeout on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem.

• To view the keepalive setting on Linux, use one of the following commands:

  sysctl net.ipv4.tcp_keepalive_time

  Or:

  cat /proc/sys/net/ipv4/tcp_keepalive_time

  The value is measured in seconds.

  Note

  Although the setting name includes ipv4, the tcp_keepalive_time value applies to both IPv4 and IPv6.

• To change the tcp_keepalive_time value, you can use one of the following commands, supplying a <value> in seconds:

  sudo sysctl -w net.ipv4.tcp_keepalive_time=<value>

  Or:

  echo <value> | sudo tee /proc/sys/net/ipv4/tcp_keepalive_time

  These operations do not persist across system reboots. To persist the setting, add the following line to /etc/sysctl.conf, supplying a <value> in seconds, and reboot the machine:

  net.ipv4.tcp_keepalive_time = <value>

  Keepalive values greater than 300 seconds, (5 minutes) will be overridden on mongod and mongos sockets and set to 300 seconds.

• To view the keepalive setting on Windows, issue the following command:

  reg query HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v KeepAliveTime

  The registry value is not present by default. The system default, used if the value is absent, is 7200000 milliseconds or 0x6ddd00 in hexadecimal.

  
  

• To change the KeepAliveTime value, use the following command in an Administrator Command Prompt, where <value> is expressed in hexadecimal (e.g. 120000 is 0x1d4c0):

  reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ /t REG_DWORD /v KeepAliveTime /d <value>

  Windows users should consider the Windows Server Technet Article on KeepAliveTime for more information on setting keepalive for MongoDB deployments on Windows systems. Keepalive values greater than or equal to 600000 milliseconds (10 minutes) will be ignored by mongod and mongos.

VMware

MongoDB is compatible with VMware.

VMware supports memory overcommitment, where you can assign more memory to your virtual machines than the physical machine has available. When memory is overcommitted, the hypervisor reallocates memory between the virtual machines. VMware's balloon driver (vmmemctl) reclaims the pages that are considered least valuable.

The balloon driver resides inside the guest operating system. Under certain configurations, when the balloon driver expands, it can interfere with MongoDB's memory management and affect MongoDB's performance.

To prevent negative performance impact from the balloon driver and memory overcommitment features, reserve the full amount of memory for the virtual machine running MongoDB. Reserving the appropriate amount of memory for the virtual machine prevents the balloon from inflating in the local operating system when there is memory pressure in the hypervisor.

Even though the balloon driver and memory overcommitment features can negatively affect MongoDB performance under certain configurations, do not disable these features. If you disable these features, the hypervisor may use its swap space to fulfill memory requests, which negatively affects performance.

Ensure that virtual machines stay on a specific ESX/ESXi host by setting VMware's affinity rules. If you must manually migrate a virtual machine to another host and the mongod instance on the virtual machine is the primary, you must first step down the primary and then shut down the instance.

Follow the networking best practices for vMotion and VMKernel. Failure to follow the best practices can result in performance problems and affect replica set and sharded cluster high availability mechanisms.

You can clone a virtual machine running MongoDB. You might use this function to deploy a new virtual host to add as a member of a replica set.

KVM

MongoDB is compatible with KVM.

KVM supports memory overcommitment, where you can assign more memory to your virtual machines than the physical machine has available. When memory is overcommitted, the hypervisor reallocates memory between the virtual machines. KVM's balloon driver reclaims the pages that are considered least valuable.

The balloon driver resides inside the guest operating system. Under certain configurations, when the balloon driver expands, it can interfere with MongoDB's memory management and affect MongoDB's performance.

To prevent negative performance impact from the balloon driver and memory overcommitment features, reserve the full amount of memory for the virtual machine running MongoDB. Reserving the appropriate amount of memory for the virtual machine prevents the balloon from inflating in the local operating system when there is memory pressure in the hypervisor.

Even though the balloon driver and memory overcommitment features can negatively affect MongoDB performance under certain configurations, do not disable these features. If you disable these features, the hypervisor may use its swap space to fulfill memory requests, which negatively affects performance.