Mogno slow - very high memory consuption by Mongo

Tamar_Nirenberg · 2021-10-11T06:36:16.443Z

I have a 5 note replicaset mongoDB - 1 primary, 3 secondaries and 1 arbiter.

My Primary is very slow - each command from the shell takes a long time to return.

Memory usage seems very high, and looks llike mongodb is consuming more than 50% of the RAM:

# free -lh
                     total        used          free         shared  buff/cache   available
Mem:          188Gi       187Gi       473Mi        56Mi       740Mi       868Mi
Low:          188Gi       188Gi       473Mi
High:            0B          0B          0B
Swap:         191Gi       117Gi        74Gi


------------------------------------------------------------------
Top Memory Consuming Process Using ps command
------------------------------------------------------------------
  PID  PPID %MEM %CPU CMD
  311 49145 97.8  498 mongod --config /etc/mongod.conf
23818 23801  0.0  3.8 /bin/prometheus --config.file=/etc/prometheus/prometheus.yml
23162 23145  0.0  8.4 /usr/bin/cadvisor -logtostderr
25796 25793  0.0  0.4 postgres: checkpointer
23501 23484  0.0  1.0 /postgres_exporter
24490 24473  0.0  0.1 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docker cfg:default.log.mode=console

Top:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  311 systemd+  20   0  313.9g 184.6g   2432 S 151.7  97.9  26229:09 mongod
23818 nobody    20   0   11.3g 150084  17988 S  20.7   0.1   8523:47 prometheus
23162 root      20   0   12.7g  93948   5964 S  65.5   0.0  18702:22 cadvisor

serverStatus memeory shows this:

octopusrs0:PRIMARY> db.serverStatus().mem
{
        "bits" : 64,
        "resident" : 189097,
        "virtual" : 321404,
        "supported" : true
}
octopusrs0:PRIMARY> db.serverStatus().tcmalloc.tcmalloc.formattedString
------------------------------------------------
MALLOC:   218206510816 (208097.9 MiB) Bytes in use by application
MALLOC: +  96926863360 (92436.7 MiB) Bytes in page heap freelist
MALLOC: +   3944588576 ( 3761.9 MiB) Bytes in central cache freelist
MALLOC: +       134144 (    0.1 MiB) Bytes in transfer cache freelist
MALLOC: +    713330688 (  680.3 MiB) Bytes in thread cache freelists
MALLOC: +   1200750592 ( 1145.1 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: = 320992178176 (306122.0 MiB) Actual memory used (physical + swap)
MALLOC: +  13979086848 (13331.5 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: = 334971265024 (319453.5 MiB) Virtual address space used
MALLOC:
MALLOC:        9420092              Spans in use
MALLOC:            234              Thread heaps in use
MALLOC:           4096              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

Actual memory used (306122.0 MiB) seems higher than server’s RAM (188Gi).
How does that makes sense, and how can I detrmine what is causing this high memory consumption?

Thanks,
Tamar

Tamar_Nirenberg · 2021-10-11T07:48:59.115Z

forgot te mention that I am using mong version 4.2.3

Tamar_Nirenberg · 2021-10-11T07:59:21.822Z

Sizes:

“dataSize” : 688.4161271536723,
“indexes” : 177,
“indexSize” : 108.41889953613281,

kevinadi · 2021-10-13T05:56:27.556Z

Hi @Tamar_Nirenberg

Actual memory used (306122.0 MiB) seems higher than server’s RAM (188Gi).

I think it’s because it includes actual RAM + swap. It’s visible in free output that the system is using swap here. Up to 74GB of swap usage, it seems:

Swap: 191Gi 117Gi 74Gi

Typically swap usage means that the server doesn’t have enough RAM to properly service the workload. It may be fine if this is not a regular occurrence, but if this is happening all the time, you might need to upgrade the server’s hardware. One silver lining is that the presence of swap allows the server to keep running. Without it, the OOMkiller will just kill the process with the highest memory usage in a not very nice manner using signal 9 (MongoDB, in this case), which is a much more disruptive process.

You can check how much work the MongoDB server is doing by using mongostat command. If the output of mongostat indicates a heavily loaded server, you might need to increase the RAM of the server.

Memory usage seems very high, and looks llike mongodb is consuming more than 50% of the RAM:

It is entirely possible for MongoDB to use more than 50% of RAM. The main ~50% allocation is for WiredTiger use only, and MongoDB would need to allocate more RAM on top of this to service incoming connections (up to 1MB per connection), in-memory sorting, and other operational works.

Best regards
Kevin

Oscar_Cervantes · 2023-03-02T06:54:21.494Z

Hi Kevin

I’m struggling with kind of the same situation but in a sharded cluster. Is it possible to limit the amount of RAM that mongo consumes?. I’d like to avoid a crash of one of my servers. This while we analyze mongostats and dicide to add more RAM.

Thanks