MongoDB 8 doesn’t deem to close old connections (each old connection stays at 100% CPU on one core)

GoLdUcK1990 · 2025-02-24T15:48:28.488Z

Hello everyone,

We have a problem on two separate replica sets (on the same cluster) plus a single database (on the same cluster) where old connections do not close. Checking with htop or top -H -p $PID shows that some connections opened long ago are never closed. Each of these connections consumes 100% of one VM core, regardless of the total number of CPU cores available.

Environment Details

Each replica set has 3 VMs with:

Almalinux 9
16 vCPUs (we’ve tested both 2 sockets × 8 cores, and 1 socket × 16 cores)
8 GB RAM
MongoDB 8.0.4
Proxmox 8.2 (hypervisor)
OPNSense firewall

Physical nodes (8× Dell PE C6420) each have:

2× Xeon Gold 6138
256 GB RAM
2 NUMA zones

MongoDB Configuration

Below is the current mongod.conf, inspired by a MongoDB Atlas configuration:

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

storage:
  dbPath: /space/mongodb
  engine: 'wiredTiger'
  wiredTiger:
    engineConfig:
       configString: 'cache_size=1024MB'

processManagement:
  pidFilePath: /var/run/mongodb/mongod.pid
  timeZoneInfo: /usr/share/zoneinfo

net:
  port: 27017
  bindIp: 172.24.200.13,REDACTED.THE.DOMAIN.com

tls:
  mode: allowTLS
  certificateKeyFile: /space/mongodb/kort-db-cat.pem
  CAFile: /space/mongodb/kort-db-cacat.pem
  allowConnectionsWithoutCertificates: true
  clusterCAFile: /space/mongodb/kort-db-cacat.pem
  disabledProtocols: 'TLS1_0,TLS1_1'

setParameter:
  allowRolesFromX509Certificates: 'true'
  authenticationMechanisms: 'SCRAM-SHA-1,SCRAM-SHA-256,MONGODB-X509'
  diagnosticDataCollectionDirectorySizeMB: '400'
  honorSystemUmask: 'false'
  internalQueryGlobalProfilingFilter: 'true'
  internalQueryStatsRateLimit: '0'
  lockCodeSegmentsInMemory: 'true'
  maxIndexBuildMemoryUsageMegabytes: '100'
  minSnapshotHistoryWindowInSeconds: '300'
  notablescan: 'false'
  reportOpWriteConcernCountersInServerStatus: 'true'
  suppressNoTLSPeerCertificateWarning: 'true'
  tlsWithholdClientCertificate: 'true'
  ttlMonitorEnabled: 'true'
  watchdogPeriodSeconds: '60'
  logLevel: 0

security:
  authorization: enabled
  keyFile: /space/mongodb/kort-db.key
  javascriptEnabled: true
  clusterAuthMode: keyFile

operationProfiling:
  mode: slowOp
  slowOpThresholdMs: 300
  slowOpSampleRate: 0.5

replication:
  replSetName: "kort-db"

We previously had a simpler config, and the issue still occurred:

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

storage:
  dbPath: /space/mongodb
  engine: 'wiredTiger'

processManagement:
  pidFilePath: /var/run/mongodb/mongod.pid
  timeZoneInfo: /usr/share/zoneinfo

net:
  port: 27017
  bindIp: 172.24.200.13,REDACTED.THE.DOMAIN.com

tls:
  mode: allowTLS
  certificateKeyFile: /space/mongodb/kort-db-cat.pem
  CAFile: /space/mongodb/kort-db-cacat.pem
  allowConnectionsWithoutCertificates: true
  clusterCAFile: /space/mongodb/kort-db-cacat.pem

security:
  authorization: enabled
  keyFile: /space/mongodb/kort-db.key
  clusterAuthMode: keyFile

replication:
  replSetName: "kort-db"

Certificates

kort-db-cat.pem contains:

[LETS ENCRYPT SPECIFIC CERT FOR DOMAIN]
[KEY FOR CERTIFICATE]

kort-db-cacat.pem is a concatenation (in this order):

[LETS ENCRYPT ROOT X1]
[LETS ENCRYPT INTERMEDIATE E6]
[LETS ENCRYPT SPECIFIC CERT FOR DOMAIN]

System-Level Modifications

In /etc/sysctl.conf:

fs.file-max = 2097152
vm.max_map_count = 1048575
vm.swappiness = 1
net.ipv4.tcp_fastopen = 3

We also have a systemd one-shot service that sets the following:

ExecStart=/bin/bash -c 'echo always > /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/bash -c 'echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag'
ExecStart=/bin/bash -c 'echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none'
ExecStart=/bin/bash -c 'echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag'
ExecStart=/bin/bash -c 'echo 1 > /proc/sys/vm/overcommit_memory'
ExecStart=/bin/bash -c 'echo 1 > /proc/sys/vm/swappiness'
ExecStart=/bin/bash -c 'echo 3 > /proc/sys/net/ipv4/tcp_fastopen'
ExecStart=/bin/bash -c 'echo 0 > /proc/sys/vm/zone_reclaim_mode'

And our mongod.service file:

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongod.conf"
Environment="MONGODB_CONFIG_OVERRIDE_NOFORK=1"
Environment="GLIBC_TUNABLES=glibc.pthread.pthread.rseq=0"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod $OPTIONS

RuntimeDirectory=mongodb
LimitFSIZE=infinity
LimitCPU=infinity
LimitAS=infinity
LimitNOFILE=64000
LimitNPROC=64000
LimitMEMLOCK=infinity
TasksMax=infinity
TasksAccounting=false

[Install]
WantedBy=multi-user.target

also:

The Linux kernel’s idle connection timeout is 7200. Lowering it to 300 didn’t help.
The cluster connection uses a mongo+srv connection string.

How the Issue Manifests

Many stuck connections (top on a specific PID for mongod):

conn-and-top1920×1010 177 KB

htop view:

conn-and-htop1920×1002 181 KB

Connection 948 shows as disconnected from the cluster half an hour ago but remains active at 100% CPU:

log-and-top1920×756 242 KB

As you can see with conn948, /var/log/mongo/mongod.log confirms that the connection was closed a while ago.

Unsuccessful Attempts So Far

Forcing the VM to use only one NUMA zone
Lowering the idle connection timeout from 7200 to 300

Running strace on the stuck process revealed attempts to access /proc/pressure, which is disabled on RHEL-like systems by default. After enabling it by adding psi=1 to the kernel boot parameters, strace no longer reported those errors, but the main problem persisted.
For add psi=1 we use

grubby --args="audit=1 selinux=1" --update-kernel=ALL

For the psi issue we cannot find nothing on the internet, hope can helps someone

Restarting the replica set one node at a time frees up the CPU for a few hours/days, until multiple connections get stuck again.

How to Reproduce

We’ve noticed the Studio 3T client on macOS immediately leaves these connections stuck. Simply open and then disconnect (with the official “disconnect” option) from the replica set: the connections remain hung, each at 100% CPU. Our connection string looks like:

studio3t-connstring1268×436 40.3 KB

Looking for Solutions

Has anyone encountered (and solved) a similar issue?
As a temporary workaround, is it possible to schedule a task that kills these inactive connections automatically? (It’s not elegant, but it might help for now.) If you have insights into the root cause, please share!

We’re still experimenting to isolate the bug. Once we figure it out, we’ll update this post.

If you’ve read this far, thank you so much!

chris · 2025-02-25T15:15:08.727Z

Try 8.0.5.

Possibly related to this:

Constantly high CPU usage after OS restart Ops and Admin

We have used a mongo v8.0.1 replicaSet (2 nodes + arbiter) on our production servers (Rocky 9.4 with 12 cores) since some weeks. I restartet the whole system of primary after RAM upgrade some days ago. We have seen 100% cpu usage caused by mongo process on all cores since that. We have tested folllowing steps: mongo service restart look for slow queries (We also tried to stop all our services calling mongo, but cpu usage won’t change.) node reinitilized switch primary to another node disabl…

GoLdUcK1990 · 2025-02-25T21:34:31.883Z

Yes, i’ve posted on Reddit and u/ MaximKorolev point me on this issue https://jira.mongodb.org/browse/SERVER-97842, the problem was openssl on el9 and Mongo, so, today, we do the upgrade to mongo 8.0.5 for the fix like the answer on the issue preannunced, and the problem is fixed! Now the CPU on the replicaset is very low as expected.
Thanks for the answer!

system · 2025-03-02T21:35:25.142Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.