Greetings, folks!
We have been sitting on Community version 4.4.18 for quite some time and have not known any problems. We decided to upgrade to 7.0.5 in stages.
The cluster consists of one master and 5 replicas (all machines 24 CPU, 94 Gb RAM). The configuration file is fairly standard and simple.
systemLog:
destination: file
logAppend: true
quiet: true
path: /var/log/mongodb/mongod.log
replication:
replSetName: global
oplogSizeMB: 15000
enableMajorityReadConcern=false
setParameter:
enableFlowControl: false
flowControlTargetLagSeconds: 1
# Where and how to store data.
storage:
dbPath: /var/lib/mongodb
journal:
storage.journal.enabled
commitIntervalMs: 500
engine: "wiredTiger"
wiredTiger:
engineConfig:
cacheSizeGB: 48
# how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
pidFilePath: /etc/mongodb/mongod.pid
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0
maxIncomingConnections: 16384
security:
authorization: enabled
keyFile: /etc/mongodb/keyfile
operationProfiling:
slowOpThresholdMs: 200
mode: slowOp
Upgraded to 7.0.5 step by step, deleted the parameters that would have deprecated
enableMajorityReadConcern=false
storage.journal.enabled
At first we didn’t notice any changes and degradations in terms of application operation. But monitoring started to show io utilization growth of more than 80% right after upgrade to version 5.
On all machines the same picture.
According to the documentation https://mongodb.prakticum-team.ru/docs/manual/reference/replica-configuration/#mongodb-rsconf-rsconf.writeConcernMajorityJournalDefault
If any voting member of a replica set uses the in-memory storage engine, you must set
writeConcernMajorityJournalDefault
to false
We changed it, IO reads and writes have dropped and utilization on the master has also dropped, but on replicas the figures are the same.
After that, we decided to try to move the journal to a separate disk, all utilization went to the disk with the journal.
After a while, we started catching various side effects.
I don’t really understand where to dig and how to remove utilization from journal disk. Any advice would be appreciated