Need help getting MongoDB working on CentOS 7: 'Job for mongod.service failed because a timeout was exceeded.'

Alex_Baker · 2020-12-11T05:43:04.513Z

Hey everyone, I’m not really much of a MongoDB user, but I have a web application, part of which relies on Mongo, that I’m trying to install. I have a CentOS 7 server running WHM/cPanel and I’ve followed the setup instructions and it all seems to be installed, but when I try to start the service I get Job for mongod.service failed because a timeout was exceeded. See "systemctl status mongod.service" and "journalctl -xe" for details.

systemctl status mongod.service gives me this:
`● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Fri 2020-12-11 04:49:27 UTC; 1min 58s ago
     Docs: https://docs.mongodb.org/manual
  Process: 26266 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=0/SUCCESS)
  Process: 26263 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 26260 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 26257 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)

Dec 11 04:47:57 chess.gates.com mongod[26266]: about to fork child process, waiting until server is ready for connections.
Dec 11 04:47:57 chess.gates.com mongod[26266]: forked process: 26268
Dec 11 04:47:58 chess.gates.com mongod[26266]: child process started successfully, parent exiting`

and journalctl:

journalctl -f _COMM=mongod -- Logs begin at Mon 2020-03-23 05:08:30 UTC. -- Dec 05 02:36:33 chess.gates.com mongod[14579]: child process started successfully, parent exiting Dec 11 03:20:15 chess.gates.com mongod[22034]: about to fork child process, waiting until server is ready for connections. Dec 11 03:20:15 chess.gates.com mongod[22034]: forked process: 22036 Dec 11 03:20:17 chess.gates.com mongod[22034]: child process started successfully, parent exiting Dec 11 03:27:44 chess.gates.com mongod[22405]: about to fork child process, waiting until server is ready for connections. Dec 11 03:27:44 chess.gates.com mongod[22405]: forked process: 22407 Dec 11 03:27:45 chess.gates.com mongod[22405]: child process started successfully, parent exiting Dec 11 04:47:57 chess.gates.com mongod[26266]: about to fork child process, waiting until server is ready for connections. Dec 11 04:47:57 chess.gates.com mongod[26266]: forked process: 26268 Dec 11 04:47:58 chess.gates.com mongod[26266]: child process started successfully, parent exiting

My mongod.conf is just the default:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
#  engine:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.

#security:
#operationProfiling:
#replication:
#sharding:
## Enterprise-Only Options
#auditLog:
#snmp:

mongod.log

This bit seems relevant:

{"t":{"$date":"2020-12-11T04:47:58.316+00:00"},"s":"W", "c":"FTDC", "id":23913, "ctx":"initandlisten","msg":"Error checking directory '{sysBlockPathStr}': {ec_message}","attr":{"sysBlockPathStr":"/sys/block","ec_message":"No such file or directory"}}

And it’s right, there’s no /sys/block folder on my server and it won’t let me create it either (mkdir: cannot create directory ‘block’: No such file or directory), and I’m not sure what the function of this folder is meant to be or why it’s not there.

But the bit that actually stops it running seems to be this:

{“t”:{“$date”:“2020-12-11T04:49:27.287+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23377, “ctx”:“SignalHandler”,“msg”:“Received signal”,“attr”:{“signal”:15,“error”:“Terminated”}}
{“t”:{“$date”:“2020-12-11T04:49:27.287+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23378, “ctx”:“SignalHandler”,“msg”:“Signal was sent by kill(2)”,“attr”:{“pid”:1,“uid”:0}}
{“t”:{“$date”:“2020-12-11T04:49:27.287+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23381, “ctx”:“SignalHandler”,“msg”:“will terminate after current cmd ends”}`

I don’t know what this could mean other than the system is terminating mongo for some reason. This is where I could use some help: where do I start in figuring out what’s killing it? Has anyone successfully run mongo in a similar environment before with any success?

Sorry for the long message and general lack of knowledge, and thanks in advance to anyone who can help me

chris · 2020-12-11T13:14:06.719Z

It is very weird that your Centos7 system would not have a /sys/block/ directory. This is something you should look in to.

The Full Time Diagnostic Data Capture (FTDC) requires access to record block device performance.

I do not recommend doing the following but rather refer you to my first comment. However FTDC can be disabled.

To disable FTDC, start up the mongod or mongos with the diagnosticDataCollectionEnabled: false option specified to the setParameter setting in your configuration file:

setParameter:
  diagnosticDataCollectionEnabled: false

mongodb.com

MongoDB Performance — MongoDB Manual

Alex_Baker · 2020-12-13T19:20:44.339Z

Thanks for the reply. Okay, I didn’t realise how strange that was (didn’t know what /sys/block was for until this issue) but I’ll contact my hosting provider (GoDaddy vps) and see if they have any idea.

I tried adding that parameter to my mongod.conf, but it didn’t change anything? I still have the same error, and in my log it still says

{"t":{"$date":"2020-12-13T16:15:43.212+00:00"},"s":"W", "c":"FTDC", "id":23913, "ctx":"initandlisten","msg":"Error checking directory '{sysBlockPathStr}': {ec_message}","attr":{"sysBlockPathStr":"/sys/block","ec_message":"No such file or directory"}} {"t":{"$date":"2020-12-13T16:15:43.212+00:00"},"s":"I", "c":"FTDC", "id":20625, "ctx":"initandlisten","msg":"Initializingfull-time diagnostic data capture","attr":{"dataDirectory":"/var/lib/mongo/diagnostic.data"}}

Is there something else I need to do to get it to recognise that I’ve changed something in the config or something like that?

Thanks

Alex_Baker · 2020-12-13T19:20:53.119Z

Actually something I noticed: I have /sys/dev/block but not /sys/block. I don’t really understand the relationship between these two and don’t know if this changes anything?

chris · 2020-12-13T23:40:02.829Z

I was expecting FTDC not to run with that parameter. But I guess I am wrong.

Anyway as you rightly point out mongod is being killed.

Did you install mongodb using the package manager or tarball ?

Looking at the provided default configuration for mongodb on centos7 by the packages mongod is setup as forking. Your logs show this is happening on the mongod side but it might be that some changes occurred to the systemd unit. The timeout seems like the Type=forking is missing from the [service] section.

If you run rpmverify mongodb-org-server the only thing output should be your /etc/mongod.conf file(if it has been edited). A yum reinstall mongodb-org-server would resolve any unexpected deviations.