Mongodb on Ubuntu -' Main process exited, code=exited, status=1/FAILURE'

Konstantina_Arnaoutaki · 2022-03-22T14:34:19.663Z

Mongo has suddenly stopped working, so I am trying to restart it I’ am getting again and again though errors like the below:

 mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Τρι 2022-03-22 12:19:53 EET; 3h 46min ago
     Docs: https://docs.mongodb.org/manual
  Process: 27559 ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf (code=exited, status=1/FAILURE)
 Main PID: 27559 (code=exited, status=1/FAILURE)

systemd[1]: Started MongoDB Database Server.
systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mongod.service: Unit entered failed state.
systemd[1]: mongod.service: Failed with result 'exit-code'.

I have revised the mongod.conf file, which is located in /etc and includes the below:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
# engine: wiredTiger
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 23654
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

security:
  authorization: enabled

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Also the mongod.service located in /lib/systemd/system:

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network.target

[Service]
User=mongodb
Group=mongodb
EnvironmentFile=-/etc/default/mongod
ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf
PIDFile=/var/run/mongodb/mongod.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target

I have also changed ownership using the below commands according to this post:

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock
sudo chown mongodb:mongodb /tmp/mongodb-23654.sock
sudo chown -R mongodb:mongodb /var/log/mongodb

I have checked the folders and removed .lock files from /var/lib/mongodb/ which now looks like below:

 /var/lib/mongodb$ ls
    collection-0-2356690320700739995.wt   collection-2-2356690320700739995.wt  collection-8-5563536140699528517.wt  index-15-5563536140699528517.wt  index-5-5563536140699528517.wt  _mdb_catalog.wt    WiredTiger.wt
    collection-0-5563536140699528517.wt   collection-2-5563536140699528517.wt  diagnostic.data                      index-1-5563536140699528517.wt   index-6-2356690320700739995.wt  sizeStorer.wt
    collection-10-5563536140699528517.wt  collection-4-2356690320700739995.wt  index-10-2356690320700739995.wt      index-21-5563536140699528517.wt  index-7-5563536140699528517.wt  storage.bson
    collection-12-5563536140699528517.wt  collection-4-5563536140699528517.wt  index-11-5563536140699528517.wt      index-3-2356690320700739995.wt   index-9-2356690320700739995.wt  WiredTiger
    collection-14-5563536140699528517.wt  collection-6-5563536140699528517.wt  index-1-2356690320700739995.wt       index-3-5563536140699528517.wt   index-9-5563536140699528517.wt  WiredTigerLAS.wt
    collection-20-5563536140699528517.wt  collection-8-2356690320700739995.wt  index-13-5563536140699528517.wt      index-5-2356690320700739995.wt   journal                         WiredTiger.turtle

According to a previous post I had once enabled wiredTiger engine from mongod.conf but have now changed it back to the .conf shared above. Should I remove these files as well from this directory?

Finally, I have also checked the firewall for this port I am trying to use, it seems active. Have no idea what I should fix I have read lots of related posts but haven’t done yet…

chris · 2022-03-22T23:28:19.094Z

What is in the log file?

What MongoDB version.
What OS, OS Version.

Konstantina_Arnaoutaki · 2022-03-23T08:28:47.791Z

The OS is Ubuntu16.04.7 LTS
Mongo db v4.2.0
Unfortunately the file /var/log/mongodb/mongod.log only includes logs two days ago, although I have just tried to start mongod again, I have also looked as syslog but nothing is in there too.

Ramachandra_Tummala · 2022-03-23T10:44:57.325Z

Is PIDfile matching in services file and mongod.cfg
Why to chmod temp sock files
Looks like you started mongod earlier on port 27017 and now on new port
Try to move these *.sock files and see if it comes up

Konstantina_Arnaoutaki · 2022-03-23T12:22:17.262Z

You mean to remove both mongodb-23654.sock and mongodb-27017.sock?

> sudo rm /tmp/mongodb-23654.sock
> sudo rm /tmp/mongodb-27017.sock

Well the 27017 should have been created once I have changed for a while the mongod.conf

This kind of suggestions
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock
were posted along with other suggested solutions in StackOverflow actually…

steevej · 2022-03-23T12:46:15.820Z

Konstantina_Arnaoutaki:

other suggested solutions in StackOverflow

Not always good.

Konstantina_Arnaoutaki:

/tmp/mongodb-27017.sock

and similar are used by a running process and you must not temper them with

Konstantina_Arnaoutaki:

sudo chown mongodb:mongodb /tmp/mongodb-27017.sock

What you need to do is:

Make sure no process is listening to the corresponding port with
ss -tlnp | grep 27017
Then if no process is listening you may safely remove with
sudo /bin/rm -f /tmp/mongodb-27017.sock

chris · 2022-03-23T13:02:27.683Z

As the logging is not providing error information. I would suggest running mongod manually at this point.

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf

This should error out. Given that nothing is appearing

Otherwise you’re basically randomly changing/removing files/configuration without cause.

Konstantina_Arnaoutaki · 2022-03-23T15:40:57.217Z

I removed this

tmp/mongodb-27017.sock

There is only 23654. sock available now, still it didn’t work.

Konstantina_Arnaoutaki · 2022-03-23T15:51:55.931Z

Nothing seems to be logged on the file

/var/lib/mongodb/mongodb.log

it is empty.
I tried also adding quotes as per the official instructions

mongodb.com

Configuration File Options — MongoDB Manual

so the path to log files looks like
path: "/var/log/mongodb/mongod.log"

still the same error…

steevej · 2022-03-23T16:11:04.967Z

We need output of the following commands

ss -tlnp
ps -aef | grep [m]ongod
ls -l /tmp/mongodb-*

Is the log file

Konstantina_Arnaoutaki:

/var/lib/mongodb/mongodb.log

or

Konstantina_Arnaoutaki:

/var/log/mongodb/mongod.log

Konstantina_Arnaoutaki · 2022-03-23T16:41:35.663Z

apologies for the misunderstanding, the correct file is

/var/log/mongodb/mongod.log

Result of

ss -tlnp
is

State       Recv-Q Send-Q                                                                         Local Address:Port                                                                                        Peer Address:Port
LISTEN      0      128                                                                                        *:26603                                                                                                  *:*                   users:(("node",pid=2711,fd=23))
LISTEN      0      128                                                                                        *:80                                                                                                     *:*
LISTEN      0      5                                                                                  127.0.1.1:53                                                                                                     *:*
LISTEN      0      128                                                                                        *:22                                                                                                     *:*
LISTEN      0      5                                                                                  127.0.0.1:631                                                                                                    *:*
LISTEN      0      128                                                                                        *:3000                                                                                                   *:*                   users:(("node",pid=2420,fd=23))
LISTEN      0      128                                                                                127.0.0.1:33980                                                                                                  *:*                   users:(("node",pid=2711,fd=22))
LISTEN      0      128                                                                                       :::80                                                                                                    :::*
LISTEN      0      128                                                                                       :::22                                                                                                    :::*
LISTEN      0      5                                                                                        ::1:631                                                                                                   :::*

and result of ls -l /tmp/mongodb-*
is
srwx------ 1 mongodb mongodb 0 Μάρ 23 17:24 /tmp/mongodb-23654.sock

The ps -aef | grep [m]ongod gave nothing.

steevej · 2022-03-23T18:08:20.143Z

Okay.

Now remove /tmp/mongodb-… with

sudo /bin/rm -f /tmp/mongodb-23654.sock

Now manually start with

chris:

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf

and post a screenshot of doing the above. Also post the content of

Konstantina_Arnaoutaki:

/var/log/mongodb/mongod.log

Konstantina_Arnaoutaki · 2022-03-23T19:16:16.382Z

It still does not work and the /var/log/mongodb/mongod.log seems empty.

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf
sudo vim /var/log/mongodb/mongod.log
sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf
sudo systemctl status mongod
● mongod.service - MongoDB Database Server
>    Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
>    Active: failed (Result: exit-code) since Τετ 2022-03-23 20:49:30 EET; 2min 56s ago
>      Docs: https://docs.mongodb.org/manual
>   Process: 27417 ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf (code=exited, status=1/FAILURE)
>  Main PID: 27417 (code=exited, status=1/FAILURE)
> 
 systemd[1]: Started MongoDB Database Server.
systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mongod.service: Unit entered failed state.
systemd[1]: mongod.service: Failed with result 'exit-code'.

I suppose the logs are saved somewhere else so tried to find where with the below, but I am not sure if it correctly used:

grep -r 'mongod.service: Main process exited, code=exited, status=1/FAILURE' *

Ramachandra_Tummala · 2022-03-24T05:08:41.981Z

Please show screenshot of the manual run
Did it complete successfully?
Can you run mongod as normal user on a different port,dbpath,logpath(not using your config file)

Konstantina_Arnaoutaki · 2022-03-24T08:30:00.194Z

The results when running the command you suggested:

image869×164 8.18 KB

Konstantina_Arnaoutaki · 2022-03-24T08:30:34.032Z

If I just try to mongod I get the below:

image1398×291 10.8 KB

Ramachandra_Tummala · 2022-03-24T08:49:59.924Z

When you run mongod without any parameters it will try to start it on default port 27017 and default dirpath /data/db
Do you have this dir existing?

I asked you to run on different dbpath,logpath,port like this to rule out any issues with dirs
Something like below

Mongod --dbpath yourhomedir --logpath yourhomedir/mongod.log --port 29000 --fork

Once your mongod is up you can connect as
Mongo --port 29000

Konstantina_Arnaoutaki · 2022-03-24T09:15:50.794Z

The dir /data/db does exist

image929×309 9.11 KB

Konstantina_Arnaoutaki · 2022-03-24T09:17:24.509Z

Anyway I run this

mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongod.log --port 29000 --fork

and get the below

image1895×134 8.02 KB

Ramachandra_Tummala · 2022-03-24T09:45:00.457Z

Please use your home dir or someother dir where mongod can write.I specifically mentioned this
Do not use the dirs like /var/* as given in /etc/mongod.conf

Regarding /data/db error what is your os?
Your files are owned as root
They should be owned by mongod