Mongodb on Ubuntu -' Main process exited, code=exited, status=1/FAILURE'

steevej · 2022-03-23T18:08:20.143Z

Okay.

Now remove /tmp/mongodb-… with

sudo /bin/rm -f /tmp/mongodb-23654.sock

Now manually start with

chris:

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf

and post a screenshot of doing the above. Also post the content of

Konstantina_Arnaoutaki:

/var/log/mongodb/mongod.log

Konstantina_Arnaoutaki · 2022-03-23T19:16:16.382Z

It still does not work and the /var/log/mongodb/mongod.log seems empty.

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf
sudo vim /var/log/mongodb/mongod.log
sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf
sudo systemctl status mongod
● mongod.service - MongoDB Database Server
>    Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
>    Active: failed (Result: exit-code) since Τετ 2022-03-23 20:49:30 EET; 2min 56s ago
>      Docs: https://docs.mongodb.org/manual
>   Process: 27417 ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf (code=exited, status=1/FAILURE)
>  Main PID: 27417 (code=exited, status=1/FAILURE)
> 
 systemd[1]: Started MongoDB Database Server.
systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mongod.service: Unit entered failed state.
systemd[1]: mongod.service: Failed with result 'exit-code'.

I suppose the logs are saved somewhere else so tried to find where with the below, but I am not sure if it correctly used:

grep -r 'mongod.service: Main process exited, code=exited, status=1/FAILURE' *

Ramachandra_Tummala · 2022-03-24T05:08:41.981Z

Please show screenshot of the manual run
Did it complete successfully?
Can you run mongod as normal user on a different port,dbpath,logpath(not using your config file)

Konstantina_Arnaoutaki · 2022-03-24T08:30:00.194Z

The results when running the command you suggested:

image869×164 8.18 KB

Konstantina_Arnaoutaki · 2022-03-24T08:30:34.032Z

If I just try to mongod I get the below:

image1398×291 10.8 KB

Ramachandra_Tummala · 2022-03-24T08:49:59.924Z

When you run mongod without any parameters it will try to start it on default port 27017 and default dirpath /data/db
Do you have this dir existing?

I asked you to run on different dbpath,logpath,port like this to rule out any issues with dirs
Something like below

Mongod --dbpath yourhomedir --logpath yourhomedir/mongod.log --port 29000 --fork

Once your mongod is up you can connect as
Mongo --port 29000

Konstantina_Arnaoutaki · 2022-03-24T09:15:50.794Z

The dir /data/db does exist

image929×309 9.11 KB

Konstantina_Arnaoutaki · 2022-03-24T09:17:24.509Z

Anyway I run this

mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongod.log --port 29000 --fork

and get the below

image1895×134 8.02 KB

Ramachandra_Tummala · 2022-03-24T09:45:00.457Z

Please use your home dir or someother dir where mongod can write.I specifically mentioned this
Do not use the dirs like /var/* as given in /etc/mongod.conf

Regarding /data/db error what is your os?
Your files are owned as root
They should be owned by mongod

Konstantina_Arnaoutaki · 2022-03-24T10:27:58.160Z

ok, I have the below:

image1085×119 6.37 KB

Konstantina_Arnaoutaki · 2022-03-24T13:13:27.996Z

My OS is Ubuntu 16.04.
And the ownerships along my files are:
for directories in / owned by root, apart from (as per directories around mongodb):

var/lib/mongodb owned by mongodb user
var/log/mongodb owned by mongodb user
I also noticed that /etc/mongod.conf is under root user, is this correct?

image921×543 16 KB

Then in my home directory all the files under /home/user/ are owned by user

steevej · 2022-03-24T13:27:04.726Z

Do not use systemctl status like

Konstantina_Arnaoutaki:

sudo systemctl status mongod

when you start a process manually like

Konstantina_Arnaoutaki:

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf

Use the commands

steevej:

ss -tlnp
ps -aef | grep [m]ongod
ls -l /tmp/mongodb-*

Konstantina_Arnaoutaki:

The dir /data/db does exist

But it is owned by root, not good.

Konstantina_Arnaoutaki · 2022-03-24T20:37:53.736Z

I have changed the ownership of the mongod.conf to user mongodb
sudo chown -R mongodb:mongodb /etc/mongod.conf

and then rerun manually with the command:
sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf

Now I am able to see logs inside

var/log/mongodb

2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten] allocator: tcmalloc
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten] modules: none
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten] build environment:
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten]     distmod: ubuntu1604
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten]     distarch: x86_64
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten]     target_arch: x86_64
2022-03-24T22:28:15.179+0200 I  CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 23654 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2022-03-24T22:28:15.182+0200 I  STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2022-03-24T22:28:15.182+0200 I  STORAGE  [initandlisten]
2022-03-24T22:28:15.182+0200 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2022-03-24T22:28:15.182+0200 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2022-03-24T22:28:15.182+0200 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1463M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2022-03-24T22:28:16.844+0200 E  STORAGE  [initandlisten] WiredTiger error (13) [1648153696:844130][24294:0x7f4bdfc9ab00], wiredtiger_open: __posix_open_file, 712: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1648153696:844130][24294:0x7f4bdfc9ab00], wiredtiger_open: __posix_open_file, 712: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied
2022-03-24T22:28:16.844+0200 E  STORAGE  [initandlisten] WiredTiger error (13) [1648153696:844481][24294:0x7f4bdfc9ab00], wiredtiger_open: __posix_open_file, 712: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1648153696:844481][24294:0x7f4bdfc9ab00], wiredtiger_open: __posix_open_file, 712: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied
2022-03-24T22:28:16.845+0200 E  STORAGE  [initandlisten] WiredTiger error (13) [1648153696:845360][24294:0x7f4bdfc9ab00], wiredtiger_open: __posix_open_file, 712: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1648153696:845360][24294:0x7f4bdfc9ab00], wiredtiger_open: __posix_open_file, 712: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied
2022-03-24T22:28:16.845+0200 W  STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
2022-03-24T22:28:16.845+0200 F  STORAGE  [initandlisten] Reason: 13: Permission denied

I believe that all the problems were some kind related to the wrong ownership some mongo related files or directories have.

steevej · 2022-03-24T20:46:17.981Z

Output of

ls -l /var/lib/mongodb/

Ramachandra_Tummala · 2022-03-25T02:15:53.082Z

All errors indicate some permissions issue
Something wrong in the way you are starting mongod
How can be your home dir /home/mongod?
How do you login to your system?
What does id command show?
Looks like you are switching to mongod user

Konstantina_Arnaoutaki · 2022-03-27T17:48:12.335Z

steevej:

ls -l /var/lib/mongodb/

image781×660 15.8 KB

Konstantina_Arnaoutaki · 2022-03-27T18:26:16.694Z

I get the below when running id

$ id
uid=1000(user) gid=1000(user) groups=1000(user),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),129(sambashare)

Well I have also tried the below with the correct homedir, using the user ‘user’ but still getting same error:

mongod  --dbpath /home/user/mongodb --logpath /home/user/mongodb/mongod.log --port 27017
2022-03-27T21:06:52.191+0300 F  CONTROL  [main] Failed global initialization: FileNotOpen: Failed to open "/home/user/mongodb/mongod.log"

I have noticed at least 3 different ownership users which basically own the various directories as below:

user:user for directory /home/user/
user :root for most directories in / apart from
- /data owned by mongodb
- /var/log/mongodb owned by mongodb
- /var/lib/mongodb owned by mongodb
- /etc/mongod.conf owned by mongodb

Is there any other file in / that is related to mongod and should change its ownership to mongodb?
Or in any case, does the above structure with their related ownerships sound problematic?

Ramachandra_Tummala · 2022-03-28T01:48:44.246Z

Konstantina_Arnaoutaki:

--logpath /home/user/mongodb/

Does this dir exist?

ls -lrt /home/user/mongodb
or cd /home/user/mongodb

Did you create and try with /data/db?
You have to use sudo to create dir under root folder and chown ownership similar to what was done for /var dirs

Konstantina_Arnaoutaki · 2022-03-29T10:32:06.809Z

I created this dir

/home/user/mongodb

and used
sudo chown -R mongodb:mongodb /home/user/mongodb/

to change ownership to mongodb.
This
/data/db
already existed in root dir, but tried to remove and then recreate and use it due to permission issues (see below)

mongod
2022-03-29T13:20:26.963+0300 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2022-03-29T13:20:26.995+0300 I  CONTROL  [initandlisten] MongoDB starting : pid=4456 port=27017 dbpath=/data/db 64-bit host=snf-870308
2022-03-29T13:20:26.997+0300 I  CONTROL  [initandlisten] db version v4.2.0
2022-03-29T13:20:26.997+0300 I  CONTROL  [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten] allocator: tcmalloc
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten] modules: none
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten] build environment:
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten]     distmod: ubuntu1604
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten]     distarch: x86_64
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten]     target_arch: x86_64
2022-03-29T13:20:26.998+0300 I  CONTROL  [initandlisten] options: {}
2022-03-29T13:20:26.999+0300 I  STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/db, terminating
2022-03-29T13:20:26.999+0300 I  NETWORK  [initandlisten] shutdown: going to close listening sockets...
2022-03-29T13:20:27.000+0300 I  NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2022-03-29T13:20:27.000+0300 I  -        [initandlisten] Stopping further Flow Control ticket acquisitions.
2022-03-29T13:20:27.000+0300 I  CONTROL  [initandlisten] now exiting
2022-03-29T13:20:27.000+0300 I  CONTROL  [initandlisten] shutting down with code:100

Should I use this to add permissions :

sudo chmod -R go+w /data/db

or better remove and start from scratch all the installation once again…although I have a few collections on .

kevinadi · 2022-03-29T11:10:10.922Z

Hi @Konstantina_Arnaoutaki welcome to the community!

From what I gathered on the information so far, it appears that the main issue here is permissions and consistency, as we are seeing some files under the dbpath are owned by one user (mongodb), and some other files are owned by root. This mix of ownership implies that mongod was not consistently started under one user: one time it was run as root, other times it was not.

In addition, it appears that the file locations are also not consistent, e.g. sometimes it’s under the root directory (/data/db), sometimes it’s under other directories (/home/user/mongodb, /var/lib/mongodb or similar, among others).

One particular thing I noticed is that in your initial post, the /etc/mongod.conf file contains instructions for mongod to listen to port 23654. The existence of the file mongodb-23654.sock and also mongodb-27017.sock suggested that this was changed for some reason (as @Ramachandra_Tummala pointed out earlier). This is not the standard MongoDB port, so it’s probably best to set all settings in the config files to its default values so that there are no unexpected values to deal with in the future.

Depending on your preference, I would suggest to stick to one method of running MongoDB, be it from a service, running manually, or some other method you prefer, and keep using that method throughout. It may be easier in the long run to use the existing service and fix the permission issues, but your mileage may vary.

I would suggest to:

Set the port setting in the /etc/mongod.conf to be 27017 instead of 23654
The ownership of the files under directory /var/lib/mongodb and /var/log/mongodb should be owned by mongodb user, as per the settings in /etc/mongod.conf
Use the service starter to start the MongoDB process (systemctl), and avoid using sudo to run mongod manually
If the service failed to start, please post the error message in /var/log/mongodb/mongod.log. Since this file should be owned by the mongodb user, please be careful in extracting the content of this file as using sudo incorrectly could change the ownership of this file and put us back to the same permission issues again
If anyone else have sudo access to this machine, please let them know not to undo all your work that was done in the previous points by running sudo mongod or change any permission in any MongoDB-related directories

Best regards
Kevin