RHEL9 systemctl gets permission error when database location moved or port changed

Kevin_Barnes1 · 2024-04-17T22:41:56.649Z

There is an issue with RHEL9 starting the mongod deamon with systemctl start mongod.

Two examples of this issue are:

Moving the database location
Changing Listing port

Example 1. Database location
storage:
dbPath: /var/lib/mongo

network interfaces

net:
port: 27017

systemctl start mongod – this works just fine as the defaults.

Now lets move the DB directory to \var\lib\mongo1 – SAME mount point, SAME permissions

[root@REDHAT-3212 firewalld]# ll /var/lib | grep mongo
drwxr-xr-x. 5 mongod mongod 4096 Apr 17 15:27 mongo
drwxr-xr-x. 2 mongod mongod 6 Apr 17 15:27 mongo1

storage:
dbPath: /var/lib/mongo1
systemctl start mongod –

FAILS with permission issue:

{“t”:{“$date”:“2024-04-17T15:30:16.960-05:00”},“s”:“E”, “c”:“WT”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error message”,“attr”:{“error”:13,“message”:“[1713385816
:960404][159449:0x7f26d62a0b80], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 815: /var/lib/mongo1/WiredTiger.lock: handle-open: open: Permission denied”}}

BUT – what if we SU to mongod and run it:
su mongod -c “mongod -f /etc/mongod.conf”

It WORKS just fine – Mongo daemon starts up just fine.
Its only an issuer when starting from systemctl

Example 2. Database port — lets change the port
storage:
dbPath: /var/lib/mongo

network interfaces

net:
port: 26019

When we start with systemctl it FAILS with BIND issue
systemctl start mongod –

{“t”:{“$date”:“2024-04-17T15:35:44.571-05:00”},“s”:“E”, “c”:“CONTROL”, “id”:20568,
“ctx”:“initandlisten”,“msg”:“Error setting up listener”,“attr”:{“error”:{“code”:9001,“codeName”:
“SocketException”,“errmsg”:“setup bind :: caused by :: Permission denied”}}}

But lets start it with SU
su mongod -c “mongod -f /etc/mongod.conf”

{“t”:{“$date”:“2024-04-17T15:36:52.742-05:00”},“s”:“I”, “c”:“NETWORK”, “id”:23016, “ctx”:“listener”,“msg”:“Waiting for connections”,“attr”:{“port”:26019,“ssl”:“off”}}
IT WORKS just fine

Why do we get permission issues with starting with systemctl – as a service it should start as mongod

When starting it with normal parameters we see the daemon running under mongod.
mongod 159959 1 29 15:38 ? 00:00:00 /usr/bin/mongod -f /etc/mongod.conf

steevej · 2024-04-18T14:50:41.072Z

It is hard to help you with your systemctl issue if you are not sharing the unit file that you are using.

The permission denied is on

Kevin_Barnes1:

/var/lib/mongo1/WiredTiger.lock

so it would be nice to have the ll on the file.

One thing I notice is that on my laptop the user is mongodb not mongod like you seem to have. My installation is old since I mostly work out of Atlas so newer install may differ. May be you had a previous installation where mongodb was used so now obviously with mongod it would not work.

Or perhaps, you manually created your directories with mongod but the unit file uses mongodb user and group.

Kevin_Barnes1 · 2024-04-18T19:12:40.562Z

I cannot do a ls on the file because it is not created - this is a new directory that is empty.

This is installed on a RHEL9 server using the following as it pulls it from the latest repo.
There was nothing creating by hand, everything was done via the install - and yes, I
agree mongod vs mongodb is a bit weird – seems it install differently on different systems.

yum install -y mongodb-org

thanks

steevej · 2024-04-18T19:55:41.927Z

Since it may install differently, it is doubly important:

steevej:

to help you with your systemctl issue if you are not sharing the unit file that you are using.

Kevin_Barnes1 · 2024-04-18T20:35:02.153Z

ok- I am not clear what the unit file you are requesting I post.
which file is this?
thanks

steevej · 2024-04-19T15:34:54.889Z

The unit file is the script used when you do systemctl start. See

Red Hat Customer Portal

Chapter 1. Working with systemd unit files Red Hat Enterprise Linux...

Access Red Hat’s knowledge, guidance, and support through your subscription.

We need to see the one for your mongod service.

Kevin_Barnes1 · 2024-04-20T15:37:16.508Z

found it - here it is

cat mongod.service

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongod
Group=mongod
Environment=“OPTIONS=-f /etc/mongod.conf”
Environment=“MONGODB_CONFIG_OVERRIDE_NOFORK=1”
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod $OPTIONS
RuntimeDirectory=mongodb

file size

LimitFSIZE=infinity

cpu time

LimitCPU=infinity

virtual memory size

LimitAS=infinity

open files

LimitNOFILE=64000

processes/threads

LimitNPROC=64000

locked memory

LimitMEMLOCK=infinity

total threads (user+kernel)

TasksMax=infinity
TasksAccounting=false

Recommended limits for mongod as specified in

https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

chris · 2024-04-20T16:17:38.015Z

Please review and update your posts:

Formatting code and log snippets in posts Getting Started

If you are including code and log snippets in a post, consider if there are ways to make these more readable using some of the available formatting options. Well formatted posts will help other users understand and engage with your discussion. Some quick tips for improved formatting: Include short code and log snippets as inline text rather than images. Text is searchable for other users who might encounter a similar issue or need the exact text to help investigate the issue you are experien…

It is very difficult to read when not formatted correctly and risks the post being ignored by some members of the community.

When MongoDB stops working on a RHEL system after a change from the default setting one of the first thing to check is whether or not SELinux is set to enforcing . Check this using the getenforce command.

If SELinux is set to enforcing then it can be set to permissive or can be completely disabled.

To change from the defaults AND keep SELinux in enforcing mode follow the SELinux Instructions from the manual.

https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-red-hat/#configure-selinux

Kevin_Barnes1 · 2024-04-21T18:10:49.901Z

Yes, it is SELINUX which is the culprit .

I found this: https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-red-hat/
Which states:
###################################################
If your MongoDB deployment uses custom settings for any of the following:

[MongoDB connection ports
[dbPath

You cannot use the MongoDB supplied SELinux policy. An alternative is to create a [custom SELinux policy]

###############################################

It gives instructions on how to build a new policy

Download the policy repository.

git clone htps://github.com/mongodb/mongodb-selinux

Build the policy.
cd mongodb-selinuxmake
3. Apply the policy.
sudo make install

However, even doing that does not fix the issue.
Even checking that the new policy is added for the new folder (/data/mongo)

/data/mongo(/.*)?                              all files          system_u:object_r:mongod_var_lib_t:s0
/data/mongo.*                                      all files          system_u:object_r:mongod_var_lib_t:s0
/run/mongodb(/.*)?                                 all files          system_u:object_r:mongod_runtime_t:s0
/usr/bin/mongod                                    regular file       system_u:object_r:mongod_exec_t:s0
/var/lib/mongo(/.*)?                               all files          system_u:object_r:mongod_var_lib_t:s0
/var/log/mongodb(/.*)?                             all files          system_u:object_r:mongod_log_t:s0

We still get permissioned denied when starting up with systemctl.

It is irritating that its documented that it needs a fix, provides a fix, yet still does not work.

I can of course turn off SELINUX but not sure that is the best solution.

Thanks for the help

steevej · 2024-04-21T19:39:19.566Z

Kevin_Barnes1:

Thanks for the help

Thanks to you for digging up the information.

With

Kevin_Barnes1:

dbPath: /var/lib/mongo1

you might need to add

/var/lib/mongo1(/.*)?                               all files          system_u:object_r:mongod_var_lib_t:s0

And please as mentioned

chris:

Please review and update your posts:

Formatting code and log snippets in posts Getting Started

If you are including code and log snippets in a post, consider if there are ways to make these more readable using some of the available formatting options. Well formatted posts will help other users understand and engage with your discussion. Some quick tips for improved formatting: Include short code and log snippets as inline text rather than images. Text is searchable for other users who might encounter a similar issue or need the exact text to help investigate the issue you are experien…

Kevin_Barnes1 · 2024-04-21T19:53:47.403Z

I moved it to /data/mongo and you can see in semodule it is added

/data/mongo(/.*)?

Same permission issue - ugh

steevej · 2024-04-26T11:44:01.972Z

Kevin_Barnes1:

I moved it to /data/mongo

Since you moved things around you would need to share the whole configuration again. This is the only way to see possible mismatch.

We need the unit file, the mongodb configuration file, the command you used to start, the selinux config, the ls -ld of all directories involved.