Mogo error after path updated

Fabio_Ramohitaj · 2023-08-10T18:56:22.776Z

Hi @sam_ames,
Can you paste here you configuration file?
Have you got data under your new Path?

Please if are present, paste the mongo process here (i.e. ps -ef |grep mongo).
Paste the /etc/passwd
In the Extreme case, can you clean your data directory?

Regards

sam_ames · 2023-08-10T20:53:02.491Z

mongod.conf:

                                                                               /etc/mongod.conf                                                                                                          
# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /data/var/lib/mongodb
#  engine:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

data under new path:

sysadmin@server:~$ ls /data
beta.domain.com  lost+found  var

mongo process:

sysadmin@server:~$ ps -ef | grep mongo
sysadmin   83521   83229  0 20:45 pts/1    00:00:00 grep --color=auto mongo

/ect/passwd:

daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-network:x:100:102:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:101:103:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
messagebus:x:102:105::/nonexistent:/usr/sbin/nologin
systemd-timesync:x:103:106:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
syslog:x:104:111::/home/syslog:/usr/sbin/nologin
_apt:x:105:65534::/nonexistent:/usr/sbin/nologin
tss:x:106:112:TPM software stack,,,:/var/lib/tpm:/bin/false
uuidd:x:107:113::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:114::/nonexistent:/usr/sbin/nologin
sshd:x:109:65534::/run/sshd:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
landscape:x:111:116::/var/lib/landscape:/usr/sbin/nologin
fwupd-refresh:x:112:117:fwupd-refresh user,,,:/run/systemd:/usr/sbin/nologin
ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash
lxd:x:999:100::/var/snap/lxd/common/lxd:/bin/false
usbmux:x:113:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
sysadmin:x:1001:1001:,,,:/home/sysadmin:/bin/bash
mongodb:x:114:65534::/home/mongodb:/usr/sbin/nologin

I can wipe /data but I’m concernjed it might trash my ubuntu system entirely… I’ve had trouble previouisly, after deleting lopst+found so Id rather not…?

| only added 2 folders as you can see, above.

Many thanks

Fabio_Ramohitaj · 2023-08-10T22:27:29.686Z

Hi @sam_ames,
Stop the service and do the following command:
chown -R mongodb:mongodb /data/var/lib/mongodb
And try to start the mongod service again.

If It doesn’ t work, paste the status of service and the logs.

Regards

sam_ames · 2023-08-11T07:52:44.050Z

sysadmin@server:~$ chown -R mongodb:mongodb /data/var/lib/mongodb
chown: cannot read directory '/data/var/lib/mongodb': Permission denied
sysadmin@server:~$ sudo chown -R mongodb:mongodb /data/var/lib/mongodb
[sudo] password for sysadmin: 

sysadmin@server:~$ sudo systerctl start mongod
sudo: systerctl: command not found
sysadmin@server:~$ sudo systemctl start mongod
sysadmin@server:~$ sudo systemctl status mongod
× mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-08-11 07:44:29 UTC; 10s ago
       Docs: https://docs.mongodb.org/manual
    Process: 85895 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=100)
   Main PID: 85895 (code=exited, status=100)
        CPU: 31ms

Aug 11 07:44:29 server systemd[1]: Started MongoDB Database Server.
Aug 11 07:44:29 server mongod[85895]: {"t":{"$date":"2023-08-11T07:44:29.870Z"},"s":"I",  "c":"CONTROL",  "id":7484500, "ctx":"-","msg":"Environment variable MONGODB_CONFIG_OVERRIDE_NOFORK == 1, overriding \"processManagement.fo>
Aug 11 07:44:29 server systemd[1]: mongod.service: Main process exited, code=exited, status=100/n/a
Aug 11 07:44:29 server systemd[1]: mongod.service: Failed with result 'exit-code'.
lines 1-12/12 (END)

sysadmin@server:~$ sudo tail -f /var/log/mongodb/mongod.log
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"MigrationUtil-TaskExecutor","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"COMMAND",  "id":4784923, "ctx":"initandlisten","msg":"Shutting down the ServiceEntryPoint"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":4784928, "ctx":"initandlisten","msg":"Shutting down the TTL monitor"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":6278511, "ctx":"initandlisten","msg":"Shutting down the Change Stream Expired Pre-images Remover"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2023-08-11T07:44:29.880+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}

Frustrating…

But thanks for your support

Fabio_Ramohitaj · 2023-08-11T08:24:03.683Z

Hi @sam_ames

sam_ames:

Environment variable MONGODB_CONFIG_OVERRIDE_NOFORK == 1, overriding \"processManagement

Here is your error.

I would delete this environment variable and check if it starts running

Regards

steevej · 2023-08-11T10:44:57.808Z

This is the error

sam_ames:

chown: cannot read directory '/data/var/lib/mongodb': Permission denied

Do the chown using sudo.

Fabio_Ramohitaj · 2023-08-11T11:37:20.072Z

Hi @steevej

sam_ames:

sudo chown -R mongodb:mongodb /data/var/lib/mongodb
[sudo] password for sysadmin

He corrected in the next line.

Best Rergards

steevej · 2023-08-11T12:41:47.685Z

I really must improve my reading skills. B-)

~~ /) ~~

Fabio_Ramohitaj · 2023-08-11T14:19:36.965Z

Don’t worry @steevej , who knows how many times I skip whole parts to read

sam_ames · 2023-08-11T15:40:21.987Z

sam_ames:

MONGODB_CONFIG_OVERRIDE_NOFORK

I commented out the var.
Now I have:

sysadmin@soft-serve:~$ sudo nano /usr/lib/systemd/system/mongod.service
sysadmin@soft-serve:~$ sudo systemctl start mongod
Warning: The unit file, source configuration file or drop-ins of mongod.service changed on disk. Run 'systemctl daemon-reload' to reload units.
sysadmin@soft-serve:~$ sudo systemctl daemon-reload
sysadmin@soft-serve:~$ sudo systemctl start mongod
sysadmin@soft-serve:~$ sudo systemctl status mongod
× mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-08-11 15:33:50 UTC; 14s ago
       Docs: https://docs.mongodb.org/manual
    Process: 88421 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=100)
   Main PID: 88421 (code=exited, status=100)
        CPU: 39ms

Aug 11 15:33:50 soft-serve systemd[1]: Started MongoDB Database Server.
Aug 11 15:33:50 soft-serve systemd[1]: mongod.service: Main process exited, code=exited, status=100/n/a
Aug 11 15:33:50 soft-serve systemd[1]: mongod.service: Failed with result 'exit-code'.

Log:

{"t":{"$date":"2023-08-11T15:33:50.795+00:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"-","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2023-08-11T15:33:50.797+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingIn>
{"t":{"$date":"2023-08-11T15:33:50.797+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-08-11T15:33:50.797+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueu>
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrati>
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMig>
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","namespace":"config.tenantSplitDonors">
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":88421,"port":27017,"dbPath":"/data/var/lib/mongodb","architecture":"64-bit","host":">
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.8","gitVersion":"3d84c0dd4e5d99be0d69003652313e7eaf4cdd74","openSSLV>
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
{"t":{"$date":"2023-08-11T15:33:50.805+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017>
{"t":{"$date":"2023-08-11T15:33:50.806+00:00"},"s":"E",  "c":"CONTROL",  "id":20557,   "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"Location28596: Unable to determine status of lock file i>
{"t":{"$date":"2023-08-11T15:33:50.806+00:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":15000}}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"REPL",     "id":4794602, "ctx":"initandlisten","msg":"Attempting to enter quiesce mode"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"-",        "id":6371601, "ctx":"initandlisten","msg":"Shutting down the FLE Crud thread pool"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"initandlisten","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"MigrationUtil-TaskExecutor","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"COMMAND",  "id":4784923, "ctx":"initandlisten","msg":"Shutting down the ServiceEntryPoint"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784928, "ctx":"initandlisten","msg":"Shutting down the TTL monitor"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":6278511, "ctx":"initandlisten","msg":"Shutting down the Change Stream Expired Pre-images Remover"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2023-08-11T15:33:50.807+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}

Would you consider downgrading, if in this situation yourself?.. As suggested in this post?

MONGODB_CONFIG_OVERRIDE_NOFORK after upgrade to 5.18 Installation & Upgrades

Hi, just updated my test server (Ubuntu 20.04) from a previous 5.x release to 5.18 and I’m now getting this error when restarting: Environment variable MONGODB_CONFIG_OVERRIDE_NOFORK == 1, overriding “processManagement.fork” to false Mongod is configured to run as a single-node replicaset on this server, more details here: storage: dbPath: /var/lib/mongodb journal: enabled: true oplogMinRetentionHours: 24 systemLog: destination: file logAppend: true logRotate: reopen path…

Thanks for your support

Sam

Fabio_Ramohitaj · 2023-08-11T16:34:53.225Z

Hi @sam_ames,
Is selinux disabled?
The error now Is changed:

sam_ames:

{"error":"Location28596: Unable to determine status of lock file

In the extreme case, I would attempt to remove the files under the path /data/var/lib/mongodb (if you haven’ t collection populated with document in your istance) because seems corrupted the lock file or as you’ve suggested, try a downgrade or update.

Regards

sam_ames · 2023-08-11T18:23:15.478Z

It is a difficult install, I don’t even know what is se Linux is to be perfectly honest

Thanks,
Sam

I will try your suggestions on Monday morning, many thanks

sam_ames · 2023-08-11T18:23:56.385Z

Default install of Ubuntu **

Fabio_Ramohitaj · 2023-08-11T19:30:31.680Z

Hi @sam_ames,
cat /etc/selinux/config
Let me know how it will go then!

Regards

sam_ames · 2023-08-13T10:14:09.050Z


sysadmin@soft-serve:~$ sudo cat /etc/selinux/config
[sudo] password for sysadmin: 
cat: /etc/selinux/config: No such file or directory

It seems like your suspicions were correct.

Please suggest how I can resolve this. I did not delete selinux and am not sure why it’s missing.

Thanks for the support,

Sam

Fabio_Ramohitaj · 2023-08-13T12:27:33.937Z

Hi @sam_ames,
Another way to get selinux status Is toselinux status with the command
getenforce or cat /etc/sysconfig/selinux.
If the getenforce command results in output of the type enforcing or permissive, use the following command:
setenforce 0.
But as suggested in a previous post, i think is better to clean your data directory or do and update for resolve the problem
Regards

sam_ames · 2023-08-14T08:37:59.903Z

Please see the following output.

sysadmin@soft-serve:~$ cat /etc/sysconfig/selinux
cat: /etc/sysconfig/selinux: No such file or directory
sysadmin@soft-serve:~$ getenforce
Command 'getenforce' not found, but can be installed with:
sudo apt install selinux-utils
sysadmin@soft-serve:~$ sudo getenforce
[sudo] password for sysadmin: 
sudo: getenforce: command not found
sysadmin@soft-serve:~$

Do I need to install selinux tools?

Thanks,
Sam

Fabio_Ramohitaj · 2023-08-14T08:40:29.479Z

Hi @sam_ames,
In theory no, it should be present by default in the os.
Try the solutions discussed last time and let me know if they work

Regards

sam_ames · 2023-08-14T09:23:16.900Z

I can’t swipe the data volume because it is part of a raid partition and it was very complicated to set up.

However, I can delete everything in the directory that I have personally added. Is this worth doing?

You previously mentioned a downgrade, which version should I consider? I think I am currently using the latest release of version six. The post I shared, which mentioned downgrading was quite an old one so I assume the recommended version is no no longer the optimal solution.

Thank you,
Sam

sam_ames · 2023-08-14T11:31:40.014Z

All I’ve done is done is changed the directory. I’m amazed that this is so complicated…

Thanks for supporting me, Fabio