Experiencing issues with parallel mongosh executions

Keerthi_Jayaprakash · 2024-11-11T04:10:15.017Z

As part of ETL process our shell scripts are using mongosh to execute queries and retrieve data from MongoDB. When multiple processes are attempting to use mongosh in parallel, we randomly observe Error: ENOENT: no such file or directory, lstat '~/.mongodb/mongosh/xxxxxxxxxxxxx_log' and process gets terminated.

It appears mongosh can only store 100 log files in ~/.mongodb/mongosh directory and the rest gets purged. When mongosh attempts to reuse an old / existing session_id (which I assume is controlled by Mongo server) and if log file with that session_id is already purged, it throws an error. We couldn’t find a way to either disable logs generated by mongosh or increase the number of log files mongosh can hold.

Has anyone experienced a similar situation? If there is any workaround or an alternate solution to try out, please do advise.

Thanks.

Gaurab_Aryal · 2024-11-11T20:11:40.921Z

Hi @Keerthi_Jayaprakash – thank you for your feedback. We expect to have the following improvements for Shell logging by the end of Jan 2025:

Ability to configure the # of files in the log directory (i.e you can change the existing 100 files limit to a higher number)
Ability to configure storage limit for logs at the directory level

Additionally the error you are seeing is likely due to the parallel processes trying to remove the same files once the limit of 100 logs exceed. We will fix this bug as well. Let us know if this is an immediate blocker for you.

Keerthi_Jayaprakash · 2024-11-14T19:54:26.889Z

Hi Gaurab,

Thanks for your response.

We had parallelized certain processes so we can complete our entire pipeline execution in a small window. We haven’t experienced this issue when using mongo cli. It appears we have to wait until new version of mongosh is released.

Even if the log file count / storage limit is made configurable, would we still have challenges in determining the exact configuration value… as mongosh attempts to reuse existing session_id for a period of time and the respective log file getting purged in background?

Is session_id created from client side or created & managed in server? If there is a documentation explaining the process of session_id management, please do share for reference.

Thanks.

Keerthi_Jayaprakash · 2025-02-21T08:39:30.965Z

Hi Gaurab,

Was the improvements which you had listed been released or yet to be?

I reviewed the release notes of mongosh, but I didn’t find anything related. Could you please advise?

Thanks.