We are encountering critical errors preventing MongoDB from starting due to WiredTiger version mismatches and journal format issues. Below is a detailed breakdown of the issue and the steps we have taken so far.
Environment Details:
- MongoDB Version: 3.6.3 (initial), 4.0 (attempted upgrade)
- Storage Engine: WiredTiger
- Deployment: MongoDB running on Docker, alongside a system-installed MongoDB instance (MongoDB was installed but not actively used; previously, MongoDB was in use before dockerization)
- OS: Debian
Issue Breakdown:
- Initial Error:
IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
- MongoDB could not write to /data/mongodb due to permission issues.
- Directory ownership was set to UID 999 (unknown user) and group docker.
Actions Taken:
- We modified permissions to
mongodb:mongodb:
sudo chown mongodb:mongodb /data/mongodb -R
- Despite this, MongoDB still failed to start, returning the following error:
Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error
- Version Mismatch Error:
txn-recover: unsupported WiredTiger file version: this build only supports versions up to 2, and the file is version 5: WT_ERROR: non-specific WiredTiger error
- MongoDB 3.6.3 only supports WiredTiger file versions up to 2.
- WiredTiger files were detected as version 5.
- This suggests that at some point, the data files were modified by a newer MongoDB version (4.x or 5.x).
- Journal Format Error (During Upgrade to MongoDB 4.0):
An unsupported journal format detected - If you are trying to rollback from version 4.0 to 3.6, please re-start a 4.0 binary and cleanly shut it down so that the journal format will be downgraded.
- Even after upgrading to MongoDB 4.0, the data files were still incompatible, indicating further issues with the journal format.
Current Situation:
- We attempted to delete the journal directory, but restarting MongoDB (3.6 or 4.0) still results in version incompatibility errors.
- Despite trying various MongoDB versions (3.6, 4.0), none successfully load the data.
Root Cause Analysis:
- WiredTiger metadata and journal files seem to have been corrupted or modified by newer MongoDB versions.
- The mismatch between MongoDB binaries and existing data files is causing the failure.
Questions for the Community:
- Has anyone successfully downgraded WiredTiger metadata or resolved a similar version mismatch without data loss?
- Are there any recommended steps to recover data from WiredTiger files created by newer MongoDB versions?
- Is there a safe way to rebuild or migrate data from existing files to a new MongoDB instance without full data loss?