19 / 19
Feb 3

Hi there,

I have been using MongoDB for over ten years, but today I am facing a problem and I don’t know how to best deal with.
So I have a large production database that started out with MongoDB 2.x (not sure exactly which, perhaps 2.4), and which has been updated all the way to 4.2 and very recently to 5.0. I thought everything was ok until I discovered, today, that mongodump no longers perform my daily backups.

Failure message is:

mongodump --oplog -u admin -p <my password here> --authenticationDatabase admin --out /data/backup/ 2023-03-27T08:11:27.198+0200 Failed: error creating intents to dump: error creating intents for database admin: error counting admin.system.new_users: (Unauthorized) not authorized on admin to execute command { count: "system.new_users", lsid: { id: UUID("11218ac0-f185-4035-ac82-e06b31273527") }, $clusterTime: { clusterTime: Timestamp(1679897486, 4), signature: { hash: BinData(0, 83B632FBABBB17FECFA209B38BD559DD413C8E97), keyId: 7158403034257555467 } }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }

After some googling/reading it seems that this admin.system.new_users collection was (automatically) created during upgrade to 2.6, but never cleaned up. Its content is obsolete and could be safely deleted, and that would fix the mongodump issue, except that, for some reason, I can’t.

Every attempt to delete it fails with “can’t drop system collection” (probably because it is in the system namespace), and it can’t be renamed either, any attempt to do so ends with “Invalid system namespace”.

Looking at mongodb source code, there seem to be no easy way to drop that old unwanted collection which prevents me from backing up my database.

Actually I managed to get my backups working again by adding a specific role to my admin user privileges: [ { resource : { "db" : "admin", "collection" : "system.new_users" }, actions: [ "find" ] }], but this isn’t fully satisfying.
Of course a full re-creation of the database without the offending collection would work, but that would result in an unacceptable downtime for my customers.

So my question: is there a way to have mongodb drop that collection, bypassing the system namespace checks ? It seems that crafting a specific oplog entry could actually do that, but I not very comfortable with playing with such dark magic on a production database!

Any idea ?
Thanks

Thanks for your reply, but unfortunately it doesn’t help.

Without permission, the result is:

db.system.new_users.drop() uncaught exception: Error: drop failed: { "ok" : 0, "errmsg" : "not authorized on admin to execute command { drop: \"system.new_users\", lsid: { id: UUID(\"39647226-1638-45dc-ba17-0edffeab4f2c\") }, $clusterTime: { clusterTime: Timestamp(1679982428, 1), signature: { hash: BinData(0, 29228B39097FD3021D97D56D945171720FEDF25C), keyId: 7158403034257555467 } }, $db: \"admin\" }", "code" : 13, "codeName" : "Unauthorized", "operationTime" : Timestamp(1679982429, 5) }

With the appropriate permission, it goes further but still won’t do the job:

pc:PRIMARY> db.updateRole("cleanup",{privileges: [ { resource : { "db" : "admin", "collection" : "system.new_users" }, actions: [ "find", "dropCollection" ] }], roles: [ "root" ]}) pc:PRIMARY> db.system.new_users.drop() uncaught exception: Error: drop failed: { "ok" : 0, "errmsg" : "can't drop system collection admin.system.new_users", "code" : 20, "codeName" : "IllegalOperation", "operationTime" : Timestamp(1679982508, 5) }

That would have been too easy…

Unfortunately yes, the collection can’t be renamed either:

pc:PRIMARY> db.updateRole("cleanup",{privileges: [ { resource : { anyResource: true }, actions: [ "renameCollectionSameDB", "find", "insert", "dropCollection", "anyAction" ] }], roles: [ "root" ]}) pc:PRIMARY> db.system.new_users.renameCollection("oldusers") { "ok" : 0, "errmsg" : "error with source namespace: Invalid system namespace: admin.system.new_users", "code" : 20, "codeName" : "IllegalOperation", "operationTime" : Timestamp(1679993389, 10) }

After some hesitation, I finally tried… But it doesn’t work either, still the same ‘IllegalOperation’ result. I feel like it cannot be solved by permissions at all.

I tested this in test db
Created a collection system.xyz when I drop it says illegal operation
Was able to rename it and drop
That jira ticket user also says he can drop the collection after rename
Can you show output of
db
Show collections when you tried rename

@Ramachandra_Tummala,

Here is the list of collection in that admin database:

pc:PRIMARY> show collections system.backup_users system.keys system.new_users system.roles system.users system.version

Not sure why the rename method doesn’t work for me, it could because it was ‘fixed’ in the mongodb version I am using (5.0.14), or because or doesn’t work with ‘admin’ database.

1 year later

Hello,
By any chance someone found the way how to drop this admin.system.new_users collection ?
Looks like is invincible in version 5.0 an higher :frowning:
Br,
Michal

Hi Ramachandra,
In simply words I have exactly same issue as Nicolase here, I went through the all the steps. Trying to play with privileges. What works perfectly fine on version 4.4

  • rename
  • drop
    Is now forbidden ( currently I’m on 6.0)
    Regards to you answer, I don’t have shared cluster so login into config server is not an option here. For the error the best thing that I managed is :
    MongoServerError: error with source namespace: Invalid system namespace: config.system.new_users
    Thanks :slight_smile:

Yeah I tried that too, This is my current user:
“user” : “clean_user”,
“db” : “admin”,
“roles” : [
{
“role” : “root”,
“db” : “admin”
},
{
“role” : “restore”,
“db” : “admin”
},
{
“role” : “readWrite”,
“db” : “admin”
},
{
“role” : “cleanup”,
“db” : “admin”
},
{
“role” : “__system”,
“db” : “admin”
}

I’m still not able rename or drop.

Just in case you wonder, I didn’t manage to drop it either and finally gave up. I had a look at the source code of MongoDB and it seemed there was just no way to drop this collection except maybe by crafting an oplog entry and restoring it. But it is way too dangerous for me.

7 months later

@Michal_Majkowski @Nicolas_Bouquet Would like to check if you guys got it working ? I have upgraded my mongo cluster to 5.0 and now my mongo dump commands are failing over system.new_users collection. Has anyone found way to delete this collection ? Thanks in Advance.

@sandip_Divekar,
I didn’t try to go further on this issue since the last time I wrote here.
The change in permissions I mention in my original post allowed me to backup me database though, and in the event those backups would be of some use, I suppose I’ll just have to skip reimporting the problematic table.

@Nicolas_Bouquet For now, I am taking backup of individual databases instead full dump and skipping problematic collections using exclude collection option in mongodump.