Why sharding cluster did not permit db.fsyncUnlock() command

Roman_Sereda1 · 2024-08-14T11:54:31.795Z

I created new sharded cluster v7.0.12 with one shard and config replica.

I try to implement file system snapshot backup but get stuck with lock unlock command at mongos context.

fsyncLock() work fine as expected.

[direct: mongos] admin> db.getSiblingDB("admin").fsyncLock()
{
  numFiles: 1,
  all: {
    raw: {
      'rs1/backup-poc-rs1-0-fsn1:27017,backup-poc-rs1-1-nbg1:27017,backup-poc-rs1-2-hel1:27017': {
        info: 'now locked against writes, use db.fsyncUnlock() to unlock',
        lockCount: Long('1'),
        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
        ok: 1
      },
      'cfg/backup-poc-cfg-0-fsn1:27017,backup-poc-cfg-1-nbg1:27017,backup-poc-cfg-2-hel1:27017': {
        info: 'now locked against writes, use db.fsyncUnlock() to unlock',
        lockCount: Long('1'),
        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
        ok: 1
      }
    }
  },
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1723557836, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('DIr2nxYdYZypvYVSU6P4ULm+4dA=', 0),
      keyId: Long('7402590685452304406')
    }
  },
  operationTime: Timestamp({ t: 1723557836, i: 1 })
}

But db.getSibling(“admin”).fsyncUnlock() did not permited.

[direct: mongos] admin> db.fsyncUnlock()
MongoServerError[Unauthorized]: not authorized on admin to execute command { fsyncUnlock: 1, lsid: { id: UUID("062761fa-34ae-4723-a93b-1bb9a97cbcca") }, $clusterTime: { clusterTime: Timestamp(1723557836, 1), signature: { hash: BinData(0, 0C8AF69F161D619CA9BD855253A3F850B9BEE1D0), keyId: 7402590685452304406 } }, $db: "admin" }

User that run both command have next roles
- “clusterAdmin”
- “dbAdminAnyDatabase”
- “root”
- “hostManager”
that should be enough according manual https://www.mongodb.com/docs/manual/reference/built-in-roles/#mongodb-authrole-hostManager

How can be solve this puzzle?

Best regards
Roman Sereda

Fabio_Ramohitaj · 2024-08-14T17:37:51.304Z

Hi @Roman_Sereda1 and welcome to the community!
curious, are you sure you have authenticated yourself?
With root privileges, you should be able to safely perform the operation.

Best Regards

Roman_Sereda1 · 2024-08-15T12:33:47.646Z

Yes. I check this first and set of real permission as second.

db.fsyncLock() command allowed in same user session

Fabio_Ramohitaj · 2024-08-15T15:12:04.975Z

Hi @Roman_Sereda1,
I just did a test and personally it works correctly for me:

[direct: mongos] test> db.getSiblingDB("admin").fsyncLock()
{
 numFiles: 1,
 all: {
   raw: {
     'myShard_0/replicadue.mongodb.int:27000,replicatre.mongodb.int:27000,replicauno.mongodb.int:27000': {
       info: 'now locked against writes, use db.fsyncUnlock() to unlock',
       lockCount: Long('1'),
       seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
       ok: 1
     },
     'myShard_1/replicadue.mongodb.int:27001,replicatre.mongodb.int:27001,replicauno.mongodb.int:27001': {
       info: 'now locked against writes, use db.fsyncUnlock() to unlock',
       lockCount: Long('1'),
       seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
       ok: 1
     },
     'configRS/replicadue.mongodb.int:27020,replicatre.mongodb.int:27020,replicauno.mongodb.int:27020': {
       info: 'now locked against writes, use db.fsyncUnlock() to unlock',
       lockCount: Long('1'),
       seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
       ok: 1
     }
   }
 },
 ok: 1,
 '$clusterTime': {
   clusterTime: Timestamp({ t: 1722613341, i: 7 }),
   signature: {
     hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
     keyId: Long('0')
   }
 },
 operationTime: Timestamp({ t: 1722613341, i: 7 })
}

[direct: mongos] test> db.getSiblingDB("admin").fsyncUnlock()
{
 raw: {
   'configRS/replicadue.mongodb.int:27020,replicatre.mongodb.int:27020,replicauno.mongodb.int:27020': { info: 'fsyncUnlock completed', lockCount: Long('0'), ok: 1 },
   'myShard_1/replicadue.mongodb.int:27001,replicatre.mongodb.int:27001,replicauno.mongodb.int:27001': { info: 'fsyncUnlock completed', lockCount: Long('0'), ok: 1 },
   'myShard_0/replicadue.mongodb.int:27000,replicatre.mongodb.int:27000,replicauno.mongodb.int:27000': { info: 'fsyncUnlock completed', lockCount: Long('0'), ok: 1 }
 },
 ok: 1,
 '$clusterTime': {
   clusterTime: Timestamp({ t: 1722613341, i: 16 }),
   signature: {
     hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
     keyId: Long('0')
   }
 },
 operationTime: Timestamp({ t: 1722613341, i: 16 })
}

Valentin_TRAEN · 2024-12-17T09:21:47.703Z

Same problem here. Do you have find any solution ?

Roman_Sereda1 · 2024-12-17T11:02:01.555Z

I did not catch root of problem. I avoid this operation and use network restriction to manage income session to prevent write operation at maintenance event time. Problem is reproducible at my environment.

Fabio_Venturi · 2025-05-07T09:26:26.062Z

Hello everyone, same problem here.
Running MongoDB 6.0.21 from TAR archive (on Debian 11 in case it matters),
fsyncLock() worked correctly via mongos, but I cannot unlock the db:

superadmin@admin || REMOTE => db.fsyncUnlock()
MongoServerError[Unauthorized]: not authorized on admin to execute command { fsyncUnlock: 1, lsid: { id: UUID("1e47fd83-9d0f-4085-a067-3ef9124df008") }, $clusterTime: { clusterTime: Timestamp(1746603456, 1), signature: { hash: BinData(0, 80792718E59049D81365EAC9C57D8A3F1AF64A66), keyId: 7501307811511402521 } }, $db: "admin" }

The user I’m using to issue the command has the current roles:

db.runCommand({ connectionStatus: 1 })
{
  authInfo: {
    authenticatedUsers: [ { user: 'superadmin', db: 'admin' } ],
    authenticatedUserRoles: [
      { role: 'clusterAdmin', db: 'admin' },
      { role: 'dbAdminAnyDatabase', db: 'admin' },
      { role: 'hostManager', db: 'admin' },
      { role: 'readWriteAnyDatabase', db: 'admin' },
      { role: 'root', db: 'admin' },
      { role: 'userAdminAnyDatabase', db: 'admin' }
    ]
  },
  ok: 1,
  .....
}

I’ve successfully unlocked the database by running db.fsyncUnlock() on the primary nodes of the 2 shards,
but according to MongoDB documentation db.fsyncUnlock() - Database Manual v8.0 - MongoDB Docs
it should be possible to lock and unlock connecting via mongos:
Starting in MongoDB 7.1 (also available starting in 7.0.2, 6.0.11, and 5.0.22) the db.fsyncLock() and db.fsyncUnlock() methods can run on mongos to lock and unlock a sharded cluster.

Is this a bug?

It would be nice to have the db.fsyncUnlock() properly working via mongos to ease the backup process.
Thank you for any hint,
_
Fabio

Fabio_Venturi · 2025-05-07T20:27:22.561Z

Hello Fabio, sorry to resurrect this thread,
which roles the user you used had? At the moment I have:

authenticatedUserRoles: [
      { role: 'clusterAdmin', db: 'admin' },
      { role: 'dbAdminAnyDatabase', db: 'admin' },
      { role: 'dbOwner', db: 'admin' },
      { role: 'hostManager', db: 'admin' },
      { role: 'readWriteAnyDatabase', db: 'admin' },
      { role: 'root', db: 'admin' },
      { role: 'userAdminAnyDatabase', db: 'admin' }

but I still get:

db.getSiblingDB("admin").fsyncUnlock()

MongoServerError[Unauthorized]: not authorized on admin to execute command { fsyncUnlock: 1, lsid: { id: UUID("f914c7ea-de33-4f81-ac62-763d02b979d3") }, $clusterTime: { clusterTime: Timestamp(1746605884, 1), signature: { hash: BinData(0, ADD7CB273715DF53828EC7D8D04CEE1E24F9B3C9), keyId: 7501307811511402521 } }, $db: "admin" }

Fabio_Ramohitaj · 2025-05-08T08:22:38.862Z

Hi @Fabio_Venturi,
When I did the test, I used a user with root role , so it’s quite strange what happens to you.

Best Regards

Fabio_Venturi · 2025-05-08T14:05:55.633Z

Hi @Fabio_Ramohitaj thank you very much for your reply.
It’s really weird indeed, I’ve the same problem with both mongodb 6.0.21 and 7.0.20 (haven’t tried 8 though).

Furthermore if I run this via pymongo:

client = MongoClient(MONGO_URI)
admin = client.admin
result = admin.command("fsync", lock=False, maxTimeMS=10000)

I got ok: 1.0

{'numFiles': 1, 'all': {'raw': {'shard0/mongodb-restore-shard01.nonprod:27018,mongodb-restore-shard02.nonprod:27018,mongodb-restore-shard03.nonprod:27018': {'numFiles': 1,
 'ok': 1.0}, 'config/mongodb-restore-config1.nonprod:27019,mongodb-restore-config2.nonprod:27019,mongodb-restore-config3.nonprod:27019': {'numFiles': 1, 'ok': 1.0}, 'shard
1/mongodb-restore-shard11.nonprod:27018,mongodb-restore-shard12.nonprod:27018,mongodb-restore-shard13.nonprod:27018': {'numFiles': 1, 'ok': 1.0}}}, 'ok': 1.0, '$clusterTim
e': {'clusterTime': Timestamp(1746702209, 2), 'signature': {'hash': b'n\xf4YF\x82\xdfJ\xd5]\x1a\xc0N\x1e\x99\x8f\xf9\x1a\x1b\xcaw', 'keyId': 7501702647854923798}}, 'operationTime': Timestamp(1746
702209, 2)}

but if I run the aggregation pipeline showed in the documentation

db.getSiblingDB("admin").aggregate( [
   {  $currentOp: { } },
   {  $facet: {
      "locked": [
         { $match: { $and: [
            { fsyncLock: { $exists: true } }
         ] } }],
       "unlocked": [
         { $match: { fsyncLock: { $exists: false } } }
       ]
   } },
   {  $project: {
      "fsyncLocked": { $gt: [ { $size: "$locked" }, 0 ] },
      "fsyncUnlocked": { $gt: [ { $size: "$unlocked" }, 0 ] }
   } }
 ] )

I still get that there is a lock

[ { fsyncLocked: true, fsyncUnlocked: false } ]

I start suspecting I did something wrong while creating the admin during cluster bootstrap, but i don’t know what. For now I’ll just connect to the replica set and lift the lock directly on the primary nodes.
Thank you again
_
Fabio