Slow drainage of collection `config.system.sessions` when performing `removeShard`

Ops and Admin Sharding
,
1

Details of the environment/setup:

  • Ubuntu 22.04
  • MongoDB 6.0
  • Sharded Cluster with 3 shards + 1 config server. (Note: each component has replication enabled, but no replicas as this is just a test by hand)
  • Cluster Components are hosted by LXD containers
  • Little to no data is written to the cluster, only testing things out.

Problem:

When I remove the shard that hosts the config.system.sessions collection, drainage takes about 20-25 minutes.

When I look at the logs for this shard I see "Slow query":

2023-10-25T11:49:21Z charmed-mongodb.mongod[4753]: {"t":{"$date":"2023-10-25T11:49:21.835+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn179","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"_shardsvrMoveRange":"config.system.sessions","toShard":"shard-one","min":{"_id":{"id":{"$uuid":"83400000-0000-0000-0000-000000000000"}}},"waitForDelete":false,"epoch":{"$oid":"6538e3c125ae3b07d28d5eaf"},"fromShard":"shard-three","maxChunkSizeBytes":200000,"forceJumbo":2,"secondaryThrottle":false,"writeConcern":{"w":1,"wtimeout":0},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1698234561,"i":1}},"signature":{"hash":{"$binary":{"base64":"BDUe5gGly3g4iPDSiuCwtTRJTLU=","subType":"0"}},"keyId":7293828730399490051}},"$configTime":{"$timestamp":{"t":1698234561,"i":1}},"$topologyTime":{"$timestamp":{"t":1698233824,"i":2}},"mayBypassWriteBlocking":false,"$db":"admin"},"numYields":0,"reslen":236,"locks":{},"writeConcern":{"w":1,"wtimeout":0,"provenance":"clientSupplied"},"remote":"10.18.246.22:60792","protocol":"op_msg","durationMillis":536}}

My Questions

  1. Why is this query slow?
  2. How can I make this query faster?
  3. Is this related to the shard key for this collection? AFAICT mongod/mongos sets this collection up. Is it okay to change the shard key?

Additional information

From the docs: config.system.sessions

There is 1024 chunks in this collection, and here is more useful info from sh.status() before removing:

    database: { _id: 'config', primary: 'config', partitioned: true },
    collections: {
      'config.system.sessions': {
        shardKey: { _id: 1 },
        unique: false,
        balancing: true,
        chunkMetadata: [ { shard: 'shard-two', nChunks: 1024 } ],
        chunks: [
          'too many chunks to print, use verbose if you want to force print'
        ],
        tags: []
      }

I have also logged into the machines hosting the shards/config servers and seen that on each:

  • that CPU consumption was low (tested with top)
  • connectivity with other components was fast (used iperf for this)
  • checked disk space and there was plenty

I have also logged into the shard hosting the collection and ran:

use config
db.system.sessions.stats()

and saw:

  sharded: false,

this seems wrong, but I am not sure.

I thought I would try re-sharding the collection, but when I tried to view the collection contents with config> db.system.sessions.find() I got:

MongoServerError: not authorized on config to execute command { find: "system.sessions", filter: {}, lsid: { id: UUID("2b7221a9-41ce-457b-a1e0-a1caf417b012") }, $clusterTime: { clusterTime: Timestamp(1698306641, 2), signature: { hash: BinData(0, 7DAE0335273F250F7F1469FD3167EBAA61253250), keyId: 7293828730399490051 } }, $db: "config" }
4

Hi @Mia_Altieri and welcome to MongoDB community forums!!

Removing a shard from a sharding deployment is expected to take time from minutes to hours even to days and also depends on how the data have been distributed in the chunks on these shards.

Removing the shard from the deployment is however not a simple process. If the balancer is enabled in the configuration, as soon as you enter the remove shard command, the balancer will try to balance all the chunks in the available shards.

This also depends on how the shard key has been selected to make the event distribution between the chunks.
Please refer to the documentation to understand how to select the shard key in more detail.

Based on the above case I tried to replicate in my local environment and test the scenario.

I have around 20000 documents like:

[direct: mongos] test> db.sample.findOne()
{ _id: ObjectId("6548e52848f65c4016a4cea6"), name: 'Millie Curtis' }
[direct: mongos] test>

sharded on the name field. Since the name is unique, all the documents have moved to the same shard while other shards are still empty.

One I remove the shard, I see the balancer trying to balance the chunks between the shards

Below is the status after around 10 mins into removing of the shard process as begun:

chunkMetadata: [
          { shard: 'shard01', nChunks: 614 },
          { shard: 'shard02', nChunks: 218 },
          { shard: 'shard03', nChunks: 192 }
        ],

Perhaps this is due to other processes running in the background, which might be causing the slowdown.

Is the removing of shard a regular process? Could you clarify how often is the removal of the shard done?
Changing the shard key might be a way to make the process fast as the chunks would be more evenly distributed among all the shards and only a few number of chunks needs to be balanced rather than balancing one chunk between all the remaining available chunk.

Finally, can you confirm the shard where you are running this command?

Regards
Aasawari