Rotate Keys for Self-Managed Sharded Clusters
Sharded cluster members can use keyfiles to authenticate each other as memers of the same deployment.
A keyfile can contain multiple keys and membership authentication is established if at least one key is common across members. This allows for rolling upgrade of the keys without downtime.
The following tutorial steps through the process to update, without any downtime, the key for a sharded cluster. [1]
Warning
The example keys in this tutorial are for illustrative purposes
only. Do NOT use for your deployement. Instead, generate a
keyfile using any method you choose (e.g. openssl rand -base64
756
, etc.).
Consider a sharded cluster where each member's keyfile contains the following key:
The following procedure updates the sharded cluster members to use a new key:
[1] | This tutorial is not applicable to the keyfile used for the MongoDB's encrypted storage engine local key management. That keyfile can only contain a single key. |
Before You Begin
Starting in MongoDB 8.0, you can use the
directShardOperations
role to perform maintenance operations
that require you to execute commands directly against a shard.
Warning
Running commands using the directShardOperations
role can cause
your cluster to stop working correctly and may cause data corruption.
Only use the directShardOperations
role for maintenance purposes
or under the guidance of MongoDB support. Once you are done
performing maintenance operations, stop using the
directShardOperations
role.
Procedure
1. Modify the Keyfile to Include Old and New Keys
Modify each member's keyfile to include both the old and new keys.
Warning
The example keys in this tutorial are for illustrative purposes
only. Do NOT use for your deployement. Instead, generate a
keyfile using any method you choose (e.g. openssl rand -base64
756
, etc.).
You can specify multiple key strings as a sequence of key strings (optionally enclosed in quotes):
1. Restart Each Member
Once all the keyfiles contain both the old and new keys, restart each member one at a time.
Config Servers
For each secondary of the config server replica set (CSRS),
connect mongosh
to the member and:
Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
For the primary, connect mongosh
to the member and
Use
rs.stepDown()
to step down the member:rs.stepDown() Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
Shard Replica Sets
For each secondary member of the shard replica sets, connect
mongosh
to the member and:
Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
For the primary of each shard replica set, connect
mongosh
to the member and
Use
rs.stepDown()
to step down the member:rs.stepDown() Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
mongos
Routers
For each mongos/router instance, connect
mongosh
to the mongos
instance and:
Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
Once all members have been restarted, the members now accept either the old or new key for membership authentication.
3. Update Keyfile Content to the New Key Only
Warning
The example keys in this tutorial are for illustrative purposes
only. Do NOT use for your deployement. Instead, generate a
keyfile using any method you choose (e.g. openssl rand -base64
756
, etc.).
Modify each member's keyfile to include only the new password.
4. Restart Each Member
Once all the keyfiles contain the new key only, restart each member one at a time.
Config Servers
For each secondary of the config server replica set (CSRS),
connect mongosh
to the member and:
Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
For the primary, connect mongosh
to the member and
Use
rs.stepDown()
to step down the member:rs.stepDown() Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
Shard Replica Sets
For each secondary member of the shard replica sets, connect
mongosh
to the member and:
Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
For the primary of each shard replica set, connect
mongosh
to the member and
Use
rs.stepDown()
to step down the member:rs.stepDown() Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
mongos
Routers
For each mongos/router instance, connect
mongosh
to the mongos
instance and:
Use the
db.shutdownServer()
method to shut down the member:use admin db.shutdownServer() Restart the member.
Once all members have been restarted, the members now accept only the new key for membership authentication.