- Administer Ops Manager >
- Manage Snapshot Storage >
- Manage S3 Snapshot Storage
Manage S3 Snapshot Storage¶
On this page
Ops Manager can back up MongoDB databases as snapshots to one or more of the following storage options:
- Another MongoDB database, called a blockstore,
- As files stored on a local or network-attached file system, and/or
- An AWS S3 bucket.
This tutorial covers backing up your MongoDB databases as snapshots stored in S3 and S3-compatible buckets. Ops Manager stores the metadata for S3 snapshot stores in a MongoDB database.
Note
You might have issues that require you to use more than one snapshot store. These issues could include needing more capacity, localizing data, or meeting privacy regulations.
To learn how to assign snapshot stores to different data centers, see Assign Snapshot Stores to Specific Data Centers.
Considerations¶
Requires a Dedicated Bucket¶
Ops Manager must be the only manager on the S3 bucket that you use for snapshots. You also need to configure the S3 bucket to avoid using features that Ops Manager does not support.
When configuring the S3 bucket:
- Do not create subfolders in the S3 buckets that you use with Ops Manager. Ops Manager only supports using entire S3 buckets.
- Disable AWS S3 bucket versioning. Versioning is not supported in Ops Manager for the S3 buckets used for snapshots.
- Do not create AWS S3 lifecycle rules. Lifecycle rules that expire or transition current versions of Ops Manager snapshot objects to archives results in incomplete snapshots that you can’t use to restore the configuration.
Can’t Move the S3 Snapshot Store¶
After you create an S3 snapshot store, you cannot move it to another S3 bucket. If you need to use a different S3 bucket to host your S3 snapshot store, you must create a new S3 snapshot store in that S3 bucket.
Supports the Storage API¶
MongoDB supports endpoints for:
- AWS S3 API
- IBM Cloud Object Storage API
- Dell EMC Elastic Cloud Storage API
IBM and Dell EMC support a subset of the full AWS S3 API.
You can use other S3-compatible endpoints. Ops Manager attempts to validate these endpoints when you save the S3 snapshot store setup. If validation passes, Ops Manager saves the configuration. If validation fails, Ops Manager displays an error and doesn’t save the configuration.
Prerequisites¶
Metadata Storage Prerequisites¶
- Deploy the dedicated MongoDB instance(s) to serve the S3 snapshot store metadata and Oplog Store. Serve these instances on separate hosts from the Ops Manager host and the application database to avoid performance and backup issues. Attach one or more storage volumes with enough capacity to store the databases these instances manage.
- Ensure the host serving the Ops Manager Backup Daemon service has enough capacity to store the head database.
- Secure the instance that stores your
S3 snapshot store metadata database
using authentication and
TLS. S3 snapshot store metadata databases
support
all authentication mechanisms.
AWS S3 Storage Prerequisites¶
Verify that you have an IAM user on AWS.
Create your own AWS access keys for your IAM user. This allows you to create S3 buckets and store snapshot files in them. MongoDB does not create or issue AWS access keys.
Create your own S3 bucket to store your S3 snapshot store snapshots.
Note
The IAM user for which you created the AWS access keys must own the AWS S3 Bucket.
(Optional) If you serve your Ops Manager instance on AWS EC2, create an IAM Role to handle authorization.
This role needs:
- AWS service as the trusted entity.
- EC2 as the use case.
- Permissions to read and write access to your S3 bucket.
To learn more, see:
IBM Cloud Object Storage Prerequisites¶
- Create an Access Key and Secret Key using IBM credential tools.
- Create your own S3-compatible bucket.
Dell EMC Elastic Cloud Storage Prerequisites¶
- Create an Access Key and Secret Key from your ECS User ID.
- Create your own S3-compatible bucket.
Other S3-Compatible Storage¶
Other S3-compatible endpoints can be used. Ops Manager attempts to validate these endpoints when you save the configuration. If validation passes, the configuration, Ops Manager saves it. If validation fails, Ops Manager displays an error and doesn’t save the configuration.
Procedures¶
The format of the Username and Password depend upon the authentication mechanism. Select one of the following tabs:
- Username and Password
- X.509
- Kerberos
- LDAP
Add One S3 Snapshot Store¶
Click Create New S3 Blockstore.¶
Provide the S3 blockstore details.¶
| Field | Necessity | Contents | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Name | Required | Type the label for the S3 snapshot store. | |||||||||
| S3 Bucket Name | Required | Type the name of the S3 bucket where you want to host the the S3 snapshot store. | |||||||||
| Region Override | Conditional | Type the region where your S3 bucket resides. Use this field only if your S3-compatible store’s S3 Endpoint doesn’t support region scoping. Don’t provide a value for this field with AWS S3 buckets. |
|||||||||
| S3 Endpoint | Required | Type the URL for this AWS S3 or S3-compatible bucket. What URL you write depends upon:
Example You created an S3 bucket called
|
|||||||||
| S3 Max Connections | Required | Type a positive integer indicating the maximum number of connections to this AWS S3 or S3-compatible bucket. | |||||||||
| Path Style Access | Optional | Select if you want your AWS S3 or S3-compatible bucket
to use a path-style URL endpoint
( To review the S3 bucket URL conventions, see the AWS S3 documentation |
|||||||||
| Server Side Encryption | Optional | Select to enable server-side encryption. Clear to disable server-side encryption. | |||||||||
| S3 Authorization Mode | Required | Select the method used to authorize access to the S3 bucket specified in S3 Bucket Name.
|
|||||||||
| Keys with Custom CA Bundle | Conditional | Click Choose file to add a custom Certificate Authority chain. This chain can validate against a self-signed certificate on the S3 bucket. Ops Manager displays this field when you set S3 Authorization Mode to Keys. |
|||||||||
| AWS Access Key | Conditional | Type your AWS Access Key ID. Ops Manager displays this field when you set S3 Authorization Mode to Keys. |
|||||||||
| AWS Secret Key | Conditional | Type your AWS Secret Access Key. Ops Manager displays this field when you set S3 Authorization Mode to Keys. |
|||||||||
| Datastore Type | Required | Select Standalone, Replica Set or Sharded Cluster. This MongoDB database stores the metadata for the blockstore. | |||||||||
| MongoDB Host List | Conditional | Type a comma-separated list of Example
Ops Manager displays this field when you set Datastore Type to Replica Set or Sharded Cluster. |
|||||||||
| MongoDB Hostname | Conditional | Type the hostname of the S3 snapshot store metadata database. Ops Manager displays this field when you set Datastore Type to Standalone. |
|||||||||
| MongoDB Port | Conditional | Type the port number of the S3 snapshot store metadata database. Ops Manager displays this field when you set Datastore Type to Standalone. |
|||||||||
| Username | Optional | If you set this value: Type the name of the user authorized to access the this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Type the RFC 2253-formatted subject from the client certificate of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Type the UPN of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the name of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP. |
|||||||||
| Password | Optional | If you set this value: Warning If you did not use the credentialstool to encrypt this password, it is stored as plaintext in the database. Type the password associated with the username that can access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Leave it blank. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Kerberos retrieves the password from its keytab file. Don’t type a password into this field. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the password of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP. |
|||||||||
| Connection Options | Optional | Type any additional configuration file options for the MongoDB instance. This field supports unescaped values only. For proper syntax, see Connection String URI Format in the MongoDB manual. |
|||||||||
| Encrypted Credentials | Optional | Select if the credentials for the database were encrypted using the credentialstool. The credentials include the Username, Password, AWS Access Key ID and AWS Secret Key. | |||||||||
| Use TLS/SSL | Optional | Select if the S3 snapshot store metadata database only accepts connection encrypted using TLS. Beyond this checkbox, to connect this S3 snapshot store using TLS, you must enable TLS on the S3 blockstore database. |
|||||||||
| New Assignment Enabled | Optional | Select if you want to enable this S3 snapshot store after creating it. This is selected by default so the S3 blockstore can be assigned backup jobs. If you clear this checkbox, the S3 snapshot store is created but you cannot assign backups to this S3 snapshot store. | |||||||||
| Disable Proxy Settings | Optional | Select if you want to disable proxying to this S3 snapshot store after
creating it. AWS S3 respects the HTTP_PROXY and the
HTTPS_PROXY environment variables. If you set either of
these environment variables, S3 still enables the proxy. |
Click Create.¶
Edit One Existing S3 Snapshot Store¶
Ops Manager lists S3 snapshot stores in a table on the Snapshot Storage page. Each row contains the settings for one S3 snapshot store.
Navigate to the Snapshot Storage page.¶
- Click the Admin link.
- Click the Backup tab.
- (Optional) If you have not previously set the head directory, set it in the Head Directory box.
- Click the Snapshot Storage page.
Go to the row for the S3 snapshot store you want to edit.¶
Update any values that need to be changed.¶
In the MongoDB Connection column, update any values that need to be changed in the following fields:
| Field | Necessity | Contents | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| S3 Bucket Name | Required | Type the name of the S3 bucket where you want to host the the S3 snapshot store. | ||||||||||||||||
| Region Override | Conditional | Type the region where your S3 bucket resides. Use this field only if your S3-compatible store’s S3 Endpoint doesn’t support region scoping. Don’t provide a value for this field with AWS S3 buckets. |
||||||||||||||||
| S3 Endpoint | Required | Type the URL for this AWS S3 or S3-compatible bucket. What URL you write depends upon:
Example You created an S3 bucket called
|
||||||||||||||||
| S3 Max Connections | Required | Type a positive integer indicating the maximum number of connections to this AWS S3 or S3-compatible bucket. | ||||||||||||||||
| Path Style Access | Optional | Click if you want your AWS S3 or S3-compatible bucket
to use a path-style URL endpoint
( To review the S3 bucket URL conventions, see the AWS S3 documentation |
||||||||||||||||
| Server Side Encryption | Optional | Click to enable server-side encryption. Clear to disable server-side encryption. | ||||||||||||||||
| S3 Authorization Mode | Required | Select the method used to authorize access to the S3 bucket specified in S3 Bucket Name.
|
||||||||||||||||
| Keys with Custom CA Bundle | Conditional | Click Choose file to add a custom Certificate Authority chain. This chain can validate against a self-signed certificate on the S3 bucket. Ops Manager displays this field when you set S3 Authorization Mode to Keys. |
||||||||||||||||
| AWS Access Key | Conditional | Type your AWS Access Key ID. Ops Manager displays this field when you set S3 Authorization Mode to Keys. |
||||||||||||||||
| AWS Secret Key | Conditional | Type your AWS Secret Access Key. Ops Manager displays this field when you set S3 Authorization Mode to Keys. Note Ops Manager doesn’t display the existing Secret Access Key. |
||||||||||||||||
| <hostname>:<port> | Required | Type in one or more hosts that comprise the S3 Snapshot
Store metadata database in the Important If these hosts are changed, the blockstore they host must have the same data as the original blockstore. Changing the host to a new blockstore results in data loss.
|
||||||||||||||||
| MongoDB Auth Username | Optional | If you set this value: Type the name of the user authorized to access the this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Type the RFC 2253-formatted subject from the client certificate of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Type the UPN of the user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the name of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP. |
||||||||||||||||
| MongoDB Auth Password | Optional | If you set this value: Type the password associated with the username that can access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring SCRAM authentication, see SCRAM. Leave it blank. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring x.509 authentication, see x.509. Kerberos retrieves the password from its keytab file. Don’t type a password into this field. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring Kerberos authentication, see Kerberos. Type the password of the LDAP user authorized to access this database. Note If your Ops Manager Application Database uses authentication or TLS, you must have connections configured to the application database. To learn more, see Configure the Connections to the Application Database. To learn more about configuring LDAP authentication, see LDAP. Warning If you did not use the credentialstool to encrypt this password, it is stored as plaintext in the database. Note Ops Manager doesn’t display the existing MongoDB Auth Password. |
||||||||||||||||
| Encrypted Credentials | Optional | Select if the credentials for the database were encrypted using the credentialstool. The credentials include the Username, Password, AWS Access Key ID and AWS Secret Key. | ||||||||||||||||
| Use TLS/SSL | Optional | Select if the blockstore database only accepts connection encrypted using TLS. Beyond this checkbox, to connect this S3 snapshot store using TLS, you must enable TLS on the S3 blockstore database. |
||||||||||||||||
| Connection Options | Optional | Type any additional configuration file options for the MongoDB instance. This field supports unescaped values only. For proper syntax, see Connection String URI Format in the MongoDB manual. |
||||||||||||||||
| Assignment Labels | Optional | Type a comma-separated list of labels to assign the S3 blockstores to specific projects. | ||||||||||||||||
| Load Factor | Optional | Type any positive integer that expresses how much backup work you want this snapshot store to perform compared to another snapshot store. Important If you have only one snapshot store, skip this setting. Backup work includes running backups, restoring snapshots or grooming blockstores. The term of backup work ratio assigned to a single snapshot store is called its Load Factor. By default, Ops Manager assigns each snapshot store a
Load Factor of As a snapshot store’s Load Factor increases, it
performs more backup work compared to another snapshot store.
If the Load Factor of snapshot store Example How to estimate Load Factor Consider a five-shard sharded cluster with the following backup storage configuration:
In this example, Set the Load Factors of Snapshot stores with greater compute or storage performance should be given a greater Load Factor:
Load Factor can be set to |
||||||||||||||||
| Write Concern | Required | Select your preferred Write Concern:
|
Select the checkbox in the Assignment Enabled column.¶
Select if you want to enable this S3 snapshot store after creating it. This is selected by default so the S3 snapshot store can be assigned backup jobs. If you clear this checkbox, the S3 Snapshot Store is created but you cannot assign backups to this S3 Snapshot Store.
Click Save.¶
Optional: Restart Ops Manager instances if needed.¶
If you change any connection string values or the Write Concern, restart all the Ops Manager instances including those running Backup Daemons.
Warning
Modifying the connection string values or the Write Concern for an existing blockstore requires all Ops Manager components, including those only running the Backup Daemon, to be restarted to apply those changes. Connection parameters include:
- <hostname>:<port>
- MongoDB Auth Username
- MongoDB Auth Password
- Encrypted Credentials
- Use TLS/SSL
- Connection Options
- Write Concern
If you change to another blockstore host, the data on the existing blockstore is not copied automatically to the other blockstore.
See also
For more details on the MongoDB connection string URI, see Connection String URI Format in the MongoDB Manual.
Delete One S3 Snapshot Store¶
Navigate to the Snapshot Storage page.¶
- Click the Admin link.
- Click the Backup tab.
- (Optional) If you have not previously set the head directory, set it in the Head Directory box.
- Click the Snapshot Storage page.