Docs Menu
Docs Home
/
MongoDB Cloud Manager
/
Troubleshoot

Automation

On this page

  • Automation Runs Only on 64-bit Architectures
  • Using Own Hardware
  • Networking
  • Automation Configuration
  • Sizing
  • Frequent Connection Timeouts
  • Deployments
  • MongoDB Agent Permissions
  • Remove Problematic MongoDB Hosts
  • Ensure TLS Certificates Contain a Subject Alternative Name
  • Store Configuration Files in Memory for Existing Clusters

Cloud Manager Automation allows you to deploy, configure, and manage MongoDB deployments with the Cloud Manager UI. Cloud Manager Automation relies on an MongoDB Agent, which must be installed on every server in the deployment. The MongoDB Agents periodically poll the Cloud Manager service to determine the current goal, and continually report their status to Cloud Manager.

Automation Runs Only on 64-bit Architectures

Cloud Manager provides only 64-bit downloads of the MongoDB Agent.

Using Own Hardware

  • If you deploy Automation manually, ensure that you have one MongoDB Agent on every server.

  • If you deploy the agent manually, you must create MongoDB's dbPath and the directory for the MongoDB binaries and ensure that the user running the agent owns these directories.

    If you install using the rpm package, the agent runs as the mongod user; if using the deb package, the agent runs as the mongodb user. If you install using the tar.gz archive file, you can run the agent as any user.

Networking

Connectivity to MongoDB Ports

All hosts must be able to allow communication between MongoDB ports. The default is 27017, but you can configure alternate port ranges in the Cloud Manager interface.

The MongoDB Agent must be able to connect to cloud.mongodb.com on port 443 (HTTPS). For more information on access to ports and IP addresses, see Security Overview.

Intra-cluster Connectivity Issue

When performing a rolling update, the MongoDB Agent tries to avoid downtime. It needs to collect state from other members of the cluster. A connectivity issue (between mongods and mongoses), such as hostname resolution or misconfigured firewall, may prevent the MongoDB Agent from determining the remote processes state and completing the change.

To ensure all members of the cluster can communicate with each other:

  1. For a non-sharded cluster:

    1. Log into each mongod.

    2. From that mongod host, log into all the other members of the replica set.

  2. For a sharded cluster:

    1. Log into each mongod.

    2. From that mongod host, log into all the other members of the shard.

    3. Log into each mongos.

    4. From that the mongos host:

      1. Log into the other mongos hosts.

      2. Log into all the other members of each shard.

Frequent Automation Connections

MongoDB Agent gathers the state from each member of the cluster every 10 seconds to ensure that the environment is in the expected state. As part of this assessment, MongoDB Agent creates a connection, checks certain files to determine state, and then closes the connection. These frequent, short-lived connections are part of MongoDB Agent's routine activity and should not impact performance.

Automation Configuration

After completing the automation configuration, ensure that the deployment plan satisfies the needs of your deployment. Check hostnames and ports before confirming the deployment.

Sizing

  • Ensure that you provision hosts with enough space to run MongoDB and support the requirements of your data set.

  • Ensure that you provision a sufficient number of hosts to run your deployment. Each mongod should run on its own host.

Frequent Connection Timeouts

The MongoDB Agent may frequently time out of connections for one or more of the following reasons:

  • Connection timeouts

  • High network latency

  • High server load

  • Large SSL keys

  • Insufficient CPU speed

By default, connections time out after 40 seconds. MongoDB recommends gradually increasing the value of the dialTimeoutSeconds MongoDB Agent configuration setting to prevent frequent premature connection timeouts. However, increasing this value also increases the time required to deploy future configuration changes. Experiment with small, incremental increases until you determine the optimum value for your deployment.

To learn more, see dialTimeoutSeconds in MongoDB Agent Connection Settings.

Deployments

A banner that displays We have detected a potential problem while deploying... appears when certain conditions apply. These are some examples.

Deployment Change Unable to Complete / Not Proceeding

If you have added or restarted a deployment and the deployment remains in the In Progress state for several minutes, the banner displays.

At this point, you have four options:

  1. Click View Diagnostics.

    The View Diagnostics modal displays any errors that may have happened during deployment.

  2. Click View Status.

    The Automation Status modal displays the deployment processes, when they last reported their deployment status, what task they are trying to complete, and the deployment status. You can filter the results in the following ways:

    • Type a hostname in the Filter processes search bar.

    • Select one or more process types from the Process Type dropdown.

    • Select one or more automation states from the Automation State dropdown.

    To learn more about the status of any of the individual processes, you can click the ellipsis icon and select either View Details or View Agent Logs.

    • View Details shows what the deployment plan for the process is and which stage of that plan the MongoDB Agent currently is.

    • View Agent Logs opens a new browser window with the Deployment > Agents > Agent Logs screen shown and the contents of the MongoDB Agent log displayed by default. Click the View menu to select a different agent log.

  3. Click View Agent Logs.

    A new browser window opens with the Deployment > Agents > Agent Logs screen shown and the contents of the MongoDB Agent log displayed by default. Click the View menu to select a different agent log.

  4. Click Allow Override & Edit Configuration.

    If you diagnose an error and need to correct the deployment configuration, follow the procedure to edit the deployment.

If you shut down the deployment and still cannot find a solution, remove the deployment from Cloud Manager.

MongoDB Agents Are Down or Not Communicating

If Automation Module of MongoDB Agent can't communicate with the Cloud Manager API endpoint or the MongoDB Server processes, Cloud Manager displays a warning banner in the Project. You can resolve this in one of two ways depending upon whether or not you expect the MongoDB Agents to be communicating:

MongoDB Agents need to communicate

If the MongoDB Agent(s) should be communicating with the Cloud Manager host or MongoDB instances, confirm the following for each MongoDB Agent:

  1. The Agent is up and running on each host.

  2. The Agent and the Cloud Manager API endpoint can communicate.

MongoDB Agents Do Not Need to Communicate

If Automation Module of MongoDB Agent(s) should be communicating with the Cloud Manager API endpoint or MongoDB Server processes, confirm the following for each automated MongoDB Server deployment:

  1. Click the Allow Editing & Override Current Configuration link in the warning banner.

  2. Remove all processes (mongod and mongos) running on the hosts that serve the unneeded MongoDB Agents.

MongoDB Agent Permissions

A permissions issue may prevent automation from completing a change. If View Status or View Diagnostics report an permissions-related error (such as open /data/db/mongod.lock: permission denied), ensure that the MongoDB Agent user owns and has read and write permissions to the dbpath and logpath files.

Remove Problematic MongoDB Hosts

You can use the console or the API to remove stale, broken, or problematic hosts from automation. This may include the circumstance when the MongoDB Agent can't be reached.

To remove a problematic host using the console:

1

In MongoDB Cloud Manager, go to the Deployment page for your project.

  1. If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.

  2. If it's not already displayed, select your desired project from the Projects menu in the navigation bar.

  3. If the Deployment page is not already displayed, click Deployment in the sidebar.

    The Deployment page displays.

2

Go to the Processes page.

Click the Processes tab for your deployment.

The Processes page displays.

3

Remove the host.

  1. Find your problematic host.

  2. Click , then Remove Server.

    Cloud Manager displays the Are you sure you want to remove this server> modal.

  3. Enter the provided hostname into the Enter the host name field an click Remove if you want to remove this server.

    Warning

    Cloud Manager removes all monitoring data for this host when you click Remove. Cloud Manager provides no confirmation or cancellation for this action.

    If you don't want to remove the server, click Cancel.

  4. Click Review & Deploy to review your changes.

    Cloud Manager displays your proposed changes.

    • If you are satisfied, click Confirm & Deploy.

      Cloud Manager removes all processes and agents at this time.

    • If you want to make further configuration changes, click Cancel.

To remove a problematic host using the API:

  1. Get the current automation config.

  2. Edit the automation configuration JSON file.

  3. Remove the stale node from processes and replica sets.

  4. Update the automation config file.

  5. Wait for a few minutes.

  6. In MongoDB Cloud Manager, go to the Deployment page for your project.

    1. If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.

    2. If it's not already displayed, select your desired project from the Projects menu in the navigation bar.

    3. If the Deployment page is not already displayed, click Deployment in the sidebar.

      The Deployment page displays.

  7. Click the Agents tab for your deployment.

    The Agents page displays.

  8. Confirm the host no longer appears on the list of Agents.

Ensure TLS Certificates Contain a Subject Alternative Name

Warning

The MongoDB Agent from version 11.11.0.7349-1 requires TLS certificates to include a value in the Subject Alternative Name field. Before upgrading to MongoDB Agent 11.11.0.7355 or later, make sure that all TLS certificates used in your MongoDB deployment contain a SAN. [1]

[1] MongoDB wrote the MongoDB Agent the Go language. Go 1.17 removed the ability to use X.509 CommonName field as a hostname when no SAN exists.When clients validate TLS certificates, the client checks the hostname or hostnames to which the certs apply from the values in the cert's SAN or Subject Distinguished Name (DN) fields. When creating TLS certificates, some people would use the Subject Common Name (CN) field to store the hostname. CNs have limitations that make them a poor choice to store hostnames. These limits include a 64-character maximum length and no support for Name Constraints. RFC 2818 deprecated using CN to store hostnames in May 2000. This RFC required clients to fall back to the CN if the certificate had no values in the SAN field. RFC 6125 removed the requirement in 2011.Go 1.15 disables adherence to RFC 2818 and uses the RFC 6125 practice of making CN optional. In practice, this change requires you to either add SAN values or enable the use of CNs.Go 1.17 removes the workaround to use the CN if no SAN exists.With the current version of the MongoDB Agent, you must use SAN.To learn more, see Fraser Tweedale's blog post on this topic

Store Configuration Files in Memory for Existing Clusters

If you use Cloud Manager version 4.2 or versions 4.4.0 - 4.4.6, you may encounter errors when setting enableLocalConfigurationServer to true and restarting your MongoDB Agent.

This issue only affects existing clusters where enableLocalConfigurationServer is set to true after the cluster is created. Setting this value before creating the cluster does not trigger this issue.

To safely change this setting for existing clusters:

  1. At the end of your MongoDB Agent configuration file, add:

    enableLocalConfigurationServer=true

  2. Shut down each process managed by the MongoDB Agent.

  3. Restart the MongoDB Agent by running the following command:

    service mongodb-mms-automation-agent restart

  4. Restart the MongoDB processes that you shut down.

  5. Verify that the automation-mongod.conf file has the __rest expansion directive.

For more information on storing configuration files in memory, see Configure How the MongoDB Agent Manages Config Files and Passwords.

Back

Authentication

Next

Monitoring