Cloud Native Enterprise Data Management with Tardigrade for MongoDB

MongoDB and Tardigrade are excited to collaborate and demonstrate managing data backups and migrations across multiple computing and storage platforms with enterprise security and SLAs. Tardigrade, the new S3-compliant storage solution from Storj Labs, can easily be configured to store all your database backups and snapshots from MongoDB Ops Manager, the management platform for enterprise MongoDB deployments. In this post, we’ll give an overview of how all this works in cloud-native environments. For this, we’ll install the entire MongoDB Enterprise stack into a Kubernetes cluster along with a small gateway app from Tardigrade and we’re ready to go!

Most companies have multiple engineering teams supporting multiple apps. These apps need data services and specifically require dynamic provisioning of new environments along with integrated platform management in order to continuously support their businesses. Each application, then, in turn, has its own requirements for security, high-availability and disaster recovery scenarios. We can combine tech from MongoDB and Tardigrade to support all these variable requirements.

The best part of Tardigrade is the service includes features by default that would otherwise cost organizations a large premium. For example, all data stored on Tardigrade is end-to-end encrypted by default, meaning you don’t have to do any special configuration or run any special software to keep your database backups and snapshots safe, private, and out of the hands of would-be hackers.

Every file uploaded to Tardigrade also has cross-geography redundancy by default, which is similar to a multi-region cloud backup architecture. This means your cloud storage backups will always be accessible, and a single outage won’t cripple your operations like it would in a centralized cloud storage architecture. Tardigrade backups are available everywhere, and with hot availability, developers can quickly rehydrate their MongoDB instance from anywhere in the world.

In addition to being half the cost of centralized cloud storage services, Tardigrade is also more affordable should you need to access your data. Egress bandwidth is half the price of Google Cloud, Amazon S3, and Azure and even more affordable than centralized backup architectures, which generally charge a hefty premium should you need to access your data.

MongoDB Enterprise supports a variety of cloud-native deployment options. This gives you and your apps access to locally deployed MongoDB clusters alongside direct access to MongoDB clusters running in the MongoDB Atlas cloud. Enterprises can now offer controlled access to advanced MongoDB data platform features directly to developers and CI/CD pipelines. Figure 1 depicts the overall architecture.

Demonstration

To see how all this works, we’ll spin up a deployment on GKE with everything. With this setup, we can create MongoDB clusters, managed by MongoDB Ops Manager. The entire deployment automates the configuration of an S3-compliant bucket to store backups.

Key Technologies

  • MongoDB Enterprise Kubernetes Operator
  • MongoDB Enterprise Ops Manager

  • S3-gateway - Tardigrade
  • Kubernetes

This example presents a v3 Helm chart which installs a complete MongoDB Enterprise database-as-a-service platform. There is a subchart for installing the Tardigrade S3-gateway to store backups and snapshots along. There are a number of components in this deployment. The workloads view in GCP shows the main ones:

This entire setup and more can be found in this repository https://github.com/jasonmimick/total-cluster. Note the special Helm chart in total-cluster/addons/tardigrade-gateway to set up communication between our private Kubernetes cluster and the Tardigrade decentralized cloud storage service.

Your first step is to procure a Kubernetes cluster. A couple of examples can be found in the total-cluster/quickstarts folder, but most any cluster should work. This full example will require 5 to 7 worker nodes with 2 nodes having at least 8 GB of ram.

Next, set up your Tardigrade Gateway. Check out the details for that over in the gateway chart in total-cluster. Installing the chart (Note: requires Helm V3),

➜  addons git:(master) ✗ helm install s3-gateway tardigade-gateway 
NAME: s3-gateway
LAST DEPLOYED: Sat May 30 07:52:43 2020
NAMESPACE: total-cluster
STATUS: deployed
REVISION: 1
TEST SUITE: None

Here, we can inspect the cluster and see our local gateway running. This service provides a local secure connection for MongoDB Ops Manager to write database backups.

➜  addons git:(master) ✗ kubectl get all --selector='app=s3-gateway-tardigrade-gateway'
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/s3-gateway-tardigrade-gateway-68fbf4b4d7-4qbvt   1/1     Running   0          104s

NAME                                        TYPE        CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/s3-gateway-tardigrade-gateway-svc   ClusterIP   10.43.65.192 <none> 7777/TCP   104s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/s3-gateway-tardigrade-gateway   1/1     1            1           104s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/s3-gateway-tardigrade-gateway-68fbf4b4d7   1         1         1       104s

Once your gateway is set up and running, install the entire MongoDB data platform with the total-cluster chart.

git clone https://github.com/jasonmimick/total-cluster
helm install mongodb total-cluster

The entire chart takes between 5 to approx 10 mins to finish installing. This is because it’s installing quite a few components.

Ops Manager needs to be configured to use the gateway. You can connect to your Ops Manager with a basic port-forward command like this,

kubectl port-forward mongodb-ops-manager-0 8080:8080

Connect your browser to localhost:8080 and use admin@mongodb.com and MongoDB12345% to login.

Note: These credentials are stored in a secret, called mongodb-ops-manager-secret. You can update this in the chart. Also note the “mongodb” in the secret name comes directly from the Helm release name.

You should see your MongoDB cluster once connected:

Setting up the S3 Blockstore pointing to local gateway

You can configure backups for your app databases following this guide: Backup Overview.

Screen shot of backup taking place.

Finally, you can see the backup data blocks stored in the Tardigrade bucket. You can even use the local tardigrade-gateway with command s3 command line tools:

$kubectl port-forward tardigrade-gateway-tardigrade-gateway-75cfdb889-nz2n4 7777:7777 &
$export AWS_ACCESS_KEY_ID=XXX
$export AWS_SECRET_ACCESS_KEY=XXX 
$aws --endpoint-url http://localhost:7777 s3 ls test

The Ops Manager extensive API can be used to automate backup restoration along with many devops tasks.

Conclusion

We look forward to seeing what MongoDB users do with Tardigrade. We regularly hear from users who would like cloud backup services that provide end-to-end encryption, multi-region redundancy, as well as lower and more predictable costs compared to centralized services. While relatively new, Tardigrade delivers on all of these needs, while still offering the same enterprise service level agreements users absolutely require.

Learn more

← Previous

Peek at your MongoDB Clusters like a Pro with Keyhole: Part 3

Performance evaluation and tuning is an important part of application development life cycle. Keyhole is an open source project tool for MongoDB.

June 11, 2020
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024