4 New MongoDB Features to Improve Security and Operations

Cara Heimbaugh
June 7, 2022

Data platforms are designed to remove operational complexity and enable developers to move and innovate faster. For applications that are critical to your users and your business, the data platform powering them must also be reliable, scalable, and global. Achieving that should take minimal work, both upfront and on an ongoing basis.

At MongoDB World 2022, we announced several new capabilities that further help organizations achieve operational excellence: Queryable Encryption, Cluster-to-Cluster Sync, Scheduled Archiving, MongoDB Atlas Operator for Kubernetes, and MongoDB Atlas Serverless.

With the introduction of Queryable Encryption, MongoDB will be the only database provider that allows customers to run expressive queries such as equality and range, prefix, suffix, substring and more on fully randomly encrypted data, just as they can do on unencrypted data.

This is a huge advantage for organizations that need to run expressive queries while also securing their data. Queryable Encryption reduces the heavy lifting involved when working with encrypted data, resulting in faster app development without undermining data protection or compliance with data privacy regulations.

Not every organization is fully — or may ever be fully — in the cloud. Many businesses also leverage hybrid or multi-cloud environments. Cluster-to-Cluster Sync enables continuous, uni-directional, real-time data synchronization of two MongoDB clusters in the same or different environments — public cloud, private cloud, on-premises, and at the edge. MongoDB now supports, for example, hybrid Atlas and Enterprise Advanced deployments, wherein a cluster’s data can be synced from on-prem to Atlas, or vice versa.

With Cluster-to-Cluster Sync, organizations have full control of the synchronization process. They can decide when to start, stop, pause, or resume your synchronization, or to reverse the direction of synchronization. And they can monitor the progress of the synchronization in real time.

This new capability will enable greater experimentation and innovation, increase organizational insights, and help developers find more efficient ways to work with data. Use cases that benefit from having the data of two MongoDB clusters fully synchronized include data migration, enhanced development lifecycles, dedicated analytics, audit compliance, and improving latency by moving data to the edge.

The MongoDB Atlas Operator for Kubernetes is the best way to use MongoDB with Kubernetes. With the Atlas Operator, developers can seamlessly integrate MongoDB Atlas into their Kubernetes deployment pipeline, controlling Atlas resources without leaving the Kubernetes control plane. They can also control Atlas projects, clusters, database users, backup policy, serverless instances, private network endpoints, and more. The operator is compatible with any certified Kubernetes distribution, including Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), Red Hat OpenShift, and dozens more.

We are enhancing the Online Archive feature of Atlas with two new features: Data expiration and scheduled archiving. With data expiration, you can define and automate for how long you need the data stored in the online archive before getting deleted. With the scheduled archiving feature, you can set rules about the time window of when you want the archive to run. This could be daily, weekly or monthly. You can also edit the archive rule and define when you want to archive your data and when you want it deleted from the archive.

One big trend in the developer world is removing operational overhead by moving to a managed database offering. This move away from day-to-day management and administration lets developers do what they do best — create. To this end, MongoDB has rolled out Atlas Serverless. With Atlas Serverless, server provisioning and management has been abstracted (hidden) from the customer or end-user of the service. This eliminates the cognitive load of sizing and scaling infrastructure to keep up with application demand. Instead of paying for idle resources, with Atlas Serverless, you pay for only what you use. By simplifying provisioning, Atlas Serverless helps organizations accelerate time to market and improve experiences for both developers and IT managers.

All of these new features have been designed to help organizations improve their operational excellence, ensuring security, consistency, and scale while alleviating repetitive operational tasks for developers and IT managers.

Learn more about MongoDB World 2022 announcements at mongodb.com/new and in these stories:

← Previous

Announcing Atlas Data Federation and Atlas Data Lake

>> Announcement: Some features mentioned below will be deprecated on Sep. 30, 2025. Learn more . Two years ago, we released the first iteration of Atlas Data Lake . Since then, we’ve helped customers combine data from various storage layers to feed downstream systems. But after years spent studying our customers’ experiences, we realized we hadn’t gone far enough. To truly unleash the genius in all our developers, we needed to add an economical cloud object storage solution with a rich MQL query experience to the world of Atlas. Today, we’re thrilled to announce that our new Atlas Data Federation and Atlas Data Lake offerings do just that. We now offer two complementary services, Atlas Data Federation (our existing query service formerly known as Atlas Data Lake) and our new and improved Atlas Data Lake (a fully managed analytic-oriented storage service). Together, these services (both in preview) provide flexible and versatile options for querying and transforming data across storage services, as well as a MongoDB-native analytic storage solution. With these tools, you can query across multiple clusters, move data into self managed cloud object storage for consumption by downstream services, query a workload-isolated inexpensive copy of cluster data, compare your cluster data across different points in time, and much, much more. In hearing from our customers about their experiences with Atlas Data Lake, we learned where they have struggled, as well as the features they’ve been looking for us to provide. With this in mind, we decided to shift the name of our current query federation service to Atlas Data Federation to better align with how customers see the service and are getting value. We’ve seen many customers benefit from the flexibility of a federated query engine service, including querying data across multiple clusters, databases, and collections, as well as exporting data to third-party systems. We also saw where our customers were struggling with data lakes. We heard them ask for a fully managed storage solution so they could achieve all of their analytic goals within Atlas. Specifically, customers wanted scalable storage that would provide high query performance at a low cost. Our new Data Lake provides a high-performance analytic object storage solution, allowing customers to query historical data with no additional formatting or maintenance work needed on their end. How it works Atlas Data Federation encompasses our existing Data Lake functionality with several new affordances. It continues to deliver the same power that it always has, with increased performance and efficiency. The new Atlas Data Lake will now allow you to create Data Lake pipelines (based on your Atlas Cluster backup schedules) and fields on which you can optimize queries. The service takes the following steps: On the selected schedule, a copy of your collection will be extracted from your Atlas backup with no impact to your cluster. During extraction, we build partition indexes based on the contents of your documents and the fields you’ve selected for optimization. These indexes allow your queries to be optimized by capturing the minimums and maximums (and other stats) of the records in each partition, letting you quickly find the relevant data for your queries. Finally, the underlying data lands in an analytic-oriented format inside of cloud object storage. This minimizes data scanned when you execute a query. Once a pipeline has run and a Data Lake dataset has been created, you can select it as a data source in our new Data Federation query experience. You can either set it as the source for a specific virtual collection in a Federated Database Instance or you can have your Federated Database Instance generate a collection name for each dataset that your pipeline has created. Amazingly, no part of this process will consume compute resources from your cluster — neither the export nor the querying of datasets. These datasets provide workload isolation and consistency for long-running analytic queries, a target for ETL jobs using the powerful $out to S3. This makes it easy to compare the state of your data over time. Advanced though this is, it’s only the beginning of the story. We’re committing to evolving the service, improving performance, adding more sources of data, and building new features. All of this will be based on the feedback you, the user, gives us. We can’t wait to see how you’ll use this powerful new tool and can’t wait to hear what you’d like to see next. Try Atlas Data Lake Today

June 7, 2022
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024