Streamline, Simplify, Accelerate: New MongoDB Features Reduce Complexity

William To
June 7, 2022 | Updated: September 9, 2024

>> Announcement: Some features mentioned below will be deprecated on Sep. 30, 2025. Learn more.

At MongoDB World 2022, we announced several developer-centric features that provide more powerful analytics, streamline operations, and reduce complexity. In this post, we look at MongoDB Atlas Data Federation, MongoDB Atlas Search, MongoDB Atlas Device Sync and its Flexible Sync, and change streams.

As consumer expectations of the applications they use grow, developers must continue to create richer experiences. To do that, many are adding a variety of data systems and components to their architectures, including single-purpose NoSQL datastores, dedicated search engines, and analytics systems.

Piecing these disparate systems together adds complexity to workflows, schedules, and processes, however. For instance, one application could utilize a solution for database management, another solution for search functionality, and a third solution for mobile data sync. Even within an organization, teams often use different products to perform the same tasks, such as data analysis.

This way of building modern applications often causes significant problems, such as data silos and overly complex architectures. Additionally, developers are forced to spend extra time and effort to learn how each of these components functions, to ensure they work together, and to maintain them over the long term. It should not be the developer’s job to rationalize all these different technologies in order to build rich application experiences.

The developer data platform

For developers and their teams, cobbling together a data infrastructure from disparate components is inefficient and time-consuming. Providers have little incentive to ensure that their solutions can function alongside the products of their competitors. Further, internal documentation, which is key to demystifying the custom code and shortcuts in a bespoke architecture, might not be available or current, and organizational knowledge gets lost over time.

MongoDB Atlas, our developer data platform, was built to solve these issues. An ecosystem of intuitive, interlinked services, Atlas includes a full array of built-in data tools, all centered around the MongoDB Atlas database. Features are native to MongoDB, work with a common API, are designed for compatibility, and are intended to support any number of use cases or workloads, from transactional to operational, analytics to search, and anything in between.

Equally important, Atlas removes the hidden, manual work of running a sprawling architecture, from scaling infrastructure to building integrations between two or more products. With these rote tasks automated or cleared away, developers are free to focus on what they do best: build, iterate, and release new products.

MongoDB Atlas Data Federation

MongoDB Atlas Data Federation allows you to write a single query to work with data across multiple sources, such as your Amazon S3, Atlas Data Lake, and MongoDB Atlas clusters. Atlas Data Federation is not a separate repository of data, but a service to combine, enrich, and transform data across multiple sources, regardless of origin, and output to your preferred location.

With Atlas Data Federation, developers who want to aggregate data or federate queries do not need to use complex data pipelines or time-consuming transformations — a key advantage for those seeking to build real-time app features. Atlas Data Federation also makes it easier to quickly convert MongoDB data into columnar file formats, such as Parquet or CSV, so you can facilitate ingestion and processing by downstream teams that are using a variety of different analytics tools.

MongoDB Atlas Search

Rich, responsive search functionality has become table stakes for both consumer-facing and internal applications. But building high-quality search experiences isn’t always easy. Developers who use a third-party, bolt-on search engine to build search experiences have to deal with problems like the need to sync data between multiple systems; more operational overhead for scaling, securing, and provisioning; and using different query interfaces for database and search.

Built on the industry-leading Apache Lucene search library, MongoDB Atlas Search is the easiest way to build rich, fast, and relevant search directly into your applications. It compresses three systems — database, search engine, and sync mechanism — into one, so developers don’t have to deal with the problems that bolt-on search engines introduce. It can be enabled with a few API calls or clicks and uses the same query language as the rest of the MongoDB product family.

Atlas Search provides all of the features developers need for rich, personalized search experiences to users, like facets, now generally available, which offers users a way to quickly filter and navigate search results. With facets, developers can index data to map fields to categories like brand, size, or cost, and update query results based on relevance. This allows users to easily define multiple search criteria and see results updated in near real-time.

MongoDB Atlas Device Sync

With apps such as TikTok, Instagram, and Spotify, mobile users have come to expect features such as real-time updates, reactive UIs, and an always-on, always-available experience. While the user experience is effortless, building these abilities into a mobile app is anything but. Such features require lots of time and resources to develop, test, debug, and maintain.

MongoDB Atlas Device Sync is designed to help developers address mobile app data challenges, including limited connectivity, dead zones, and multiple collaborators (all with varying internet speeds and access) by gathering, syncing, and resolving any sync conflicts between the mobile database and MongoDB Atlas — without the burden of learning, deploying, and managing separate data technologies.

At World 2022, MongoDB announced Flexible Sync, a new way to sync data between devices and the cloud. Using Flexible Sync, developers can now define synced data using language-native queries and fine-grained permissioning, resulting in a faster, more seamless way of working — and one analogous to the way developers code and build.

Previously, developers had to sync full partitions of data; Flexible Sync enables synchronization of only the data that’s relevant. With support for filter logic, asymmetric sync, and hierarchical permissioning, Flexible Sync can reduce the amount of required code by 20% or more, and speed up build times from months to weeks.

Change Streams

Data changes quickly, and your applications need to react just as quickly. When a customer’s order is shipped, for instance, they expect an in-app or email notification — and they expect it immediately. Yet building applications that can respond to events in real time is difficult and often requires the use of polling infrastructure or third-party tools, both of which add to developer overhead. Latency and long reaction times result in data that is outdated, and poor experiences for users of that data.

Like Atlas’s Database Triggers, change streams enable developers to build event-driven applications and features that react to data changes as they happen. Along with reducing the complexity and cost of building this infrastructure from scratch, the new change stream enhancements (available in MongoDB 6.0) will enable you to determine the state of your database before and after an event occurs, so you can act on the changes and build business logic, analytics, and policies around it. That opens up new use cases, such as retrieving a copy of a document immediately after it is updated.

All of these updates and new capabilities focus on the critical need to eliminate complexity in order to build, deploy, and secure modern applications in any environment. Together, MongoDB helps solve what MongoDB president and CEO Dev Ittycheria called a key developer challenge in his MongoDB World 2022 keynote: reducing the friction and cost of working with data.

Learn more about MongoDB World 2022 announcements at mongodb.com/new and in these stories:

← Previous

Announcing Atlas Data Federation and Atlas Data Lake

>> Announcement: Some features mentioned below will be deprecated on Sep. 30, 2025. Learn more . Two years ago, we released the first iteration of Atlas Data Lake . Since then, we’ve helped customers combine data from various storage layers to feed downstream systems. But after years spent studying our customers’ experiences, we realized we hadn’t gone far enough. To truly unleash the genius in all our developers, we needed to add an economical cloud object storage solution with a rich MQL query experience to the world of Atlas. Today, we’re thrilled to announce that our new Atlas Data Federation and Atlas Data Lake offerings do just that. We now offer two complementary services, Atlas Data Federation (our existing query service formerly known as Atlas Data Lake) and our new and improved Atlas Data Lake (a fully managed analytic-oriented storage service). Together, these services (both in preview) provide flexible and versatile options for querying and transforming data across storage services, as well as a MongoDB-native analytic storage solution. With these tools, you can query across multiple clusters, move data into self managed cloud object storage for consumption by downstream services, query a workload-isolated inexpensive copy of cluster data, compare your cluster data across different points in time, and much, much more. In hearing from our customers about their experiences with Atlas Data Lake, we learned where they have struggled, as well as the features they’ve been looking for us to provide. With this in mind, we decided to shift the name of our current query federation service to Atlas Data Federation to better align with how customers see the service and are getting value. We’ve seen many customers benefit from the flexibility of a federated query engine service, including querying data across multiple clusters, databases, and collections, as well as exporting data to third-party systems. We also saw where our customers were struggling with data lakes. We heard them ask for a fully managed storage solution so they could achieve all of their analytic goals within Atlas. Specifically, customers wanted scalable storage that would provide high query performance at a low cost. Our new Data Lake provides a high-performance analytic object storage solution, allowing customers to query historical data with no additional formatting or maintenance work needed on their end. How it works Atlas Data Federation encompasses our existing Data Lake functionality with several new affordances. It continues to deliver the same power that it always has, with increased performance and efficiency. The new Atlas Data Lake will now allow you to create Data Lake pipelines (based on your Atlas Cluster backup schedules) and fields on which you can optimize queries. The service takes the following steps: On the selected schedule, a copy of your collection will be extracted from your Atlas backup with no impact to your cluster. During extraction, we build partition indexes based on the contents of your documents and the fields you’ve selected for optimization. These indexes allow your queries to be optimized by capturing the minimums and maximums (and other stats) of the records in each partition, letting you quickly find the relevant data for your queries. Finally, the underlying data lands in an analytic-oriented format inside of cloud object storage. This minimizes data scanned when you execute a query. Once a pipeline has run and a Data Lake dataset has been created, you can select it as a data source in our new Data Federation query experience. You can either set it as the source for a specific virtual collection in a Federated Database Instance or you can have your Federated Database Instance generate a collection name for each dataset that your pipeline has created. Amazingly, no part of this process will consume compute resources from your cluster — neither the export nor the querying of datasets. These datasets provide workload isolation and consistency for long-running analytic queries, a target for ETL jobs using the powerful $out to S3. This makes it easy to compare the state of your data over time. Advanced though this is, it’s only the beginning of the story. We’re committing to evolving the service, improving performance, adding more sources of data, and building new features. All of this will be based on the feedback you, the user, gives us. We can’t wait to see how you’ll use this powerful new tool and can’t wait to hear what you’d like to see next. Try Atlas Data Lake Today

June 7, 2022
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024