Atlas Stream Processing Adds AWS Regions, VPC Peering, & More!

Since announcing the general availability of Atlas Stream Processing—MongoDB’s native stream processing solution—it’s been exciting to see development teams across technology, retail, and manufacturing begin to run production stream processing workloads critical to their businesses.

Today, we're announcing four key updates to Atlas Stream Processing.

Graphic highlighting the four key updates to Atlas Stream Processing. The key updates are More AWS Regions, VPC Peering, Expanded Kafka Capabilities, and Expanded API Support.

Support for AWS Regions across the US, Europe, and APAC

First, we're thrilled to announce that Atlas Stream Processing now supports eight new AWS regions. This expansion enhances deployment flexibility across the US, Europe, and APAC. Adding these new AWS regions broadens our reach and opens up a world of possibilities for users. We're committed to further expanding our reach by adding more regions and cloud providers in the future.

Newly supported regions launched today include:

Region AWS Region Name
Oregon, USA
us-west-2
Sao Paulo, Brazil  
sa-east-1
Ireland
eu-west-1
London, England
eu-west-2
Frankfurt, Germany  
eu-central-1
Mumbai, India
ap-south-1
Singapore
ap-southeast-1
Sydney, Australia
ap-southeast-2

Adding these new AWS regions for Atlas Stream Processing is the latest example of the close partnership between MongoDB and AWS. For example, over the past year, MongoDB announced integrations with Amazon Bedrock and Amazon Q Developer; MongoDB was named an AWS Generative AI Competency Partner; we launched the MongoDB AI Applications Program—which helps customers rapidly build AI applications—with AWS and other tech leaders; and MongoDB was named the AWS Technology Partner of the Year at the AWS Partner Summit Taiwan.

Support for VPC peering

Next, Atlas Stream Processing now supports VPC peering for self-hosted Apache Kafka on AWS and Amazon Managed Streaming for Apache Kafka (AWS MSK).

VPC peering is a secure method for connecting between virtual private clouds. As stream processing solutions like Atlas Stream Processing inherently connect to external data sources outside of MongoDB, the ability to make these connections as if your resources are on the same private network is a critical security requirement for many organizations.

Users can select from any VPC peer configured within an Atlas project when setting up Kafka connections. Because peering is at the stream processing connection level, developers can configure Atlas Stream Processing to consume events from one Kafka cluster and produce them to another in a different VPC. Note that this feature has an additional cost. You can learn more in our documentation.

Expanded support for Apache Kafka

Third, we’re expanding capabilities for Apache Kafka in this release. Kafka is one of two key data sources Atlas Stream Processing supports today. One of Kafka’s strengths is its flexibility, allowing developers to customize configurations to suit various use cases, including those that rely on continuous stream processing.

That flexibility can also create complexity, but Atlas Stream Processing focuses on making Kafka’s critical features easily accessible using the MongoDB Query API.

By adding support for Kafka keys, developers can now read and write Kafka keys on their events, which enables filtering, partitioning, and aggregating based on key values. This ability provides greater control over routing processed data and is powerful for many stream processing use cases.

Expanded Atlas Admin API support

Lastly, we have added support for creating and deleting stream processors, as well as fetching operational stats of stream processors using the Atlas Admin API. Developers relying on the Admin API as a primary interface for interacting with Atlas will find this a welcome addition for managing their stream processors. Learn more in the documentation.

With these new capabilities—additional AWS region support, VPC peering, the ability to use Kafka keys, and improved stream processing support for the Atlas Admin API—we've made it easier than ever for developers to integrate stream processing into their applications. We're excited to see the innovative ways you'll use these features.

Ready to unlock the full potential of Atlas Stream Processing? Log in to Atlas today and start exploring the new features.

We're eager to hear your feedback, so don't hesitate to share it with us on UserVoice. Your insights help us continue to improve and innovate.

← Previous

Agnostiq & MongoDB: High-Performance Computing for All

Material scientists, computational biologists, and AI researchers all have at least one thing in common; the need for huge amounts of processing power, or ‘compute’, to turn raw data into results. But here’s the problem. Many researchers lack the skills needed to build the workflows that move data through huge networks of distributed servers, CPUs, and GPUs that actually do the number crunching. And that’s where Agnostiq comes in. Since the company’s inception in 2018, Agnostiq has put the power of high-performance computing (HPC) in the hands of researchers, bypassing the need for development expertise to build these essential data and compute pipelines. Power to the people “We started research on high-performance computing needs in fields like finance and chemistry, and through the process of onboarding, researchers quickly realized how hard it was for [researchers] to access and scale up on the cloud, or tap into HPC and GPU resources,'' said Santosh Kumar Radha, Agnostiq’s Head of Product. “If you wanted to scale up, there were not many tools available in the major cloud providers to do this.” To address this bottleneck, the team at Agnostiq built Covalent, a Python-based framework that allows researchers to easily design and run massive compute jobs on cloud platforms, on-prem clusters, and HPC services. With Covalent, startups and enterprises can build any AI or HPC application in a simple, scalable, and cost-effective way using a Python notebook, negating the need to interact with underlying infrastructure. One of the hardest challenges the Covalent team faced was combining traditional HPC with modern cloud technology. Because traditional HPC infrastructure was never designed to run in the cloud, the team spent considerable resources marrying techniques like GPU and CPU parallelization, task parallelization, and graph optimization with distributed cloud computing environments. As a result, researchers can use Covalent to quickly create a workflow that combines the convenience of cloud computing with specialized GPU providers and other HPC services. Everything, everywhere, all at once As the name suggests, Agnostiq has always focused on making their platform as open and resource neutral as possible. MongoDB Atlas , with its native multi-cloud capability, was a perfect complement. “At Agnostiq, everything we build has to be technology and vendor neutral. Interoperability is key for us,” said Radha. “We do all the mapping for our customers, so our platform has to perform a seamless transition from cloud to cloud.” The ability to move data between clouds became even more critical following the release of ChatGPT. With an explosion in generative AI research and development, the availability of GPU resources plummeted. “Resource scarcity in the ‘GPT era’ means you couldn’t get access to GPUs anywhere,” Radha added. “If you didn’t have a default cloud posture, you were nowhere, which is why we doubled down on multi-cloud and MongoDB Atlas to give our clients that optionality.” Open source opening doors Since the beginning, the team at Agnostiq has chosen MongoDB as their default NoSQL database. At first, the team adopted MongoDB’s free, open source product. “We didn’t have any DBAs as a small agile team. MongoDB gave us the freedom to build and manage our data workflows without the need for a specialist,” said William Cunningham, Head of HPC at Agnostiq. As their customer base grew along with the demand for cloud computing access, Agnostiq moved to MongoDB Atlas, gaining the freedom to move data seamlessly between AWS, Google Cloud, and Microsoft Azure. This gave Covalent the flexibility to reach multi-cloud compatibility at a faster rate than with standard tooling. Covalent provides a workflow management service by registering jobs, dispatching IDs, and collecting other metadata that allows fellow researchers and developers to reproduce the original work. MongoDB is used in the front-end, allowing a high volume of metadata and other assets to be published and cached in accordance with an event-driven architecture. This near real-time experience is key to a product aimed at delivering a unified view over distributed resources. MongoDB Atlas further provided the autoscaling required to grow with the user base and the number of workloads while keeping costs in check. “MongoDB Atlas helps us provide an ideal foundation for modern HPC and AI applications which require serverless compute, autoscaling resources, distributed workloads, and rapidly reconfigurable infrastructure,” added Radha. The future Looking to the future, Agnostiq is focused on servicing the huge demand for gen AI modeling and workflow building. To that end, the company released its own inference service called Function Serve within Covalent. Function Serve offers customers a complete, enterprise-grade solution for AI development and deployment, supporting serverless AI model training and fine-tuning. With Function Serve, customers can fine-tune, host, and serve any open-source or proprietary model with full infrastructure abstraction, all with only a few additional lines of code. MongoDB Atlas was used to rapidly develop a minimal service catalog while remaining cloud-agnostic. Looking ahead, the team plans to leverage MongoDB Atlas for enterprise and hybrid-cloud deployments in order to quickly meet customers in their existing cloud platforms. Agnostiq is a member of the MongoDB AI Innovators program , providing their team with access to Atlas credits and technical best practices. You can get started with your AI-powered apps by registering for MongoDB Atlas and exploring the tutorials available in our AI resources center . Additionally, if your company is interested in being featured, we'd love to hear from you. Reach out to us at ai_adopters@mongodb.com .

August 5, 2024
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024