Atlas Stream Processing Adds AWS Regions, VPC Peering, & More!

Since announcing the general availability of Atlas Stream Processing—MongoDB’s native stream processing solution—it’s been exciting to see development teams across technology, retail, and manufacturing begin to run production stream processing workloads critical to their businesses.

Today, we're announcing four key updates to Atlas Stream Processing.

Graphic highlighting the four key updates to Atlas Stream Processing. The key updates are More AWS Regions, VPC Peering, Expanded Kafka Capabilities, and Expanded API Support.

Support for AWS Regions across the US, Europe, and APAC

First, we're thrilled to announce that Atlas Stream Processing now supports eight new AWS regions. This expansion enhances deployment flexibility across the US, Europe, and APAC. Adding these new AWS regions broadens our reach and opens up a world of possibilities for users. We're committed to further expanding our reach by adding more regions and cloud providers in the future.

Newly supported regions launched today include:

Region AWS Region Name
Oregon, USA
us-west-2
Sao Paulo, Brazil  
sa-east-1
Ireland
eu-west-1
London, England
eu-west-2
Frankfurt, Germany  
eu-central-1
Mumbai, India
ap-south-1
Singapore
ap-southeast-1
Sydney, Australia
ap-southeast-2

Adding these new AWS regions for Atlas Stream Processing is the latest example of the close partnership between MongoDB and AWS. For example, over the past year, MongoDB announced integrations with Amazon Bedrock and Amazon Q Developer; MongoDB was named an AWS Generative AI Competency Partner; we launched the MongoDB AI Applications Program—which helps customers rapidly build AI applications—with AWS and other tech leaders; and MongoDB was named the AWS Technology Partner of the Year at the AWS Partner Summit Taiwan.

Support for VPC peering

Next, Atlas Stream Processing now supports VPC peering for self-hosted Apache Kafka on AWS and Amazon Managed Streaming for Apache Kafka (AWS MSK).

VPC peering is a secure method for connecting between virtual private clouds. As stream processing solutions like Atlas Stream Processing inherently connect to external data sources outside of MongoDB, the ability to make these connections as if your resources are on the same private network is a critical security requirement for many organizations.

Users can select from any VPC peer configured within an Atlas project when setting up Kafka connections. Because peering is at the stream processing connection level, developers can configure Atlas Stream Processing to consume events from one Kafka cluster and produce them to another in a different VPC. Note that this feature has an additional cost. You can learn more in our documentation.

Expanded support for Apache Kafka

Third, we’re expanding capabilities for Apache Kafka in this release. Kafka is one of two key data sources Atlas Stream Processing supports today. One of Kafka’s strengths is its flexibility, allowing developers to customize configurations to suit various use cases, including those that rely on continuous stream processing.

That flexibility can also create complexity, but Atlas Stream Processing focuses on making Kafka’s critical features easily accessible using the MongoDB Query API.

By adding support for Kafka keys, developers can now read and write Kafka keys on their events, which enables filtering, partitioning, and aggregating based on key values. This ability provides greater control over routing processed data and is powerful for many stream processing use cases.

Expanded Atlas Admin API support

Lastly, we have added support for creating and deleting stream processors, as well as fetching operational stats of stream processors using the Atlas Admin API. Developers relying on the Admin API as a primary interface for interacting with Atlas will find this a welcome addition for managing their stream processors. Learn more in the documentation.

With these new capabilities—additional AWS region support, VPC peering, the ability to use Kafka keys, and improved stream processing support for the Atlas Admin API—we've made it easier than ever for developers to integrate stream processing into their applications. We're excited to see the innovative ways you'll use these features.

Ready to unlock the full potential of Atlas Stream Processing? Log in to Atlas today and start exploring the new features.

We're eager to hear your feedback, so don't hesitate to share it with us on UserVoice. Your insights help us continue to improve and innovate.

← Previous

Agnostiq & MongoDB: High-Performance Computing for All

Material scientists, computational biologists, and AI researchers all have at least one thing in common; the need for huge amounts of processing power, or ‘compute’, to turn raw data into results. But here’s the problem. Many researchers lack the skills needed to build the workflows that move data through huge networks of distributed servers, CPUs, and GPUs that actually do the number crunching. And that’s where Agnostiq comes in. Since the company’s inception in 2018, Agnostiq has put the power of high-performance computing (HPC) in the hands of researchers, bypassing the need for development expertise to build these essential data and compute pipelines. Power to the people “We started research on high-performance computing needs in fields like finance and chemistry, and through the process of onboarding, researchers quickly realized how hard it was for [researchers] to access and scale up on the cloud, or tap into HPC and GPU resources,'' said Santosh Kumar Radha, Agnostiq’s Head of Product. “If you wanted to scale up, there were not many tools available in the major cloud providers to do this.” To address this bottleneck, the team at Agnostiq built Covalent, a Python-based framework that allows researchers to easily design and run massive compute jobs on cloud platforms, on-prem clusters, and HPC services. With Covalent, startups and enterprises can build any AI or HPC application in a simple, scalable, and cost-effective way using a Python notebook, negating the need to interact with underlying infrastructure. One of the hardest challenges the Covalent team faced was combining traditional HPC with modern cloud technology. Because traditional HPC infrastructure was never designed to run in the cloud, the team spent considerable resources marrying techniques like GPU and CPU parallelization, task parallelization, and graph optimization with distributed cloud computing environments. As a result, researchers can use Covalent to quickly create a workflow that combines the convenience of cloud computing with specialized GPU providers and other HPC services. Everything, everywhere, all at once As the name suggests, Agnostiq has always focused on making their platform as open and resource neutral as possible. MongoDB Atlas , with its native multi-cloud capability, was a perfect complement. “At Agnostiq, everything we build has to be technology and vendor neutral. Interoperability is key for us,” said Radha. “We do all the mapping for our customers, so our platform has to perform a seamless transition from cloud to cloud.” The ability to move data between clouds became even more critical following the release of ChatGPT. With an explosion in generative AI research and development, the availability of GPU resources plummeted. “Resource scarcity in the ‘GPT era’ means you couldn’t get access to GPUs anywhere,” Radha added. “If you didn’t have a default cloud posture, you were nowhere, which is why we doubled down on multi-cloud and MongoDB Atlas to give our clients that optionality.” Open source opening doors Since the beginning, the team at Agnostiq has chosen MongoDB as their default NoSQL database. At first, the team adopted MongoDB’s free, open source product. “We didn’t have any DBAs as a small agile team. MongoDB gave us the freedom to build and manage our data workflows without the need for a specialist,” said William Cunningham, Head of HPC at Agnostiq. As their customer base grew along with the demand for cloud computing access, Agnostiq moved to MongoDB Atlas, gaining the freedom to move data seamlessly between AWS, Google Cloud, and Microsoft Azure. This gave Covalent the flexibility to reach multi-cloud compatibility at a faster rate than with standard tooling. Covalent provides a workflow management service by registering jobs, dispatching IDs, and collecting other metadata that allows fellow researchers and developers to reproduce the original work. MongoDB is used in the front-end, allowing a high volume of metadata and other assets to be published and cached in accordance with an event-driven architecture. This near real-time experience is key to a product aimed at delivering a unified view over distributed resources. MongoDB Atlas further provided the autoscaling required to grow with the user base and the number of workloads while keeping costs in check. “MongoDB Atlas helps us provide an ideal foundation for modern HPC and AI applications which require serverless compute, autoscaling resources, distributed workloads, and rapidly reconfigurable infrastructure,” added Radha. The future Looking to the future, Agnostiq is focused on servicing the huge demand for gen AI modeling and workflow building. To that end, the company released its own inference service called Function Serve within Covalent. Function Serve offers customers a complete, enterprise-grade solution for AI development and deployment, supporting serverless AI model training and fine-tuning. With Function Serve, customers can fine-tune, host, and serve any open-source or proprietary model with full infrastructure abstraction, all with only a few additional lines of code. MongoDB Atlas was used to rapidly develop a minimal service catalog while remaining cloud-agnostic. Looking ahead, the team plans to leverage MongoDB Atlas for enterprise and hybrid-cloud deployments in order to quickly meet customers in their existing cloud platforms. Agnostiq is a member of the MongoDB AI Innovators program , providing their team with access to Atlas credits and technical best practices. You can get started with your AI-powered apps by registering for MongoDB Atlas and exploring the tutorials available in our AI resources center . Additionally, if your company is interested in being featured, we'd love to hear from you. Reach out to us at ai_adopters@mongodb.com .

August 5, 2024
Next →

MongoDB’s 2024 Year in Review

It’s hard to believe that another year is almost over! 2024 was a transformative year for MongoDB, and it was marked by both innovation and releases that further our commitment to empowering customers, developers, and partners worldwide. So without further ado, let’s dive into MongoDB’s 2024 highlights. We’ll also share our executive team’s predictions of what 2025 might have in store. A look back at 2024 MongoDB 8.0: The most performant version of MongoDB ever In October we released MongoDB 8.0 , the fastest, most resilient, secure, and reliable version of MongoDB yet. Architectural optimizations in MongoDB 8.0 have significantly improved the database’s performance, with 36% faster reads and 59% higher throughput for updates. Our new architecture also makes horizontal scaling cheaper and faster. Finally, working with encrypted data is easier than ever, thanks to the addition of range queries in Queryable Encryption (which allows customers to encrypt, store, and perform queries directly on data). Whether you’re a startup building your first app, or you’re a global enterprise managing mission-critical workloads, MongoDB 8.0 offers unmatched power and flexibility, solidifying MongoDB’s place as the world’s most popular document database. Learn more about what makes 8.0 the best version of MongoDB ever on the MongoDB 8.0 page . Delivering customer value with the MongoDB AI Applications Program AI applications have become a cornerstone of modern software, and MongoDB is committed to equipping customers with the technology, tools, and support they need to succeed on their AI journey. That’s why we launched the MongoDB AI Applications Program (MAAP) in 2024, a comprehensive program designed to accelerate the development of AI applications. By offering customers resources like access to AI specialists, an ecosystem of leading AI and tech companies, and AI architectural best practices supported by integrated services, MAAP helps solve customers’ most pressing business challenges, unlocks competitive advantages, and accelerates time to value for AI investments. Overall, MAAP’s aim is to set customers on the path to AI success. Visit the MongoDB AI Applications Program page or watch our session from AWS re:Invent to learn more! Advancing AI with MongoDB Atlas Vector Search In 2024, MongoDB further cemented its role in the AI space with enhancements to MongoDB Atlas Vector Search . Recognized in 2024 (for the second consecutive year!) as one of the most loved vector databases , MongoDB continues to provide a scalable, unified, and secure platform for building cutting-edge AI use cases. Recent advancements like vector quantization in Atlas Vector Search help deliver even more value to our customers, enabling them to scale applications to billions of vectors at a lower cost. Head over to our Atlas Vector Search quick start guide to get started with Atlas Vector Search today, or visit our AI resources hub to learn more about how MongoDB can power AI applications. Search Nodes: Performance at scale Search functionality is indispensable in modern applications, and with Atlas Search Nodes, organizations can now optimize their search workloads like never before. By providing dedicated infrastructure for Atlas Search and Vector Search workloads, Search Nodes ensure high performance (e.g., a 40–60% decrease in query times), scalability, and reliability, even for the most demanding use cases. As of this year , Search Nodes are generally available across AWS, Google Cloud, and Microsoft Azure. This milestone underscores MongoDB’s commitment to delivering powerful solutions that scale alongside our customers’ needs. To learn more about Search Nodes, check out our documentation or watch our tutorial . Looking ahead: MongoDB’s 2025 predictions After the excitement of the past few years, 2025 will be defined by ensuring that technology investments deliver tangible value. Organizations remain excited about the potential AI and emerging technologies hold to solve real business challenges, but are increasingly focused on maintaining a return on investment. “Enterprises need to innovate faster than ever, but speed is no longer the only measure of success. Increasingly, organizations are laser-focused on ensuring that their technology investments directly address critical business challenges and provide clear ROI and competitive advantage—whether it’s optimizing supply chains, delivering hyper-personalized customer experiences, or scaling operations efficiently,” said Sahir Azam, Chief Product Officer at MongoDB. “In 2025, I expect to see organizations make significant strides in driving this innovation and efficiency by applying AI to more production use cases and by maturing the way they leverage their data to build compelling and differentiated customer experiences.” Indeed, we expect to see organizations make more strategic investments in emerging technologies like gen AI—innovating with a sharp focus on solving business challenges. “In 2025, we can expect the focus to shift from ‘what AI can do’ to ‘what AI should do,’ moving beyond the hype to a clearer understanding of where AI can provide real value and where human judgment is still irreplaceable,” said Tara Hernandez, VP of Developer Productivity at MongoDB. “As we advance, I think we’ll see organizations begin to adopt more selective, careful applications of AI, particularly in areas where stakes are high, such as healthcare, finance, and public safety. A refined approach to AI development will be essential—not only for producing quality results but also to build trust, ensuring these tools genuinely support human goals rather than undermining them.” With more capable, accessible application development tools and customer-focused programs like MAAP at developers’ fingertips, 2025 is an opportunity to make a data-driven impact faster than ever before. "Right now, organizations have an opportunity to leverage their data to reimagine how they do business, to more effectively adapt to a changing world, and to revolutionize our quality of life,” said Andrew Davidson, SVP of Products at MongoDB. “By harnessing our latest technologies, developers can build a foundation for a transformative future." Head over to our updates page to learn more about the new releases and updates from MongoDB in 2024. Keep an eye on our events page to learn what's to come from MongoDB in 2025!

December 19, 2024