Product Updates
The most recent MongoDB product releases and updates
MongoDB Atlas Integration with Ably Unlocks Real-time Capabilities
Enterprises across sectors increasingly realize that data, like time, doesn’t wait. Indeed, harnessing and synchronizing information in real time is the new currency of business agility. Enter the alliance between MongoDB and Ably—a partnership that has led to Ably's new database connector for MongoDB Atlas . The new database connector provides a robust framework for businesses to create real-time, data-intensive applications that can provide top-notch user experiences thanks to an opinionated client SDK to be used on top of LiveSync, ensuring both data integrity and real-time consistency—without compromising your existing tech stack. The synergy of MongoDB Atlas and Ably LiveSync This new MongoDB Atlas-Ably integration tackles a fundamental challenge in modern application architecture: maintaining data consistency across distributed systems in real-time. MongoDB Atlas serves as the foundation—a flexible, scalable database service that adapts to the ebb and flow of data demands. Meanwhile, Ably LiveSync acts as the nervous system, ensuring that every change, every update, resonates instantly across the entire application ecosystem. The Ably LiveSync database connector for MongoDB Atlas offers a transformative approach to real-time data management, combining unparalleled scalability with seamless synchronization. This solution effortlessly adapts to growing data volumes and expanding user bases, catering to businesses of all sizes—from agile startups to established enterprises. By rapidly conveying database changes to end-users, it ensures that all stakeholders operate from a single, up-to-date source of truth, fostering data consistency across the entire organization. At its core, LiveSync is built with robust resilience in mind, featuring built-in failover mechanisms and connection recovery capabilities. This architecture provides businesses with the high availability they need to maintain continuous operations in today's always-on digital landscape. Moreover, by abstracting away the complexities of real-time infrastructure, LiveSync empowers developers to focus on creating features that drive business value. This focus on developer productivity, combined with its scalability and reliability, positions Ably LiveSync for MongoDB Atlas as a cornerstone technology for companies aiming to harness the power of real-time data synchronization. Figure 1: Ably real-time integration with MongoDB Atlas. Industry transformation: A real-time revolution This new integration has a number of implications across various sectors. For example, in the banking and financial services sector , the MongoDB Atlas-Ably integration enables instantaneous fraud detection systems that can promptly react to potential threats. Live trading platforms benefit as well, seamlessly updating to reflect every market change as it happens. Banking applications are equally enhanced, with real-time updating of account balances and transactions, ensuring that users always have access to the most recent financial information. In the retail industry , meanwhile, the integration facilitates real-time inventory management across both physical and online stores, ensuring that supply matches demand at all times. This capability supports dynamic pricing strategies that can adapt instantly to fluctuations in consumer interest, and it powers personalized shopping experiences with live product recommendations tailored to individual customer preferences. Manufacturing and mobility sectors also see transformative benefits. With the capability for real-time monitoring of production lines, businesses can implement just-in-time manufacturing processes, streamlining operations and reducing waste. Real-time tracking of vehicles and assets enhances logistics efficiency, while predictive maintenance systems provide foresight into potential equipment failures, allowing for timely interventions. The healthcare sector stands to gain significantly from this technology. Real-time patient monitoring systems offer healthcare providers immediate alerts, ensuring swift medical responses when necessary. Electronic health records receive seamless updates across multiple care settings, promoting coherent patient care. Efficient resource allocation is achieved through live tracking of hospital beds and equipment, optimizing hospital operations. Insurance companies are not left out of this technological leap. The integration allows for dynamic risk assessment and pricing models that adapt in real-time, refining accuracy and responsiveness. Instant claim processing and status updates enhance customer satisfaction, while live tracking of insured assets facilitates more accurate underwriting and expedites the resolution of claims. Finally, in telecommunications and media this integration promises buffer-free content delivery and streaming services, vastly improving the end-user experience. real-time network performance monitoring enables proactive issue resolution, maintaining service quality. Users can enjoy synchronized experiences across multiple devices and platforms, fostering seamless interaction with digital content. Today's business imperative As industries continue to evolve at a rapid pace, the integration of MongoDB Atlas and Ably LiveSync provides a compelling way for businesses to not only keep up but lead the real-time revolution. For IT decision-makers looking to put their organizations at the forefront of innovation, this integration turns static data into a dynamic driver of business growth and market leadership. Access MongoDB Atlas and Ably LiveSync Resources and start your journey towards real-time innovation today. Learn more about how MongoDB Atlas can power industry-specific solutions .
Leveraging BigQuery JSON for Optimized MongoDB Dataflow Pipelines
We're delighted to introduce a major enhancement to our Google Cloud Dataflow templates for MongoDB Atlas. By enabling direct support for JSON data types, users can now seamlessly integrate their MongoDB Atlas data into BigQuery, eliminating the need for complex data transformations. This streamlined approach not only saves users time and resources, but it also empowers customers to unlock the full potential of their data through advanced data analytics and machine learning. Figure 1: JSON feature for user options on Dataflow Templates Limitations without JSON support Traditionally, Dataflow pipelines designed to handle MongoDB Atlas data often necessitate the transformation of data into JSON strings or flattening complex structures to a single level of nesting before loading into BigQuery. Although this approach is viable, it can result in several drawbacks: Increased latency: The multiple data conversions required can lead to increased latency and can significantly slow down the overall pipeline execution time. Higher operational costs: The extra data transformations and storage requirements associated with this approach can lead to increased operational costs. Reduced query performance: Flattening complex document structures in JSON String format can impact query performance and make it difficult to analyze nested data. So, what’s new? BigQuery's Native JSON format addresses these challenges by enabling users to directly load nested JSON data from MongoDB Atlas into BigQuery without any intermediate conversions. This approach offers numerous benefits: Reduced operating costs: By eliminating the need for additional data transformations, users can significantly reduce operational expenses, including those associated with infrastructure, storage, and compute resources. Enhanced query performance: BigQuery's optimized storage and query engine is designed to efficiently process data in Native JSON format, resulting in significantly faster query execution times and improved overall query performance. Improved data flexibility: users can easily query and analyze complex data structures, including nested and hierarchical data, without the need for time-consuming and error-prone flattening or normalization processes. A significant advantage of this pipeline lies in its ability to directly leverage BigQuery's powerful JSON functions on the MongoDB data loaded into BigQuery. This eliminates the need for a complex and time-consuming data transformation process. The JSON data within BigQuery can be queried and analyzed using standard BQML queries. Whether you prefer a streamlined cloud-based approach or a hands-on, customizable solution, the Dataflow pipeline can be deployed either through the Google Cloud console or by running the code from the github repository . Enabling data-driven decision-making To summarize, Google’s Dataflow template provides a flexible solution for transferring data from MongoDB to BigQuery. It can process entire collections or capture incremental changes using MongoDB's Change Stream functionality. The pipeline's output format can be customized to suit your specific needs. Whether you prefer a raw JSON representation or a flattened schema with individual fields, you can easily configure it through the userOption parameter. Additionally, data transformation can be performed during template execution using User-Defined Functions (UDFs). By adopting BigQuery Native JSON format in your Dataflow pipelines, you can significantly enhance the efficiency, performance, and cost-effectiveness of your data processing workflows. This powerful combination empowers you to extract valuable insights from your data and make data-driven decisions. Follow the Google Documentation to learn how to set up the Dataflow templates for MongoDB Atlas and BigQuery. Get started with MongoDB Atlas on Google Marketplace . Learn more about MongoDB Atlas on Google Cloud on our product page .
Checkpointers and Native Parent Child Retrievers with LangChain and MongoDB
MongoDB and LangChain, the company known for its eponymous large language model (LLM) application framework, are excited to announce new developments in an already strong partnership. Two additional enhancements have just been added to the LangChain codebase, making it easier than ever to build cutting-edge AI solutions with MongoDB. Checkpointer support In LangGraph, LangChain’s library for building stateful, multi-actor applications with LLMs, memory is provided through checkpointers . Checkpointers are snapshots of the graph state at a given point in time. They provide a persistence layer, allowing developers to interact and manage the graph’s state. This has a number of advantages for developers—human-in-the-loop, "memory" between interactions, and more. Figure adapted from “Launching Long-Term Memory Support in LangGraph”. LangChain Blog. Oct. 8, 2024. https://blog.langchain.dev/launching-long-term-memory-support-in-langgraph/ MongoDB has developed a custom checkpointer implementation, the " MongoDBSaver " class, that, with just a MongoDB URI (local or Atlas ), can easily store LangGraph state in MongoDB. By making checkpointers a first-class feature, developers can have confidence that their stateful AI applications built on MongoDB will be performant. That’s not all, since there are actually two new checkpointers as part of this implementation— one synchronous and one asynchronous . This versatility allows the new functionality to be even more versatile, and serving developers with a myriad of use cases. Both implementations include helpful utility functions to make using them painless, letting developers easily store instances of StateGraph inside of MongoDB. A performant persistence layer that stores data in an intuitive way will mean a better end-user experience and a more robust system, no matter what a developer is building with LangGraph. Native parent child retrievers Second, MongoDB has implemented a native parent child retriever inside LangChain. This approach enhances the performance of retrieval methods utilizing the retrieval-augmented Generation (RAG) technique by providing the LLM with a broader context to consider. In essence, we divide the original documents into relatively small chunks, embed each one, and store them in MongoDB. Using such small chunks (a sentence or a couple of sentences) helps the embedding models to better reflect their meaning. Now developers can use " MongoDBAtlasParentDocumentRetriever " to persist one collection for both vector and document storage. In this implementation, we can store both parent and child documents in a single collection while only having to compute and index embedding vectors for the chunks. This has a number of performance advantages because storing vectors with their associated documents means no need to join tables or worry about painful schema migrations. Additionally, as part of this work, MongoDB has also added a " MongoDBDocStore " class which provides many helpful utility functions. It is now easier than ever to use documents as a key-value store and insert, update, and delete them with ease. Taken together, these two new classes allow developers to take full advantage of MongoDB’s abilities. MongoDB and LangChain continue to be a strong pair for building agentic AI—combining performance and ease of development to provide a developer-friendly experience. Stay tuned as we build out additional functionality! To learn more about these LangChain integrations, here are some resources to get you started: Check out our tutorial . Experiment with checkpointers and native parent child retrievers to see their utility for yourself. Read the previous announcement with LangChain about AI Agents, Hybrid Search, and Indexing.
Binary Quantization & Rescoring: 96% Less Memory, Faster Search
We are excited to share that several new vector quantization capabilities are now available in public preview in MongoDB Atlas Vector Search : support for binary quantized vector ingestion, automatic scalar quantization, and automatic binary quantization and rescoring. Together with our recently released support for scalar quantized vector ingestion , these capabilities will empower developers to scale semantic search and generative AI applications more cost-effectively. For a primer on vector quantization, check out our previous blog post . Enhanced developer experience with native quantization in Atlas Vector Search Effective quantization methods—specifically scalar and binary quantization—can now be done automatically in Atlas Vector Search. This makes it easier and more cost-effective for developers to use Atlas Vector Search to unlock a wide range of applications, particularly those requiring over a million vectors. With the new “quantization” index definition parameters, developers can choose to use full-fidelity vectors by specifying “none,” or they can quantize vector embeddings by specifying the desired quantization type—”scalar” or “binary” (Figure 1). This native quantization capability supports vector embeddings from any model provider as well as MongoDB’s BinData float32 vector subtype . Figure 1: New index definition parameters for specifying automatic quantization type in Atlas Vector Search Scalar quantization—converting a float point into an integer—is generally used when it's crucial to maintain search accuracy on par with full-precision vectors. Meanwhile, binary quantization—converting a float point into a single bit of 0 or 1—is more suitable for scenarios where storage and memory efficiency are paramount, and a slight reduction in search accuracy is acceptable. If you’re interested in learning more about this process, check out our documentation . Binary quantization with rescoring: Balance cost and accuracy Compared to scalar quantization, binary quantization further reduces memory usage, leading to lower costs and improved scalability—but also a decline in search accuracy. To mitigate this, when “binary” is chosen in the “quantization” index parameter, Atlas Vector Search incorporates an automatic rescoring step, which involves re-ranking a subset of the top binary vector search results using their full-precision counterparts, ensuring that the final search results are highly accurate despite the initial vector compression. Empirical evidence demonstrates that incorporating a rescoring step when working with binary quantized vectors can dramatically enhance search accuracy, as shown in Figure 2 below. Figure 2: Combining binary quantization and rescoring helps retain search accuracy by up to 95% And as Figure 3 shows, in our tests, binary quantization reduced processing memory requirement by 96% while retaining up to 95% search accuracy and improving query performance. Figure 3: Improvements in Atlas Vector Search with the use of vector quantization It’s worth noting that even though the quantized vectors are used for indexing and search, their full-fidelity vectors are still stored on disk to support rescoring. Furthermore, retaining the full-fidelity vectors enables developers to perform exact vector search for experimental, high-precision use cases, such as evaluating the search accuracy of quantized vectors produced by different embedding model providers, as needed. For more on evaluating the accuracy of quantized vectors, please see our documentation . So how can developers make the most of vector quantization? Here are some example use cases that can be made more efficient and scaled effectively with quantized vectors: Massive knowledge bases can be used efficiently and cost-effectively for analysis and insight-oriented use cases, such as content summarization and sentiment analysis. Unstructured data like customer reviews, articles, audio, and videos can be processed and analyzed at a much larger scale, at a lower cost and faster speed. Using quantized vectors can enhance the performance of retrieval-augmented generation (RAG) applications. The efficient processing can support query performance from large knowledge bases, and the cost-effectiveness advantage can enable a more scalable, robust RAG system, which can result in better customer and employee experience. Developers can easily A/B test different embedding models using multiple vectors produced from the same source field during prototyping. MongoDB’s flexible document model lets developers quickly deploy and compare embedding models’ results without the need to rebuild the index or provision an entirely new data model or set of infrastructure. The relevance of search results or context for large language models (LLMs) can be improved by incorporating larger volumes of vectors from multiple sources of relevance, such as different source fields (product descriptions, product images, etc.) embedded within the same or different models. To get started with vector quantization in Atlas Vector Search, see the following developer resources: Documentation: Vector Quantization in Atlas Vector Search Documentation: How to Measure the Accuracy of Your Query Results Tutorial: How to Use Cohere's Quantized Vectors to Build Cost-effective AI Apps With MongoDB
Atlas Stream Processing Now Supports Azure and Azure Private Link
Today, we’re excited to announce that Atlas Stream Processing now supports Microsoft Azure! This update opens new possibilities for developers leveraging Azure’s cloud ecosystem, offering a way to: Seamlessly integrate MongoDB Atlas and Apache Kafka Effortlessly handle complex and rapidly changing data structures Use the familiarity of the MongoDB Query API for processing streaming data Benefit from a fully managed service that eliminates operational overhead Azure support in four regions At launch, we’re supporting four Azure regions spanning both the U.S. and Europe: Azure Region Location US East Virginia, US US East 2 Virginia, US US West California, US West Europe Netherlands We’ll continue adding more regions across cloud providers in the future. Let us know which regions you need next in UserVoice . Atlas Stream Processing simplifies integrating MongoDB with Apache Kafka to build event-driven applications. New to Atlas Stream Processing? Watch our 3-minute explainer . How it works Working with Atlas Stream Processing on Azure will feel just like it does already today when using AWS. During the Stream Processing Instance (SPI) tier selection in the Atlas UI or CLI, simply select Azure as your provider and then choose your desired region. Figure 1: Stream Processing instance setup via Atlas UI $ atlas streams instances create AzureSPI --provider AZURE --region westus --tier SP10 Figure 2: Stream Processing instance setup via the Atlas CLI Secure networking for Azure Event Hubs via Azure Private Link In addition to adding support for Azure in multiple regions, we’re introducing Azure Private Link support for developers using Azure Event Hubs . Event Hubs is Azure’s native, Kafka-compatible data streaming service. As a reminder, Atlas Stream Processing supports any service that uses the Kafka Wire Protocol . That includes Azure Event Hubs, AWS Managed Service for Kafka (MSK), Redpanda, and Confluent Cloud. As we have written before , security is critical for data services, and it’s especially important with stream processing systems where connecting to technologies like Apache Kafka external to a database like MongoDB Atlas, is required. For this reason, we’re engineering Atlas Stream Processing to leverage the advanced networking capabilities available through the major cloud providers (AWS, Azure, and GCP). Networking To better understand the value of support for private link, let’s summarize the three key ways that developers typically connect between data services: Public networking Private networking through VPC peering Private networking through private link Public networking connects services using public IP addresses. It’s the least secure of all approaches. This makes it the easiest to set up, but it's a less secure approach than either VPC peering or private link. Private networking through VPC peering connects services across two virtual private clouds (VPCs). This improves security compared with public networking by keeping traffic off the public internet and is commonly used for testing and development purposes. Private networking through private link is even more secure by enforcing connections to specific endpoints. While VPC peering lets resources from one VPC connect to all of the resources in the other VPC, private link ensures that each specific resource can only connect to defined services with specific associated endpoints. This connection method is important for use cases relying on sensitive data. Figure 3: Private Link allows for connecting to specific endpoints Ready to get started? With support for Azure Private Link, Atlas Stream Processing now makes it simple to implement the most secure method for networking across MongoDB and Kafka on Azure Event Hubs. Login today to get started, or check out our documentation to create your first private link connection.
What’s New From MongoDB at AWS re:Invent 2024
As thousands of attendees make their way home after a week in Vegas—a week packed with learning, product launches, and round-the-clock events—we thought we’d reflect on the show’s highlights. MongoDB was excited to showcase our latest integrations and solutions with Amazon Web Services (AWS), which range from new ways to optimize generative AI, to faster, more cost-effective methods for modernizing applications. But first, we want to thank our friends at AWS for recognizing MongoDB as the AWS Technology Partner of the Year NAMER! This prestigious award recognizes AWS Technology Partners that are using AWS to lower costs, increase agility, and innovate faster. Announced during the re:Invent Partner Awards Gala, the Technology Partner of the Year Award is a testament to the specialization, innovation, and cooperation MongoDB and AWS have jointly brought to customers this year. In addition, MongoDB also received AWS Partner of the Year awards for Italy, Turkey, and Iberia. These awards follow wins in the ASEAN Global Software Partner of the Year and Taiwan Technology Partner of the Year categories earlier in the year, further demonstrating the global reach and popularity of the world’s most popular document database! Harnessing the potential of gen AI If 2024 (and 2023, and 2022…) was the year of gen AI excitement, then 2025 may turn out to be marked by realistic gen AI implementation. Indeed, we’ve seen customers shift their 2025 AI focus toward optimizing resource-intensive gen AI workloads to drive down costs—and to get the most out of this groundbreaking technology. Retrieval-augmented generation (RAG), one of the main ways companies use their data to customize the output of foundation models, has become the focus of this push for optimization. Customers are looking for easier ways to fine-tune their RAG systems, asking questions like, “How do I evaluate the efficiency and accuracy of my current RAG workflow?” To that end, AWS and MongoDB are introducing new services and technologies for enterprises to optimize RAG architecture compute costs, while also maintaining accuracy. First up is vector quantization. By reducing vector storage and memory requirements while preserving performance, these capabilities empower developers to build AI-enriched applications with more scale—and at a lower cost. Leading foundation models like Amazon Titan are already compatible with vector quantization, helping to maintain high accuracy of generated responses while simultaneously reducing costs. You can read more about vector quantization on our blog. As for RAG evaluation, AWS has launched a new feature for Amazon Bedrock called, naturally, RAG Evaluator. This tool allows Bedrock users to evaluate and monitor RAG Apps natively within the Bedrock environment, eliminating the need for third-party frameworks to run tests and comparisons. As a Knowledge Base for Amazon Bedrock, MongoDB Atlas is ready on day one to take advantage of Bedrock RAG Evaluator, allowing companies to gauge and compare the quality of their RAG Apps across different applications. The RAG Evaluator, built on several joint integrations and solutions AWS and MongoDB released in 2024, and vector quantization together can streamline the deployment of enterprise generative AI. For example, in October MongoDB, Anthropic, and AWS announced a joint solution to create a memory-enhanced AI agent . Together, the three partners offer enterprise-grade, trusted, secure technologies to build generative AI apps quickly and flexibly using a family of foundation models in a fully managed environment. Overall, MongoDB and AWS are making it easier—and more cost-effective—for developers to build innovative applications that harness the full potential of generative AI on AWS. From cars to startups to glue MongoDB and AWS have been hard at work on a number of other solutions for developers across industries. Here’s a quick roundup: AWS Amplify + AppSync + MongoDB For startups, or for any organization looking to quickly test and launch applications, speed is everything. That’s why MongoDB teamed up with AWS to create a full-stack solution that provides developers with the same high standards of performance and scalability they would demand for any app. By combining AWS Amplify, AWS AppSync, and MongoDB Atlas, AWS and MongoDB have created a full-stack solution that enables seamless front-end development, robust and scalable backend services, out-of-the-box CI/CD, and a flexible and powerful database solution, allowing developers to drastically reduce the coding time required to launch new applications. Check out this tutorial and repository for a starter template . Digital twins on AWS CMS For those in the automotive sector, MongoDB and AWS have developed a connected mobility solution to help remove the undifferentiated integration, or “technical plumbing” work, of connecting vehicles to the cloud. When used together, Connected Mobility Solution (CMS) on AWS and MongoDB Atlas help accelerate the development of next-generation digital twin use cases and applications, including connected car use cases. MongoDB’s document model allows easy and flexible modeling and storage of connected vehicle sensor data. Read our joint blog with AWS to learn how the MongoDB Atlas document model helps with data modeling of connected vehicles data and how this capability can be leveraged via AWS Automotive Cloud Developer Portal (ACDP). AWS Glue + MongoDB Atlas Speaking of undifferentiated plumbing, MongoDB Atlas is now integrated into AWS Glue’s visual interface. The new integration simplifies data integration between MongoDB Atlas and AWS, making it easy to build efficient ETL (Extract, Transform, Load) pipelines with minimal effort. With its visual interface, AWS Glue allows users to seamlessly transfer, transform, and load data to and from MongoDB Atlas without needing deep technical expertise in Spark or SQL. In this blog post , we look at how AWS Glue and MongoDB Atlas can transform the way you manage data movement. Buy with AWS In the spirit of making things easier for our joint customers, in early 2025 MongoDB will also join the AWS ‘Buy with AWS’ program. Once up and running, Buy With AWS will allow customers to pay for Atlas using their AWS account directly from the Atlas UI, further reducing friction for customers wanting to get started with Atlas on AWS. New Atlas Updates Announced at re:Invent Aside from our joint endeavors with AWS, MongoDB has also been hard at work on improving the core Atlas platform. Here’s an overview of what we announced: Asymmetrical sharding support for Terraform Atlas Provider Customers are constantly seeking ways to optimize costs to ensure they get the best value for their resources. With Asymmetrical Sharding, now available in the Terraform MongoDB Atlas Provider, MongoDB Atlas users can customize the Cluster Tier and IOPS for each shard, encouraging better resource allocation, improved operational efficiency, and cost savings as customer needs evolve. Atlas Flex Tier Our new Atlas Flex tier offers the scaled flexibility of serverless, with the cost-capped assurance of shared tier clusters. With Atlas Flex Tier, developers can build and scale applications cost-effectively without worrying about runaway bills or resource provisioning. New test bench feature in Query Converter At MongoDB, we firmly believe that the document model is the best way for customers to build applications with their data. In our latest update to Relational Migrator , we’ve introduced Generative AI to automatically convert SQL database objects and validate them using the test bench in a fraction of the time, producing deployment-ready code up to 90% faster. This streamlined approach reduces migration risks and manual development effort, enabling fast, efficient, and precise migrations to MongoDB. For more about MongoDB’s work with AWS—including recent announcements and the latest product updates—please visit the MongoDB Atlas on AWS page ! Visit our product page to learn more about MongoDB Atlas .
MongoDB, Microsoft Team Up to Enhance Copilot in VS Code
As modern applications grow increasingly complex, developers face the challenge of meeting market demands for faster, smarter solutions. To stay ahead, they need tools that streamline their workflows, available directly in the environments where they build. According to the 2024 Stack Overflow Developer Survey , Microsoft’s Visual Studio Code (VS Code) is the integrated development environment (IDE) of choice for 74% of professional developers, serving as a central hub for building, testing, and deploying applications. With the rise of AI-powered tools like GitHub Copilot—which is used by 44% of professional developers—there’s a growing demand for intelligent assistance in the development process without disrupting flow. At MongoDB, we believe that the future of development lies in democratizing the value of these experiences by incorporating domain-specific knowledge and capabilities directly into developer flows. That’s why we’re thrilled to announce the public preview of MongoDB’s extension to GitHub Copilot in VS Code. With this integration, developers can effortlessly generate MongoDB queries, inspect collection schemas, and get answers from the latest MongoDB docs—all without leaving their IDE. Our collaboration with MongoDB continues to bring powerful, integrated solutions to developers building the modern applications of the future. The new MongoDB extension for GitHub Copilot exemplifies a shared commitment to the developer experience, leveraging AI to ensure that workflows are optimized for developer productivity by keeping everything developers need within reach, without breaking their flow. Isidor Nikolic, Senior Product Manager for VS Code, Microsoft But we’re not stopping there. As AI continues to evolve, so will the ways developers interact with their tools. Stay tuned for more exciting developments next week at Microsoft Ignite , where we’ll unveil more ways we’re pushing the boundaries of what’s possible with AI through MongoDB and Microsoft’s partnership! What is MongoDB's Copilot extension? MongoDB’s Copilot extension supercharges your GitHub Copilot in VS Code with MongoDB domain knowledge. The Copilot integration is built into the MongoDB for VS Code extension , which has more than 1.8M downloads in the VS Code marketplace today. Type ‘@MongoDB’ in Copilot chat and take advantage of three transformative commands: Generate queries from natural language (/query) —this generates accurate MongoDB queries by passing collection schema as context to Github Copilot Query MongoDB documentation (/docs) —this answers any documentation questions using the latest MongoDB documentation through Retrieval-Augmented Generation (RAG) Browse collection schema (/schema) —this provides schema information for any collection and is useful for data modeling with the Copilot extension. Generate queries from natural language This command transforms natural language prompts into MongoDB queries, leveraging your collection schema to produce precise, valid queries. It eliminates the need to manually write complex query syntax, and allows developers to quickly extract data without taking their focus away from building applications. Whether you run the query directly from the Copilot chat or refine it in a MongoDB playground file, we’ve sped up the query-building process by deeply integrating these capabilities into the existing flow of MongoDB VS Code extension. Query MongoDB documentation The /docs command answers MongoDB documentation-specific questions, complemented by direct links to the official documentation site. There’s no need to switch back and forth between your browser and your IDE; the Copilot extension calls out to the MongoDB Documentation Chatbot API that leverages retrieval-augmented generation technology to generate responses that are informed by the most recent version of the MongoDB documentation. In the near future, these questions will be smartly routed to documentation for the specific server version of the cluster you are connected to in the MongoDB VS Code extension. Browse collection schema The /schema command offers quick access to collection schemas, making it easier for developers to access and interact with their data model in real-time. This can be helpful in situations where developers are debugging with Copilot or just want to know valid field names while developing their applications. Developers can additionally export collection schemas into JSON files or ask follow-up questions directly to brainstorm data modeling techniques with the MongoDB Copilot extension. On the Horizon This is just the start of our work on MongoDB’s Copilot extension. As we continue to improve the experience with new features—like translating and testing queries to and from popular programming languages, and in-line query generation in Playgrounds—we remain focused on democratizing AI-driven workflows, empowering developers to access the tools and knowledge they need to build smarter, faster, and more efficiently, right within their existing environments. Download MongoDB’s VS Code extension and enable the MongoDB chat experience to get started today.
MongoDB Atlas Introduces Enhanced Cost Optimization Tools
MongoDB Atlas was designed with elasticity at its core and has always allowed customers to scale capacity vertically and horizontally, as required and automatically. Today, these inherent capabilities are even better and more cost-effective. At the recent MongoDB.local London, MongoDB announced several new MongoDB Atlas features that improve elasticity and help optimize costs while maintaining the performance and availability that business-critical applications demand. These include scaling each shard independently, extending storage beyond 4 TB or more , and 5X more responsive auto-scaling . Organizations and their customers are inherently dynamic, with operations, web traffic, and application usage growing unpredictably and non-linearly. For example, website traffic can spike due to a single video going viral on social media, and holidays are a frequent cause of application usage slowdowns. Traditionally, organizations have tackled this volatility by over-provisioning infrastructure, often at significant cost. Cloud adoption has improved the speed at which infrastructure can be provisioned in response to growing and volatile demand. Simultaneously, companies are focused on striking the perfect balance between performance and cost efficiency. This balance is acute in the current economic climate, where cost optimization is a top priority for Infrastructure & IT Operations (I&O) leaders. The goal is not balance between supply and demand. The goal is to meet the most profitable and mission-critical demand with the resources available. Nathan Hill, Distinguished VP Analyst, Gartner - Dec 2023 However, scaling infrastructure to meet demand without overprovisioning can be complex and costly. Organizations have often relied on manual processes (like scheduled scripts) or dedicated teams (like IT ops) to manage this challenge. MongoDB Atlas enables a more effective approach. With MongoDB Atlas, customers can manage flexible provisioning, zero-downtime scaling, and easy auto-scaling of their clusters. From October 2024, all Atlas customers with dedicated tier clusters can employ these recently announced enhancements for improved cost optimization. Granular resource provisioning MongoDB’s tens of thousands of customers have complex and diverse workloads with constantly changing requirements. Over time, workloads can grow unpredictably, requiring scaling up storage, compute, and IOPS independently and at differing granularities. Imagine a global retailer preparing for Cyber Monday, when traffic could be 512% higher than average — additional resources to serve customers are vital. Independent shard scaling enables customers running MongoDB Atlas to do this in a cost-optimal manner. Customers can independently scale the tier of individual shards in a cluster when one or more shards experience disproportionately higher traffic. For customers running workloads on sharded clusters, scaling each shard independently of all other shards is now an option (for example, only the shards serving US traffic during Thanksgiving). Customers can scale operational and analytical nodes independently in a single shard. This improves scalability and cost-optimization by providing fine-grained control to add resources to hot shards while maintaining the resources provisioned to other shards. All Atlas customers running dedicated clusters can use this feature through Terraform and the Admin API . Support for independent shard auto-scaling and configuration management via the Admin API and Terraform will be available in late 2024. Extended Storage and IOPS in Azure : MongoDB is introducing the ability to provision additional storage and IOPS on Atlas clusters running on Azure. This enables support for optimal performance without over-provisioning. Customers can create new clusters on Azure to provision additional IOPS and extended storage with 4TB or more on larger clusters (M40+). This feature is being rolled out and will be available to all Atlas clusters by late 2024. Head over to our docs page to learn more. With these updates, customers have greater flexibility and granularity in provisioning and scaling resources across their Atlas clusters on all three major cloud providers. Therefore, customers can optimize for performance and costs more effectively. More responsive auto-scaling Granular provisioning is excellent for optimizing costs while ensuring availability for an expected increase in traffic. However, what happens if a website gets 13X higher traffic or a surge in app interactions due to an unexpected social media post? Several enhancements to the algorithms and infrastructure powering MongoDB’s auto-scaling capabilities were announced in October 2024 at .local London . Cumulatively, these improve the time taken to scale and the responsiveness of MongoDB’s auto-scaling engine. Customers running dynamic workloads, particularly those with sharper peaks, will see up to 5X improvement in responsiveness. Smarter scaling decisions by Atlas will ensure that resource provisioning is optimized while maintaining high performance. This capability is available on all Atlas clusters with auto-scaling turned on, and customers should experience the benefits immediately. Industry-leading MongoDB Atlas customers like Conrad and Current use auto-scaling to automatically scale their compute capacity, storage capacity, or both without needing custom scripts, manual intervention, or third-party consulting services. Customers can set upper and lower tier limits, and Atlas will automatically scale their storage and tiers depending on their workload demands. This ensures clusters always have the optimal resources to maintain performance while optimizing costs. Take a look at how Coinbase is optimizing for both availability and cost in the volatile world of cryptocurrency with MongoDB Atlas’ help, or read our auto-scaling docs page to learn more. Optimize price and performance with MongoDB Atlas As businesses focus more on optimizing cloud infrastructure costs, the latest MongoDB Atlas enhancements— independent shard scaling, more responsive auto-scaling, and extended storage with IOPS—empower organizations to manage resources efficiently while maintaining top performance. These tools provide the flexibility and control needed to achieve cost-effective scalability. Ready to take control of your cloud costs? Sign up for a free trial today or spin up a cluster to get the performance, availability, and cost efficiency you need.
Announcing Hybrid Search Support for LlamaIndex
MongoDB is excited to announce enhancements to our LlamaIndex integration. By combining MongoDB’s robust database capabilities with LlamaIndex’s innovative framework for context-augmented large language models (LLMs), the enhanced MongoDB-LlamaIndex integration unlocks new possibilities for generative AI development. Specifically, it supports vector (powered by Atlas Vector Search ), full-text (powered by Atlas Search ), and hybrid search, enabling developers to blend precise keyword matching with semantic search for more context-aware applications, depending on their use case. Building AI applications with LlamaIndex LlamaIndex is one of the world’s leading AI frameworks for building with LLMs. It streamlines the integration of external data sources, allowing developers to combine LLMs with relevant context from various data formats. This makes it ideal for building application features like retrieval-augmented generation (RAG), where accurate, contextual information is critical. LlamaIndex empowers developers to build smarter, more responsive AI systems while reducing the complexities involved in data handling and query management. Advantages of building with LlamaIndex include: Simplified data ingestion with connectors that integrate structured databases, unstructured files, and external APIs, removing the need for manual processing or format conversion. Organizing data into structured indexes or graphs , significantly enhancing query efficiency and accuracy, especially when working with large or complex datasets. An advanced retrieval interface that responds to natural language prompts with contextually enhanced data, improving accuracy in tasks like question-answering, summarization, or data retrieval. Customizable APIs that cater to all skill levels—high-level APIs enable quick data ingestion and querying for beginners, while lower-level APIs offer advanced users full control over connectors and query engines for more complex needs. MongoDB's LlamaIndex integration Developers are able to build powerful AI applications using LlamaIndex as a foundational AI framework alongside MongoDB Atlas as the long term memory database. With MongoDB’s developer-friendly document model and powerful vector search capabilities within MongoDB Atlas, developers can easily store and search vector embeddings for building RAG applications. And because of MongoDB’s low-latency transactional persistence capabilities, developers can do a lot more with MongoDB integration in LlamIndex to build AI applications in an enterprise-grade manner. LlamaIndex's flexible architecture supports customizable storage components, allowing developers to leverage MongoDB Atlas as a powerful vector store and a key-value store. By using Atlas Vector Search capabilities, developers can: Store and retrieve vector embeddings efficiently ( llama-index-vector-stores-mongodb ) Persist ingested documents ( llama-index-storage-docstore-mongodb ) Maintain index metadata ( llama-index-storage-index-store-mongodb ) Store Key-value pairs ( llama-index-storage-kvstore-mongodb ) Figure adapted from Liu, Jerry and Agarwal, Prakul (May 2023). “Build a ChatGPT with your Private Data using LlamaIndex and MongoDB”. Medium. https://medium.com/llamaindex-blog/build-a-chatgpt-with-your-private-data-using-llamaindex-and-mongodb-b09850eb154c Adding hybrid and full-text search support Developers may use different approaches to search for different use cases. Full-text search retrieves documents by matching exact keywords or linguistic variations, making it efficient for quickly locating specific terms within large datasets, such as in legal document review where exact wording is critical. Vector search, on the other hand, finds content that is ‘semantically’ similar, even if it does not contain the same keywords. Hybrid search combines full-text search with vector search to identify both exact matches and semantically similar content. This approach is particularly valuable in advanced retrieval systems or AI-powered search engines, enabling results that are both precise and aligned with the needs of the end-user. It is super simple for developers to try out powerful retrieval capabilities on their data and improve the accuracy of their AI applications with this integration. In the LlamaIndex integration, the MongoDBAtlasVectorSearch class is used for vector search. All you have to do is enable full-text search, using VectorStoreQueryMode.TEXT_SEARCH in the same class. Similarly, to use Hybrid search, enable VectorStoreQueryMode.HYBRID . To learn more, check out the GitHub repository . With the MongoDB-LlamaIndex integration’s support, developers no longer need to navigate the intricacies of Reciprocal Rank Fusion implementation or to determine the optimal way to combine vector and text searches—we’ve taken care of the complexities for you. The integration also includes sensible defaults and robust support, ensuring that building advanced search capabilities into AI applications is easier than ever. This means that MongoDB handles the intricacies of storing and querying your vectorized data, so you can focus on building! We’re excited for you to work with our LlamaIndex integration. Here are some resources to expand your knowledge on this topic: Check out how to get started with our LlamaIndex integration Build a content recommendation system using MongoDB and LlamaIndex with our helpful tutorial Experiment with building a RAG application with LlamaIndex, OpenAI, and our vector database Learn how to build with private data using LlamaIndex, guided by one of its co-founders
Introducing: Multi-Kubernetes Cluster Deployment Support
Resilience and scalability are critical for today's production applications. MongoDB and Kubernetes are both well known for their ability to support those needs to the highest level. TA single Kubernetes cluster is typically limited to a single region, so to better enable developers using MongoDB and Kubernetes, we’ve introduced a series of updates and capabilities that makes it possible to manage MongoDB across multiple Kubernetes clusters. Since those Kubernetes clusters can be located in different regions, this offers new levels of resilience and control over where your data lives. In addition to the previously released support for running MongoDB replica sets and Ops Manager across multiple Kubernetes clusters, we're excited to announce the public preview release of support for Sharded Clusters spanning multiple Kubernetes clusters (GA to follow in November 2024). Support for deployment across multiple Kubernetes clusters is facilitated through the Enterprise Kubernetes Operator. As a recap for anyone unaware, the Enterprise Operator automates the deployment, scaling, and management of MongoDB clusters in Kubernetes. It simplifies database operations by handling tasks such as configuration, resizing, upgrades, and failover, whilst ensuring consistent performance and reliability in the Kubernetes environment. Multi-Kubernetes cluster deployment support enhances availability, resilience, and scalability for critical MongoDB workloads, empowering developers to efficiently manage these workloads within Kubernetes. This approach unlocks the highest level of availability and resilience by allowing shards to be located closer to users and applications, increasing geographical flexibility and reducing latency for globally distributed applications. Deploying replica sets across multiple Kubernetes clusters MongoDB replica sets are engineered to ensure high availability, data redundancy, and automated failover in database deployments. A replica set consists of multiple MongoDB instances—one primary and several secondary nodes—all maintaining the same dataset. The primary node handles all write operations, while the secondary nodes replicate the data and are available to take over as primary if the original primary node fails. This architecture is critical for maintaining continuous data availability, especially in production environments where downtime can be costly. Support for deploying MongoDB replica sets across multiple Kubernetes clusters helps remove the Kubernetes cluster itself as a single point of failure. Deploying MongoDB replica sets across multiple Kubernetes clusters enables you to distribute your data, not only across nodes in the Kubernetes cluster, but across different clusters and geographic locations, ensuring your deployments operational (even if one or more Kubernetes clusters or locations fail) and facilitating faster disaster recovery. To learn more about how to deploy replica sets across multiple Kubernetes clusters using the Enterprise Kubernetes Operator, visit our documentation . Sharding MongoDB across multiple Kubernetes clusters While replica sets duplicate data for resilience (and higher read rates), MongoDB sharded clusters divide the data up between shards, each of which is effectively a replica set, providing resilience for each portion of the data. Crucially, this also helps your database handle large datasets and higher-throughput operations since each shard has a primary member handling write operations to that portion of the data; this allows MongoDB to scale up the write throughput horizontally, rather than requiring vertical scaling of every member of a replica set. In a Kubernetes environment, each shard can now be deployed across multiple Kubernetes clusters, giving every shard higher resilience in the event of a loss of a Kubernetes cluster or an entire geographic location. This also offers the ability to locate shards or their primaries in the same region as the applications or users accessing that portion of the data, reducing latency and improving user experience. Sharding is particularly useful for applications with large datasets and those requiring high availability and resilience as they grow. Support for sharding MongoDB across multiple Kubernetes clusters is currently in public preview and will be generally available in November. Deploying Ops Manager across multiple Kubernetes clusters Ops Manager is the self-hosted management erver that supports automation, monitoring, and backup of MongoDB on your own infrastructure. Ops Manager's most critical function is backup of MongoDB deployments, and deploying it across multiple Kubernetes clusters greatly improves resilience and disaster recovery for your MongoDB deployments in Kubernetes. With Ops Manager distributed across several Kubernetes clusters, you can ensure that backups of deployments remain robust and available, even if one Kubernetes cluster or site fails. Furthermore, it allows Ops Manager to efficiently manage and monitor MongoDB deployments that are themselves distributed across multiple clusters, improving central oversight of your deployments. To learn more about how to deploy Ops Manager across multiple Kubernetes clusters using the Enterprise Kubernetes Operator, visit our documentation . To leverage multi-Kubernetes-cluster support, you can get started with the Enterprise Kubernetes Operator .
Introducing Dark Mode for MongoDB Documentation
We’re excited to announce a highly requested feature: Dark mode is now available for MongoDB Documentation ! Every day, developers from all backgrounds—beginners to experts—turn to the MongoDB Documentation. It’s packed with comprehensive resources that help you build modern applications using MongoDB and the Atlas developer data platform. With detailed information and step-by-step guides, it’s an invaluable tool for improving your skills and making your development work smoother. From troubleshooting tricky queries to exploring new features, MongoDB Documentation is there to support your projects and help you succeed. With dark mode, you can now switch to a darker interface that’s easier on the eyes. Whether you’re working late or prefer a subdued color palette, dark mode enhances your MongoDB Documentation experience. How to enable dark mode Enabling dark mode is simple. Just click on the sun and moon icon at the top right of the page to switch between dark mode, light mode, and system settings. It will initially default to your system settings. This is a personal setting and won't affect other users within the project or organization. We’ve designed dark mode to provide the same user-friendly experience you’re used to and stay consistent across different tools in the developer workflow, including MongoDB Atlas, which is also available in dark mode . We're all about making your reading experience top-notch! Dark mode is here because you asked for it through our feedback widget on the Docs page. Whether you’re an early adopter of dark mode or just trying it out, we’d love your opinion. Just drop your feedback in the widget next to the color theme selector on the MongoDB Documentation page. Less strain, more gain Dark mode offers a sleek, modern look that brings a refreshing change from the traditional light mode. Beyond its stylish appearance, dark mode also provides significant practical benefits. Reducing the amount of bright light emitted from your screen helps minimize eye strain and fatigue, making extended periods of device use more comfortable. For those using OLED screens, dark mode can help conserve battery life, as these screens consume less power by displaying darker pixels. Whether you’re coding into the late hours or just looking for a more comfortable viewing experience, dark mode is a simple yet powerful tool to enhance your development experience. Try out dark mode on MongoDB Documentation today and enjoy a more comfortable, stylish, and efficient reading experience!
Vector Quantization: Scale Search & Generative AI Applications
Update 12/12/2024: The upcoming vector quantization capabilities mentioned at the end of this blog post are now available in public preview: Support for ingestion and indexing of binary (int1) quantized vectors: gives developers the flexibility to choose and ingest the type of quantized vectors that best fits their requirements. Automatic quantization and rescoring: provides a native mechanism for scalar quantization and binary quantization with rescoring, making it easier for developers to implement vector quantization entirely within Atlas Vector Search. View the documentation to get started. We are excited to announce a robust set of vector quantization capabilities in MongoDB Atlas Vector Search . These capabilities will reduce vector sizes while preserving performance, enabling developers to build powerful semantic search and generative AI applications with more scale—and at a lower cost. In addition, unlike relational or niche vector databases, MongoDB’s flexible document model—coupled with quantized vectors—allows for greater agility in testing and deploying different embedding models quickly and easily. Support for scalar quantized vector ingestion is now generally available, and will be followed by several new releases in the coming weeks. Read on to learn how vector quantization works and visit our documentation to get started! The challenges of large-scale vector applications While the use of vectors has opened up a range of new possibilities , such as content summarization and sentiment analysis, natural language chatbots, and image generation, unlocking insights within unstructured data can require storing and searching through billions of vectors—which can quickly become infeasible. Vectors are effectively arrays of floating-point numbers representing unstructured information in a way that computers can understand (ranging from a few hundred to billions of arrays), and as the number of vectors increases, so does the index size required to search over them. As a result, large-scale vector-based applications using full-fidelity vectors often have high processing costs and slow query times, hindering their scalability and performance. Vector quantization for cost-effectiveness, scalability, and performance Vector quantization, a technique that compresses vectors while preserving their semantic similarity, offers a solution to this challenge. Imagine converting a full-color image into grayscale to reduce storage space on a computer. This involves simplifying each pixel's color information by grouping similar colors into primary color channels or "quantization bins," and then representing each pixel with a single value from its bin. The binned values are then used to create a new grayscale image with smaller size but retaining most original details, as shown in Figure 1. Figure 1: Illustration of quantizing an RGB image into grayscale Vector quantization works similarly, by shrinking full-fidelity vectors into fewer bits to significantly reduce memory and storage costs without compromising the important details. Maintaining this balance is critical, as search and AI applications need to deliver relevant insights to be useful. Two effective quantization methods are scalar (converting a float point into an integer) and binary (converting a float point into a single bit of 0 or 1). Current and upcoming quantization capabilities will empower developers to maximize the potential of Atlas Vector Search. The most impactful benefit of vector quantization is increased scalability and cost savings through reduced computing resources and efficient processing of vectors. And when combined with Search Nodes —MongoDB’s dedicated infrastructure for independent scalability through workload isolation and memory-optimized infrastructure for semantic search and generative AI workloads— vector quantization can further reduce costs and improve performance, even at the highest volume and scale to unlock more use cases. "Cohere is excited to be one of the first partners to support quantized vector ingestion in MongoDB Atlas,” said Nils Reimers, VP of AI Search at Cohere. “Embedding models, such as Cohere Embed v3, help enterprises see more accurate search results based on their own data sources. We’re looking forward to providing our joint customers with accurate, cost-effective applications for their needs.” In our tests, compared to full-fidelity vectors, BSON-type vectors —MongoDB’s JSON-like binary serialization format for efficient document storage—reduced storage size by 66% (from 41 GB to 14 GB). And as shown in Figures 2 and 3, the tests illustrate significant memory reduction (73% to 96% less) and latency improvements using quantized vectors, where scalar quantization preserves recall performance and binary quantization’s recall performance is maintained with rescoring–a process of evaluating a small subset of the quantized outputs against full-fidelity vectors to improve the accuracy of the search results. Figure 2: Significant storage reduction + good recall and latency performance with quantization on different embedding models Figure 3: Remarkable improvement in recall performance for binary quantization when combining with rescoring In addition, thanks to the reduced cost advantage, vector quantization facilitates more advanced, multiple vector use cases that would have been too computationally-taxing or cost-prohibitive to implement. For example, vector quantization can help users: Easily A/B test different embedding models using multiple vectors produced from the same source field during prototyping. MongoDB’s document model —coupled with quantized vectors—allows for greater agility at lower costs. The flexible document schema lets developers quickly deploy and compare embedding models’ results without the need to rebuild the index or provision an entirely new data model or set of infrastructure. Further improve the relevance of search results or context for large language models (LLMs) by incorporating vectors from multiple sources of relevance, such as different source fields (product descriptions, product images, etc.) embedded within the same or different models. How to get started, and what’s next Now, with support for the ingestion of scalar quantized vectors, developers can import and work with quantized vectors from their embedding model providers of choice (such as Cohere, Nomic, Jina, Mixedbread, and others)—directly in Atlas Vector Search. Read the documentation and tutorial to get started. And in the coming weeks, additional vector quantization features will equip developers with a comprehensive toolset for building and optimizing applications with quantized vectors: Support for ingestion of binary quantized vectors will enable further reduction of storage space, allowing for greater cost savings and giving developers the flexibility to choose the type of quantized vectors that best fits their requirements. Automatic quantization and rescoring will provide native capabilities for scalar quantization as well as binary quantization with rescoring in Atlas Vector Search, making it easier for developers to take full advantage of vector quantization within the platform. With support for quantized vectors in MongoDB Atlas Vector Search, you can build scalable and high-performing semantic search and generative AI applications with flexibility and cost-effectiveness. Check out these resources to get started documentation and tutorial . Head over to our quick-start guide to get started with Atlas Vector Search today.