MongoDB 8.0: Raising the Bar

I recently received an automated reminder that I was approaching a work anniversary, which took me somewhat by surprise. It’s hard to believe that it’s already been a year (to the day) that I joined MongoDB! So I thought I’d take a moment to reflect on my MongoDB journey so far, share some exciting product updates, and signal where we’re headed next.

Our customers

I joined MongoDB because it built a product developers love. The innovation of MongoDB’s document model empowered developers to simply build. No longer encumbered by having to formalize and denormalize their data schema before their application was even designed, MongoDB enabled developers to interact with data in an intuitive JSON format, and made it easy to evolve data structures as the life of their application evolved.

One of my first steps upon joining the company was to learn more about our customers. I was excited to learn that in addition to delighting developers, MongoDB had launched capabilities that enabled it to win mission-critical workloads from enterprise class customers—including 70% of the Fortune 100 and highly regulated global financial institutions, health care providers, and government agencies. I found it remarkable that customers could replicate data across AWS, Google Cloud, and Microsoft Azure in MongoDB Atlas (our fully-managed cloud database service) with just a few mouse clicks, and that some customers replicate data between the cloud and on premises using MongoDB Enterprise Advanced. This optionality struck me as powerful in the era of rapid advancements in AI, as it enables customers to easily bring their data to the best cloud provider for AI.

Soon after I joined MongoDB, the team was firming up the development roadmap for the next version of MongoDB, and they asked for my input on the plan. The team was debating whether to focus on features developers would love, or governance capabilities required by large enterprises. I knew that ideally we would please all of our customers, so we had to try to make this an “and” and not an “or.” While I was new to MongoDB, from my 17+ years at AWS I learned that all customers demand security, durability, availability, and performance (in that order) from any modern technology offering. If a product or service doesn’t have those four elements, customers won’t buy whatever you’re selling. So as a team, we agreed that our next release of MongoDB—MongoDB 8.0—had to raise the bar for all of our customers, delivering great security, durability, availability, and performance.

The plan

We had less than a year before our target launch, so we knew we had to get moving, fast. My team and I brought MongoDB’s product and engineering organizations together to align on the plan for our next release. We set goals around delivering significant improvements in security, durability, and availability. And we set a line in the sand—that we weren’t going to release MongoDB 8.0 unless it was the best-performing version of MongoDB yet.

Measuring the performance of a feature-rich database like MongoDB can be tricky, as customers run a wide range of workloads. So we decided to run a suite of benchmarks to simulate customer workloads. We also developed Andon cord-inspired automation that would automatically roll back any code contributions that regressed our performance metrics. Finally, a set of senior engineering leaders met regularly to review our progress and immediately escalated any blockers that could jeopardize our launch, so that we could quickly fix things.

From my experience, I knew that great teams really respond when they’re given clear goals, and when they’re empowered to innovate, so I was excited to see what they would come up with. I’m proud to say that our product and engineering teams rose to the challenge.

Announcing MongoDB 8.0

Today, I’m thrilled to announce the general availability of MongoDB 8.0—the most secure, durable, available, and performant version of MongoDB yet! The team came up with architectural optimizations in MongoDB 8.0 that have significantly reduced memory usage and query times, and have made batch processing more efficient than previous versions.

Specifically, MongoDB 8.0 features:

  • 36% better read throughput

  • 56% faster bulk writes

  • 20% faster concurrent writes during data replication

  • 200% faster on complex aggregations of times series data

In making these improvements, we're seeing benchmarks for typical web applications perform 32% better overall. Here’s a breakdown of how MongoDB 8.0 performs against some of our benchmarks:

Image depicting the performance improvements of MongoDB 8.0. In order going down the image, YCSB Bulk Load is 56% faster, YCSB 100% Read is 36% faster, TCSV 95/5 is 32% faster, Linkbench is 24% faster, and TSBS is 60% faster.

Improved performance benefits all users of applications built atop MongoDB, and for MongoDB customers, it can mean reduced costs (due to an improved price/performance ratio).

In addition to significant performance gains, MongoDB 8.0 delivers a wide range of improvements, including (but not limited to):

  • Improving availability by delivering sharding enhancements to distribute data across shards up to 50 times faster and at up to 50% lower starting cost, with reduced need for additional configuration or setup.

  • Improving support for a wide range of search and AI applications at higher scale and lower cost, via the delivery of quantized vectors—compressed representations of full-fidelity vectors—that require up to 96% less memory and are faster to retrieve while preserving accuracy.

  • Enabling customers to encrypt data at rest, in transit, and in use by expanding MongoDB’s Queryable Encryption to also support range queries. Queryable Encryption is a groundbreaking, industry-first innovation developed by the MongoDB Cryptography Research Group that allows customers to encrypt sensitive application data, store it securely as fully randomized encrypted data in the MongoDB database, and run expressive queries on the encrypted data—with no cryptography expertise required.

You might wonder why we’re so confident that customers are going to love MongoDB 8.0. Well, we’ve been acting as our own customer, and have moved our own applications over to 8.0. This approach is generally called “dogfooding,” but we think that “eating our own pizza” sounds more appetizing. Our internal build system—which our software developers use daily—is built atop MongoDB, and when we upgraded to MongoDB 8.0 we saw query latencies drop by approximately 75%! This was a double win, as it improved the performance of our own tooling, and it set our performance chat room abuzz with excitement in anticipation of delighting external customers. While results may vary based on your particular workload, the point is that we just couldn’t wait to share MongoDB 8.0’s performance gains with customers.

Graph showing different latency trends across different dates. The Y axis represents the amount of latency and the x axis marks the date.

Indeed, customers are also already seeing great results on MongoDB 8.0. For example, Felix Horvat, Chief Technology Officer at OCELL, a climate technology company in Germany, said: “With MongoDB 8.0, we have seen an incredible boost in performance, with some of our queries running twice as fast as before. This improvement not only enhances our data processing capabilities but also aligns perfectly with our commitment to resource efficiency. By optimizing our backend operations, we can be more effective in our climate initiatives while conserving resources—a true reflection of our dedication to sustainable solutions.”

I encourage you to check out MongoDB 8.0 yourself. It’s available today via MongoDB Atlas, as part of MongoDB Enterprise Advanced for on-premises and hybrid deployments, and as a free download from mongodb.com/try with MongoDB Community Edition. In addition, customers upgrading from previous versions of MongoDB to 8.0 can find helpful upgrade guides on mongodb.com.

What’s next?

We’re excited for you to try MongoDB 8.0 and to share your feedback, as customer feedback helps us guide our roadmap for future releases.

Going forward, please watch this space. Over the next few weeks, we’ll be publishing a series of engineering blog posts that dig into MongoDB’s investments in the technology behind MongoDB 8.0. We’re also planning posts about horizontal scaling in MongoDB 8.0, and one that will look closely at queryable encryption (QE), but let me know what you’d like to hear more about.

It’s been an exciting year at MongoDB—I can’t wait to see what the next one has in store!

–Jim

← Previous

Bringing Gen AI Into The Real World with Ramblr and MongoDB

How do you bring the benefits of gen AI, a technology typically experienced on a keyboard and screen, into the physical world? That's the problem the team at Ramblr.ai , a San Francisco-based startup, is solving with its powerful and versatile 3D annotation and recognition capabilities. “With Ramblr you can record continuously what you are doing, and then ask the computer, in natural language, ‘Where did I go wrong’ or ‘What should I do next?” said Frank Angermann, Lead Pipeline & Infrastructure Engineer at Ramblr.ai. Gen AI for the real world One of the best examples of Ramblr’s technology, and its potential, is its work with the international chemical giant BASF. In a video demonstration on Ramblr’s website, a BASF engineer can be seen tightening bolts on a connector (or ‘flange’) joining two parts of a pipeline. Every move the engineer makes is recorded via a helmet-mounted camera. Once the worker is finished for the day this footage, and the footage of every other person working on the pipeline, is uploaded to a database. Using Ramblr’s technology, quality assurance engineers from BASF then query the collected footage from every worker, asking the software to, ‘Please assess footage from today’s pipeline connection work and see if any of the bolts were not tightened enough.’ Having processed the footage, Ramblr assesses whether those flanges had been assembled correctly and identifies any that required further inspection or correction. The method behind the magic “We started Ramblr.ai as an annotation platform, a place where customers could easily label images from a video and have machine learning models then identify that annotation throughout the video automatically,” said Frank. “In the past this work would be carried out manually by thousands of low-paid workers tagging videos by hand. We thought we could be better by automating that process,” he added. The software allows customers to easily customize and add annotations to footage for their particular use case, and with its gen-AI powered active learning approach Ramblr then ‘fills in’ the rest of the video based on those annotations. Why MongoDB? MongoDB has been part of the Ramblr technology stack since the beginning. “We use MongoDB Atlas for half of our storage processes. Metadata, annotation data, etc., can all be stored in the same database. This means we don’t have to rely on separate databases to store different types of data,” said Frank. Flexibility of data storage was also a key consideration when choosing a database. “With MongoDB Atlas, we could store information the way we wanted to,” he added. The built-in vector database capabilities of Atlas were also appealing to the Rambler team, “The ability to store vector embeddings without having to do any more work - for instance not having to move a 3mb array of data somewhere else to process it, was a big bonus for us.” The future Aside from infrastructure and construction Q&A, robotics is another area in which the Ramblr team is eager to deploy their technology. “Smaller robotics companies don’t typically have the data to train the models that inform their products. There are quite a few use cases where we could support these companies and provide a more efficient and cost-effective way to teach the robots more efficiently. We are extremely efficient in providing information for object detectors,” said Frank. But while there are plenty of commercial uses for Ramblr’s technology, the growth in spatial computing in the consumer sector - especially following the release of Apple’s Vision Pro and Meta Quest headsets - opens up a whole new category of use cases. “Spatial computing will be a big part of the world. Being able to understand the particular processes, taxonomy, and what the person is actually seeing in front of them will be a vital part of the next wave of innovation in user interfaces and the evolution of gen AI,” Frank added. Are you building AI apps? Join the MongoDB AI Innovators Program today! Successful participants gain access to free Atlas credits, technical enablement, and invaluable connections within the broader AI ecosystem. If your company is interested in being featured, we’d love to hear from you. Connect with us at ai_adopters@mongodb.com. Head over to our quick-start guide to get started with Atlas Vector Search today.

September 30, 2024
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024