Leveraging BigQuery JSON for Optimized MongoDB Dataflow Pipelines

We're delighted to introduce a major enhancement to our Google Cloud Dataflow templates for MongoDB Atlas. By enabling direct support for JSON data types, users can now seamlessly integrate their MongoDB Atlas data into BigQuery, eliminating the need for complex data transformations.

This streamlined approach not only saves users time and resources, but it also empowers customers to unlock the full potential of their data through advanced data analytics and machine learning.

Screenshot of the dashboard for Dataflow Templates and the option for picking the JSON format.
Figure 1: JSON feature for user options on Dataflow Templates

Limitations without JSON support

Traditionally, Dataflow pipelines designed to handle MongoDB Atlas data often necessitate the transformation of data into JSON strings or flattening complex structures to a single level of nesting before loading into BigQuery. Although this approach is viable, it can result in several drawbacks:

  • Increased latency: The multiple data conversions required can lead to increased latency and can significantly slow down the overall pipeline execution time.

  • Higher operational costs: The extra data transformations and storage requirements associated with this approach can lead to increased operational costs.

  • Reduced query performance: Flattening complex document structures in JSON String format can impact query performance and make it difficult to analyze nested data.

So, what’s new?

BigQuery's Native JSON format addresses these challenges by enabling users to directly load nested JSON data from MongoDB Atlas into BigQuery without any intermediate conversions.

This approach offers numerous benefits:

  • Reduced operating costs: By eliminating the need for additional data transformations, users can significantly reduce operational expenses, including those associated with infrastructure, storage, and compute resources.

  • Enhanced query performance: BigQuery's optimized storage and query engine is designed to efficiently process data in Native JSON format, resulting in significantly faster query execution times and improved overall query performance.

  • Improved data flexibility: users can easily query and analyze complex data structures, including nested and hierarchical data, without the need for time-consuming and error-prone flattening or normalization processes.

A significant advantage of this pipeline lies in its ability to directly leverage BigQuery's powerful JSON functions on the MongoDB data loaded into BigQuery. This eliminates the need for a complex and time-consuming data transformation process. The JSON data within BigQuery can be queried and analyzed using standard BQML queries.

Whether you prefer a streamlined cloud-based approach or a hands-on, customizable solution, the Dataflow pipeline can be deployed either through the Google Cloud console or by running the code from the github repository.

Enabling data-driven decision-making

To summarize, Google’s Dataflow template provides a flexible solution for transferring data from MongoDB to BigQuery. It can process entire collections or capture incremental changes using MongoDB's Change Stream functionality. The pipeline's output format can be customized to suit your specific needs. Whether you prefer a raw JSON representation or a flattened schema with individual fields, you can easily configure it through the userOption parameter. Additionally, data transformation can be performed during template execution using User-Defined Functions (UDFs).

By adopting BigQuery Native JSON format in your Dataflow pipelines, you can significantly enhance the efficiency, performance, and cost-effectiveness of your data processing workflows. This powerful combination empowers you to extract valuable insights from your data and make data-driven decisions.

Follow the Google Documentation to learn how to set up the Dataflow templates for MongoDB Atlas and BigQuery.

Get started with MongoDB Atlas on Google Marketplace.

Learn more about MongoDB Atlas on Google Cloud on our product page.

← Previous

Checkpointers and Native Parent Child Retrievers with LangChain and MongoDB

MongoDB and LangChain, the company known for its eponymous large language model (LLM) application framework, are excited to announce new developments in an already strong partnership. Two additional enhancements have just been added to the LangChain codebase, making it easier than ever to build cutting-edge AI solutions with MongoDB. Checkpointer support In LangGraph, LangChain’s library for building stateful, multi-actor applications with LLMs, memory is provided through checkpointers . Checkpointers are snapshots of the graph state at a given point in time. They provide a persistence layer, allowing developers to interact and manage the graph’s state. This has a number of advantages for developers—human-in-the-loop, "memory" between interactions, and more. Figure adapted from “Launching Long-Term Memory Support in LangGraph”. LangChain Blog. Oct. 8, 2024. https://blog.langchain.dev/launching-long-term-memory-support-in-langgraph/ MongoDB has developed a custom checkpointer implementation, the " MongoDBSaver " class, that, with just a MongoDB URI (local or Atlas ), can easily store LangGraph state in MongoDB. By making checkpointers a first-class feature, developers can have confidence that their stateful AI applications built on MongoDB will be performant. That’s not all, since there are actually two new checkpointers as part of this implementation— one synchronous and one asynchronous . This versatility allows the new functionality to be even more versatile, and serving developers with a myriad of use cases. Both implementations include helpful utility functions to make using them painless, letting developers easily store instances of StateGraph inside of MongoDB. A performant persistence layer that stores data in an intuitive way will mean a better end-user experience and a more robust system, no matter what a developer is building with LangGraph. Native parent child retrievers Second, MongoDB has implemented a native parent child retriever inside LangChain. This approach enhances the performance of retrieval methods utilizing the retrieval-augmented Generation (RAG) technique by providing the LLM with a broader context to consider. In essence, we divide the original documents into relatively small chunks, embed each one, and store them in MongoDB. Using such small chunks (a sentence or a couple of sentences) helps the embedding models to better reflect their meaning. Now developers can use " MongoDBAtlasParentDocumentRetriever " to persist one collection for both vector and document storage. In this implementation, we can store both parent and child documents in a single collection while only having to compute and index embedding vectors for the chunks. This has a number of performance advantages because storing vectors with their associated documents means no need to join tables or worry about painful schema migrations. Additionally, as part of this work, MongoDB has also added a " MongoDBDocStore " class which provides many helpful utility functions. It is now easier than ever to use documents as a key-value store and insert, update, and delete them with ease. Taken together, these two new classes allow developers to take full advantage of MongoDB’s abilities. MongoDB and LangChain continue to be a strong pair for building agentic AI—combining performance and ease of development to provide a developer-friendly experience. Stay tuned as we build out additional functionality! To learn more about these LangChain integrations, here are some resources to get you started: Check out our tutorial . Experiment with checkpointers and native parent child retrievers to see their utility for yourself. Read the previous announcement with LangChain about AI Agents, Hybrid Search, and Indexing.

December 16, 2024
Next →

Commerce at Scale: Zepto Reduces Latency by 40% With MongoDB

Zepto is one of the fastest-growing Indian startups and a pioneer in introducing quick commerce to India. Quick commerce, sometimes referred to as “Q-commerce” is a new, faster form of e-commerce promising ultra-quick deliveries, typically in less than one hour. Founded in July 2021, Zepto has revolutionized the Indian grocery delivery industry, offering users a choice of over 15,000 products with a promised 10-minute delivery. Since its launch, the company has rapidly expanded its operations, recording 20% monthly growth and achieving annualized sales of $1.5 billion by July 2024. Zepto’s order processing and delivery system is instrumental in meeting its promise to customers. Zepto’s system routes new orders to a “dark store,” where bleeding-edge assignment systems help pack orders in under 75 seconds. A proprietary navigation system ensures riders can then deliver these orders promptly. As Zepto expanded, its monolithic infrastructure, based on a relational SQL database, could not achieve the scalability and operational efficiency the company needed. Zepto changed the game by turning to MongoDB Atlas . Mayank Agarwal, Senior Architect at Zepto, shared the company’s journey with MongoDB during a presentation at MongoDB.local Bengaluru in September 2024 . “We had a big monolith. All the components were being powered by PostgreSQL and a few Redis clusters,” said Agarwal. “As our business was scaling, we were facing a lot of performance issues, as well as restrictions in terms of the velocity at which we wanted to operate.” Zepto’s legacy architecture posed four key issues: Performance bottlenecks: As Zepto grew, the need for complex database queries increased. These queries required multiple joins, which put a significant strain on the system, resulting in high CPU usage and an inability to provide customers and delivery partners with accurate data. Latency: Zepto needed its API response times to be fast. However, as the system grew, background processing tasks slowed down. This led to delays and caused the system to serve stale data to customers. A need for real-time analytics: Teams on the ground, such as packers and riders, required real-time insights on stock availability and performance metrics. Building an extract, transform, and load (ETL) pipeline for this was both time-consuming and resource-intensive. Increased data scaling requirements: Zepto’s data was growing exponentially. Managing it efficiently became increasingly difficult, especially when real-time archival and retrieval were required. MongoDB Atlas meets Zepto’s goals “We wanted to break our monolith into microservices and move to a NoSQL database . But we wanted to evaluate multiple databases,” said Agarwal. Zepto was looking for a document database that would let its team query data even when the documents were structured in a nested fashion. The team also needed queryability on array-based attributes or columns. MongoDB fulfilled both use cases. “Very optimally, we were able to do some [proofs of concept]. The queries were very performant, given the required indexes we had created, and that gave us confidence,” said Agarwal. “The biggest motivation factor was when we saw that MongoDB provides in-memory caching , which could address our huge Redis cluster that we couldn’t scale further.” Beyond scalability, MongoDB Atlas also provided high reliability and several built-in capabilities. That helped Zepto manage its infrastructure day to day, and create greater efficiencies for both its end users and its technical team. Speaking alongside Agarwal at MongoDB.local Bengaluru, Kshitij Singh, Technical Lead for Zepto, explained: “When we discovered MongoDB Atlas, we saw that there were a lot of built-in features like the MongoDB chat support , which gave us very qualitative insights whenever we faced any issues. That was an awesome experience for us.” Data archival , sharding support , and real-time analytic capabilities were also key in helping the Zepto team improve operational efficiencies. With MongoDB, Zepto was able to deploy new features more quickly. Data storage at the document level meant less management overhead and faster time to market for new capabilities. Furthermore, MongoDB’s archival feature made it easier for Zepto to manage large datasets. The feature also simplified the setup of secondary databases for ETL pipelines, reducing the heavy lifting for developers. “You go on the MongoDB Atlas platform and can configure archival in just one click,” said Singh. Zepto reduces latency, handles six times more traffic, and more The results of migrating to MongoDB Atlas were immediate and significant: Zepto saw a 40% reduction in latency for some of its most critical APIs, which directly improved the customer experience. Postmigration, Zepto’s infrastructure could handle six times more traffic than before, without any degradation in performance. This scalability enabled the company to continue its rapid growth without bottlenecks. Page load times improved by 14% , leading to higher conversion rates and increased sales. MongoDB’s support for analytical nodes helped Zepto segregate customer-facing workloads from internal queries. This ensured that customer performance was never compromised by internal reporting or analytics. “MongoDB is helping us grow our business exponentially,” said Agarwal at the end of his presentation. Visit our product page to learn more about MongoDB Atlas.

December 17, 2024