Ways to Integrate MongoDB Atlas in Your DevOps Processes

Darshana Paithankar
October 11, 2023 | Updated: September 27, 2024
#AtlasCLI#DevOps

MongoDB Atlas - the industry-leading developer data platform integrates all of the data services you need to build modern applications in a unified developer experience. We want to meet you where you are and offer various ways to begin with Atlas and make the most of its features. Starting with the Atlas User Interface is a good initial step. Now what if your requirement is to automate the deployment of Atlas clusters at scale, and by leveraging tools that are already integral to your application ecosystem? Through this blog, we will address this question.

We will discuss the programmatic methods for starting with Atlas and deploying Atlas resources and infrastructure according to your specific needs. The tools we provide cover most of the control plane management ( easily deploy, manage, scale) Atlas clusters. Thereby, enabling you to create the building blocks of Atlas such as clusters, database users, projects, Atlas Search indexes, backups, alerts, and more. You can interact directly with the data plane through the command line or programmatically using tools like MongoDB Shell, Compass, language drivers, etc., to work with data and perform CRUD operations.

1. Atlas Administration API: Leveraging your choice of client

No matter what your choice of client: be it cURL commands, Postman, Insomnia etc. You can use a client of your choice to directly interact with Atlas through the Atlas Administration API. The Atlas Administration API gives you a RESTful interface to interact with Atlas resources and perform various actions within MongoDB Atlas. Each endpoint represents a specific resource in Atlas ( e.g. cluster). You can programmatically deploy and manage all of the Atlas resources from an administrative standpoint such as creating a cluster, database user, projects, advanced clusters, backups, monitoring etc.

All of the tools that we will cover below are built on top of the Atlas Administration API and abstract away the complexities of using the Atlas Atlas Administration API directly.

2. GoSDK Client: A simplified way to get started with the Atlas Admin API

One of the mechanisms that simplify the interaction with the API is the availability of SDK (Software Development Kit). If you are a Go developer, the GoSDK client gives you a much simpler experience of getting started with the Atlas Administration API. It also has full endpoint coverage of the Administration API and improves the speed of getting started with the Atlas Administration API. Getting started with the Admin API through the Go SDK client only takes a few lines of code since the SDK includes pre-built functions, structs, and methods that encapsulate the complexity of HTTP requests, authentication, error handling, versioning, and other low-level details. We will be adding SDK support for more languages in the future. Feel free to add feedback on what other languages you would like to see supported.

3. MongoDB Atlas CLI: Simple command-line tool to easily deploy Atlas resources

If you prefer to manage your Atlas resources using simple commands in the terminal, the MongoDB Atlas Command-Line Interface (CLI) is your answer. With the Atlas CLI, you can seamlessly manage your clusters, automate database user creation, control network access, and perform various other administrative tasks, all from the command line. You can also script these actions out with the CLI for even easier repeatability. It is available for multiple operating systems, including Windows, Linux, and macOS. It is often used in conjunction with other command-line tools to automate workflows and integrate with CI/CD pipelines. An easy way to get started with the Atlas CLI is through the quickstart.

4. Infrastructure as Code (IaC) integrations: Automating deployment of Atlas using Infrastructure as Code tools

There are several advantages of using Infrastructure as Code ( IaC) tools to provision application infrastructure. By treating your infrastructure code similar to your application code, IaC tools offer benefits such as version control, scalability, security, repeatability etc. What if you could easily deploy your Atlas resources using your preferred IaC tools? MongoDB Atlas gives you that flexibility. Whether you're an AWS CloudFormation or a HashiCorp Terraform enthusiast, you can provision and manage Atlas resources using these tools with ease through the integrations we offer:

AWS CloudFormation integration: MongoDB Atlas supports three ways to provision resources using AWS CloudFormation

  • The first is by leveraging Atlas resources directly from the CloudFormation Public Registry. We have 33+ resources available today. The configurations are defined in JSON/ YAML and can be executed using the AWS CLI/ AWS management console.
  • If you want a faster way to get started, you can explore our AWS Partner Solutions ( formerly known as quickstarts), which have pre-built Cloudformation templates to help you provision a group of Atlas resources for specific use cases instead of deploying them one by one.
  • And if you are passionate about using languages you are comfortable with such as JavaScript/TypeScript, Python, Java, C#, and Go instead of learning YAML/JSON, you can leverage the AWS Cloud Development Kit (CDK) to deploy Atlas resources. Under the hood, when your AWS CDK applications are run, they translate your code to CloudFormation templates which use the CloudFormation service for provisioning.

HashiCorp Terraform integration: If you are already using Terraform as your IaC tool of choice, we have integrations with Hashicorp Terraform as well. There are two easy ways to get started with Hashicorp Terraform

  • By directly provisioning Atlas resources on AWS, Azure, and Google Cloud using the Terraform MongoDB Atlas provider.
  • If you prefer to use your favorite language, e.g. Typescript, Python, C#, Java, Go, you can use CDKTF (Cloud Development Kit for Terraform), which will allow you to deploy Atlas using Terraform under the hood without knowing the specifics of Terraform’s configuration language

5. Atlas Kubernetes Operator: Use your existing Kubernetes tooling to manage Atlas resources

For organizations leveraging Kubernetes for container orchestration, the Atlas Kubernetes Operator provides seamless integration between Kubernetes and MongoDB Atlas. This operator allows you to deploy and manage Atlas resources using your existing Kubernetes tooling, streamlining the process of spinning up and scaling databases alongside your applications. You can manage Atlas in exactly the same way you manage your applications running in Kubernetes. This is done by managing Atlas directly via custom resources in Kubernetes. These custom resources can be created, managed, and stored as the source of truth in your repository. You can then leverage a Continuous Deployment tool of your choice such as ArgoCD to apply them into Kubernetes.

Whether you prefer working with the command line, a RESTful API, Kubernetes, or IaC tools, MongoDB Atlas provides a diverse set of tools to help you achieve your automation goals. By embracing these methods, you can streamline your operations, improve efficiency, and pave the way for a more agile and responsive development process.

Learn more from our Atlas Programmatic Access documentation page

← Previous

Boost the Accuracy of E-commerce Search Results with Atlas Vector Search

Artificial Intelligence’s (AI) growth has led to transformative advancements in the retail industry, including natural language processing, image recognition, and data analysis. These capabilities are pivotal to enhancing the efficiency and accuracy of e-commerce search results. E-commerce, characterized by its vast product catalogs and diverse customer base, generates enormous amounts of data every day. From user preferences and search histories to product reviews and purchase patterns — and add to that images, video, and audio associated with product campaigns and user search — the data is both a goldmine and a challenge. Traditional search mechanisms, which rely on exact keyword matches, are inadequate at handling such nuanced and voluminous data. This is where vector search comes into play as the perfect data mining tool . As a sophisticated search mechanism, it leverages AI-driven algorithms to understand the intrinsic relationships between data points. This enables it to discern complex patterns, similarities, and contexts that conventional keyword-based searches might overlook. Let’s dig deeper into the differences between traditional keyword matching search and vector search, and answer questions like: What type of queries does vector search improve in the retail search landscape? What are the challenges associated with it? And how can your business tap into the competitive advantage it represents? Check out our AI resource page to learn more about building AI-powered apps with MongoDB. Traditional Keyword Matching vs. Vector Search Traditional search functionalities for e-commerce platforms — keyword matching, typo tolerance, autocomplete, highlighting, facets, and scoring — are often built in-house or implemented on top of typical search engines like Apache Lucene, AtlasSearch, or ElasticSearch, relying heavily on metadata textual descriptions. While this has served the industry well for years, it often falls short of understanding the nuanced needs of modern consumers. For instance, a customer might be looking for a "blue floral summer dress," but if the product description lacks these terms, it might not appear in the search results, even if it perfectly matches the visual description. Figure 1: As embeddings encode numerically the meaning of documents, semantically close documents will be geometrically close as well. Vector search is a method that finds similar items in a dataset based on their vector representations, and offers a more efficient and accurate way to sift through large datasets. Instead of relying on exact matches, it uses mathematical techniques to measure the similarity between vectors, allowing it to retrieve items that are semantically similar to the user's query, even if the query and the item descriptions don't contain exact keyword matches. Figure 2: Data flow diagram showcasing how applications, vector embedding algorithms, and search engines work together at a high level. One great thing about Vector search is that by encoding any type of data, i.e. text, images or sound, you can perform queries on top of that, creating a much more comprehensive way of improving the relevance of your search results. Let’s explore examples of queries that involve context, intent, and similarity. Visual similarity queries Query: "Find lipsticks in shades similar to this coral lipstick." Vector Search Benefit: Vector search can recognize the color tone and undertones of the specified lipstick and suggest similar shades from the same or different brands. Data type: image or text Contextual queries Query: "Affordable running shoes for beginners." Vector Search Benefit: Vector search can consider both the price range and the context of "beginners," leading to relevant shoe suggestions tailored to the user's experience level and budget. Data type: text, audio (voice) Natural language queries Query: "Show me wireless noise-canceling headphones under $100." Vector Search Benefit: Capture intent. Vector search can parse the query's intent to filter headphones with specific features (wireless, noise-canceling) and a price constraint, offering products that precisely match the request. Data type: text, audio (voice) Complementary product queries Query: "Match this dress with elegant heels and a clutch." Vector Search Benefit: Vector search can comprehend the user's request to create a coordinated outfit by suggesting shoes and accessories that complement the selected dress. Data type: text, audio (voice), image Challenging landscape, flexible stack Now that we've explored different queries and their associated data types that could be used in vector embeddings for search, we can see how much more information can be used to deliver more accurate results and fuel growth. Let’s consider some of the challenges associated with a vector search solution data workflow and how MongoDB Atlas Vector Search helps bridge the gap between challenges and opportunities. Data overload The sheer volume of products and user-generated data can be overwhelming, making it challenging to offer relevant search results. By embedding different types of data inputs like images, audio (voice), and text queries for later use with vector search, we can simplify this workload. Storing your vector encoding in the same shared operational data layer your applications are built on top of, but also generating search indexes based on those vectors, makes it simple to add context to your application search functionalities. Using Atlas Vector Search combined with MongoDB App Services , you can reduce operational overhead by creating a trigger that could “see” when a new document is created in your collections and automatically make the call to the embedding API of your preference, pushing the document to it and storing the retrieved embedding data in the same document stored in your collection. Figure 3: Storing vectors with the data simplifies the overall architecture of your application. As the number of documents or vectors grows, efficient indexing structures ensure that search performance remains reasonable. By simply creating an index based on the embedded data field, you can leverage the optimized retrieval of the data, reduce the computational load, and accelerate its performance, especially for nearest neighbor search tasks, where the goal is to find items that are most similar to a given query. Altogether, the combination of MongoDB Vector Search capabilities with App Services and indexing provides a robust and scalable solution to achieve real-time responsiveness. An indexed vector search database can provide rapid query results, making it suitable for applications like recommendation engines or live search interfaces. Changing consumer behavior Developing an effective vector search solution involves understanding the nuances of the retail domain. Retailers must consider factors like seasonality, trends, and user behavior to improve the accuracy of search results. To overcome this challenge, retailers will need to be able to adjust their business model by categorizing their product catalogs and user data according to different criteria, for example: So as you can see all this vast amount of information can be embedded to build more comprehensive criteria for relevance, but first it needs to be properly captured and organized. This is where the value of the flexible document model comes into play. The document model allows you to define different fields and attributes for each category of data. This can be used to capture the various categorization criteria. Retailers could also utilize embedded subdocuments to associate relevant information with products or customers. For instance, you can embed a subdocument containing marketing campaign data, engagement channels, and geographic location within products to track their performance. As categorization criteria evolve, dynamic schema evolution allows you to add or modify fields without disrupting existing data. This flexibility easily accommodates changing business needs. Retailers may also use embedded arrays to record purchase history for customers. Each array element can represent a transaction, including product details and purchase date, facilitating segmentation based on recency and frequency. By embedding all these different data types, and leveraging the flexible capabilities of the document model, retailers can create a comprehensive and dynamic system that effectively categorizes data according to diverse criteria in a fast and resilient way. This enables personalized search experiences and enhanced customer engagement in the e-commerce space. Sitting on a goldmine Every retailer worldwide now realizes that with their customer data, they are sitting on a goldmine. Using the proper enabling technologies would allow them to build better experiences for their customers while infusing their applications with automated, data-driven decision-making. Retailers offering more intuitive and contextual search results can ensure their customers find what they're looking for by personalizing the relevance of their search results, enhancing satisfaction, and increasing the likelihood of successful transactions. The future of e-commerce search lies in harnessing the power of technologies like Atlas Vector Search , as it’s not only another vector search database, but also an extended product for the developer data platform , providing them with an integrated set of data and application services. For retailers, the message is clear: to offer unparalleled shopping experiences, embracing and integrating vector search functionalities with a performant and reliant platform that simplifies your data organization and storage is not just beneficial, it's essential. Learn more and discover How to Implement Databricks Workflows and Atlas Vector Search for Enhanced E-commerce Search Accuracy with our developer guide, and check out our GitHub repository explaining the full code for deploying an AI-Enhanced e-commerce search solution Head over to our quick-start guide to get started with Atlas Vector Search today.

October 11, 2023
Next →

Hanabi Technologies Uses MongoDB to Power AI Assistant, Hana

For all the hype surrounding generative AI, cynics tend to view the few real-world implementations as little more than “fancy chatbots.” But for Abhinav Aggarwal, CEO of Hanabi Technologies, the idea of a generative AI-powered bot that is more than just an assistant was intriguing. “I’d been using ChatGPT since it launched,” said Aggarwal. “That got me thinking: How could we make a chatbot that was like a team member?” And with that concept, Hana was born. The problem with bots “Most generative AI chatbots do not act like people; they wait for a command and give a response,” said Aggarwal. “We wanted to create a human-like chatbot that would proactively help people based on what they wanted—automating reminders, for example, or fetching time zones from your calendar to correctly schedule meetings.” Hanabi’s flagship product, Hana, is an AI assistant designed to enhance team collaboration within Google Chat, working in concert with Google Workspace and its suite of products. “Our target customers are smaller companies of between 10 and 50 people. At this size you’re not going to build your own agent from scratch,” he said. Hana integrates with Google APIs to deliver a human-like assistant that chimes in with helpful interventions, such as automatically setting reminders and making sure meetings are booked in the right time zone for each participant. “Hana is designed to bring AI to smaller companies and help them collaborate in a space where they are already working—Google Workspace,” Aggarwal explained. The MongoDB Atlas solution For Hana to act like a member of the team, Hanabi needed to process massive amounts of data to support advanced features like retrieval-augmented generation (RAG) for better information retrieval across Google Docs and many other sources. And with a rapidly growing user base of over 600 organizations and 17,000+ installs, Hanabi also required a secure, scalable, and high-performing data storage solution. MongoDB Atlas provided a flexible document model, built-in vector database, and scalable cloud-based infrastructure, freeing Hanabi engineers to build new features for Hana rather than focusing on rote tasks like data extract, transform, and load processes or manual scaling and provisioning. Now, MongoDB Atlas handles a variety of responsibilities: Scalability and security: MongoDB Atlas’s auto-scaling and automatic backup features have enabled Hanabi to seamlessly grow its user base without the need for manual database management. RAG: MongoDB Atlas plays a critical role in Hana’s RAG functionality. The platform enables Hanabi to split Google Docs into small sections, create embeddings, and store these sections in Atlas’s vector database. Development Processes: According to Aggarwal, MongoDB’s flexibility in managing changing schemas has been essential to the company’s fast-paced development cycle. Data Visualization: Using MongoDB Atlas Charts has enabled Hanabi to create comprehensive dashboards for real-time data visualization. This has helped the team track usage, set reminders, and optimize performance without needing to build a manual dashboard. Impact and results With MongoDB Atlas, Hanabi can successfully scale Hana to meet the demands of its rapidly expanding user base. The integration is also enabling Hana to offer powerful features like automatic interactions with customers, advanced information retrieval from Google Docs, and manually added memory snippets, making it an essential tool for teams around the world. Next steps Hanabi plans to continue integrating more tools into Hana while expanding its reach to personal Gmail users. The company is also rolling out a new automatic-interaction feature, further enhancing Hana’s ability to proactively assist users without direct commands. MongoDB Atlas remains a key component of Hanabi’s stack, alongside Google Kubernetes Engine, NestJS, and LangChain, enabling Hanabi to focus on innovating to improve the customer experience. Tech Stack MongoDB Atlas Google Kubernetes Engine NestJS LangChain Are you building AI apps? Join the MongoDB AI Innovators Program today! Successful participants gain access to free MongoDB Atlas credits, technical enablement, and invaluable connections within the broader AI ecosystem. If your company is interested in being featured, we’d love to hear from you. Connect with us at ai_adopters@mongodb.com.

November 21, 2024