Rich text vector-databases

Background Wrapper /resources/basics/databases/vector-databases

Central to the functionality of a vector database is the principle of embeddings. In essence, a vector or embedding model translates data into a consistent format: vectors.

While a vector is fundamentally an ordered set of numbers, an embedding transforms it into a representation of various data types, including text, images, and audio.

Transformations — the process of converting data from one format to another — situate vectors in multi-dimensional vector space. One of the most striking features of this spatial arrangement is that data points with similar attributes or characteristics naturally gravitate toward each other, forming clusters.

Vector embeddings are not just numerical translations; they encapsulate the deeper semantic essence and the contextual nuances of the original data. This makes them invaluable assets for a range of AI applications — from natural language processing (NLP) to sentiment analysis to text categorization.

Querying a vector database is different than querying a conventional database. Instead of hunting for precise matches between identical vectors, a vector database uses similarity search to identify vectors that reside in close proximity to the given query vector, within the multi-dimensional space. This approach not only more closely aligns with the inherent nature of the data but also offers a speed and efficiency that traditional search can't match.

Words, sentences, and even entire documents can be transformed into vectors that capture their essence. For example, a standard word embedding method is Word2Vec. With Word2Vec, words with similar meanings are represented by vectors that are close in a multi-dimensional space. The most famous example is: king – man + woman = queen. Adding the vectors associated with the words “king” and “woman” while subtracting “man” is equal to the vector associated with “queen.”

Even with their intricate patterns and colors, images can be translated into vectors. For instance, in a dataset teeming with animal images, a trained convolutional neural network (CNN) would cluster all dog images close together, distinctly separate from, say, clusters of cats or birds.

​By capturing the inherent data structure, and patterns within the data, vector embeddings offer semantically enriched portrayals. This richness not only facilitates a deeper understanding of the data but also expedites computations related to determining relationships and gauging similarities between different entities.

​Vector search is critical for vector databases because of its distinct data retrieval method.

​Unlike traditional databases that rely on exact matches, in a vector database, vector search operates based on similarity. This semantic understanding means that even if two pieces of data aren't identical but are contextually or semantically similar, they can be matched.

​Traditional keyword searches excel when pinpointing specific terms within documents or tables. However, they fall short with unstructured data, such as videos, books, social media posts, PDFs, and audio files.

​Vector search fills this gap by enabling searches within unstructured data. It doesn't just look for exact matches but identifies content based on semantic similarity, understanding the inherent relationships between search terms.

The efficiency of vector search becomes evident when dealing with high-dimensional data. Vector databases are adept at handling data points that span hundreds or even thousands of dimensions. Algorithms optimized for vector search of high dimensional vectors, such as approximate nearest neighbor (ANN) search, can swiftly identify the most similar vectors in this vast space without the need to scan every vector. This efficiency translates to faster and more resource-effective searches.

From a user experience perspective, the benefits of vector search are manifold. Applications like recommendation systems or image recognition can provide results based on similarity rather than exact matches. For instance, in an e-commerce setting, showing products similar to a user's search query can enhance customer satisfaction and increase sales. As datasets expand, the scalability of vector search becomes evident. While exact match searches might become progressively slower with growing data, vector search maintains consistent query performance throughout, ensuring timely results even with vast datasets.

The flexibility offered by vector search is another notable advantage. It accommodates new data types, evolving data structures, and changing search requirements with minimal adjustments.

Moreover, flexibility is invaluable in the rapidly evolving data management landscape, especially as many of today’s AI and machine learning models, particularly those rooted in deep learning, produce data in vector form. A database that can natively search through vector data becomes indispensable for advanced applications like facial recognition or voice recognition.

​The global economic landscape is complex and competitive — and data remains at its core. In the past, many have called data the “new oil.” In the generative AI era, vector embeddings are the oil and vector databases have emerged as sophisticated refineries, adept at processing high-dimensional data and executing similarity searches.

​For the C-suite, generative AI isn’t just a buzzword; it’s a strategy. For developers, the primary allure of vector databases is efficiency. Traditional databases might require complex query structures to fetch relevant data, especially when dealing with vast datasets. Vector databases simplify this, allowing developers to retrieve data based on similarity, reducing both the complexity of the code and the time taken for data retrieval.

​A sampling of vector database use cases

 Visual content dominates our visual culture, and vector databases shine brightly in it. They are adept at sifting through vast repositories of images and videos to pinpoint those that bear a striking resemblance to a given input. This isn't just about matching pixel-by-pixel; it's about understanding the underlying patterns and features. Such capabilities are crucial for applications like facial recognition, object detection, and even copyright infringement detection in media platforms.

Natural language processing and text search:

 Synonyms, paraphrasing, and context can make exact text matching a daunting task. However, vector databases can discern the semantic essence of phrases or sentences, enabling them to identify matches that might not be identical in terms of wording but are contextually similar. This prowess is a game-changer for chatbots, ensuring they respond aptly to user queries. Similarly, search engines can deliver more relevant results, enhancing user experience.

 Vector databases play a pivotal role in personalization. By understanding user preferences and analyzing patterns, these databases can suggest songs that resonate with a listener's taste or products that align with a shopper's preferences. It's all about gauging similarity and delivering content or products that strike a chord with the user.

The horizon of vector databases is ever-expanding. In healthcare, they're aiding drug discovery by analyzing molecular structures for potential therapeutic properties. In the financial sector, vector databases are assisting in anomaly detection, spotting unusual patterns that might indicate fraudulent activities.

With the ascent of generative AI, vector databases emerge as vital enablers, assisting developers in transforming intricate AI blueprints into practical, value-driven tools.

MongoDB Atlas Vector Search: A game-changer

 is the latest addition to the MongoDB developer data platform. It enables customers to build intelligent applications powered by semantic search and generative AI over any type of data. Visit the Atlas Vector Search 

Historically, development teams seeking a vector database for tasks like image or efficient similarity search faced a dilemma: Opt for a bolt-on vector database, adding another tool to the tech stack, or juggle a mix of search tools and open-source solutions. Using a full-text search for semantic capabilities often meant developers were bogged down with extensive synonym mapping. The limitations were clear: If users weren't precise in their queries, the results were far from relevant.

Atlas Vector Search simplifies designing applications enriched by semantic search and generative AI, capable of processing a range of data types, from videos to social media content. Harnessing the robustness of MongoDB Atlas, Vector Search allows developers to craft cutting-edge, relevance-based search tools on a trusted platform with a unified query interface.

Vector Search provides MongoDB Atlas with the knowledge required to understand a query without the need to define synonyms. Even when users don’t know what they’re looking for, Vector Search is able to return relevant results based on the meaning of the query. For example, a search for “ice cream” would return “sundae,” even if the user didn't know sundaes existed.

When you use Vector Search, you’ll store vector embeddings alongside the original data and metadata in Atlas. This ensures any updates or additions to your vector data are instantly synchronized, streamlining the architecture and offering a unified developer experience.

With Vector Search, you’ll index and query data using one of the most powerful vector search algorithms: approximate k-nearest neighbors (or “k-NN,” which uses hierarchical navigable small world, or HNSW, graphs to find vector similarity).

You can create vastly improved search experiences that address use cases that traditional search tools can’t, including:

Atlas Vector Search is compatible with popular application frameworks like LlamaIndex and LangChain. It also integrates seamlessly with ecosystem partners such as Google Vertex AI, AWS, Azure, and Databricks, ensuring proprietary business data enhances the performance and accuracy of AI-powered applications.

Atlas Vector Search: For intelligent applications powered by semantic search

Vector databases, with their unique approach to data storage and retrieval, are changing the way we think about databases. Their ability to perform rapid similarity searches makes them indispensable in today's data-driven world. And when combined with the power and flexibility of MongoDB Atlas, they offer a solution that's hard to beat.

Atlas Vector Search powers advanced use cases — like semantic search, image search, and similarity search — that can’t be addressed by traditional full-text search. Developers can store their vector embeddings in MongoDB, supplement their existing search functionality with machine learning models, and query them to get relevant, contextual results. Engineering leaders benefit from the peace of mind that comes with running Atlas: a fully managed, battle-tested, multi-cloud developer data platform.

Whether you're building a recommendation system, a search engine, or any other application that requires fast and accurate data matching, consider leveraging the combined power of vector databases and MongoDB. The future is vectorized, and MongoDB is here to help you navigate it.

Rich text2 vector-databases

FAQs

FAQs section overview

MongoDB Atlas Vector Search is the latest addition to the MongoDB developer data platform. It enables customers to build intelligent applications powered by semantic search and generative AI over any type of data.

Approximate nearest neighbor search is when an algorithm is allowed to return points whose distance from the query is at most c times the distance from the query to its nearest points. In other words, the returned points can be up to c times further away from the query point than its nearest neighbor. This is useful because it allows the algorithm to find good enough neighbors quickly.

A vector index is a data structure designed to store and manage vector embeddings from a large dataset of data objects.

none

Accordions /resources/basics/databases/vector-databases

Get started with MongoDB Atlas

Section Overview /resources/basics/databases/vector-databases

Join us at AWS re:Invent 2024! Learn how to use MongoDB for AI use cases. Learn more >>

You will need access to a MongoDB Atlas database for this tutorial. If you don’t have an account, you can sign up for free to follow along.

What are Vector Databases?

FAQs

What is MongoDB Atlas Vector Search?

What is the approximate nearest neighbor search?

What is a vector index?

Get started with MongoDB Atlas