Semantic Search Made Easy With LangChain and MongoDB
Rate this video
00:00:00Introduction to Semantic Search
The video begins by explaining the multi-step process of enabling semantic search on user-specific data. This process includes loading, transforming, embedding, and storing data before it can be queried. The tutorial uses the AT&T Wikipedia page as a data source and demonstrates how to use libraries from Lang chain to load, transform, embed, and store vectors.00:02:36Setting Up MongoDB Atlas
The next section of the video focuses on setting up MongoDB Atlas as the Vector store. The tutorial explains how to clone a repo, add an OpenAI API key and MongoDB connection string, and create a new Python environment. It also demonstrates how to install all the necessary requirements and vectorize the data.00:05:12Querying and Retrieving Data
The final section of the video explains how to retrieve and query relevant data using Vector search. The tutorial demonstrates how to set up a vector search index on the Vector embeddings field and how to run a search using methods like similarity search or maximum marginal reference search. The video concludes by showing how to run a query and see the results.The main theme of the video is a tutorial on how to enable semantic search on user-specific data using Lang chain and MongoDB Atlas.
🔑 Key Points
- The process of enabling semantic search involves loading, transforming, embedding, and storing data.
- Lang chain provides a set of utilities to simplify this process.
- MongoDB Atlas is used as the Vector store in this tutorial.
- The AT&T Wikipedia page is used as the data source.
- The tutorial demonstrates how to use libraries from Lang chain to load, transform, embed, and store vectors.
- The video also explains how to retrieve and query relevant data using Vector search.
🔗 Related Links
Full Video Transcript
enabling semantic search on user specific data is a multi-step process that includes loading transforming embedding and storing Data before it can be queried now that graphic is from the team over at Lang chain whose goal is to provide a set of utilities to greatly simplify this process in this tutorial we're going to walk through each of these steps using mongodb Atlas as our Vector store and we're going to use the AT&T Wikipedia page as our data source we'll then use libraries from Lang chain to load transform embed and store our vectors and then we can retrieve and query relevant data to get started you'll need a free mongodb Atlas account and an open aai API key and today we're going to be using python but this can also be accomplished using JavaScript and nodejs first we're going to clone a repo that has all the code written for us to get started and link to this repo is in the video description below so I'm going to go over to code and copy the link in vs code let's clone this Repository I'll select my location and open this up the first thing we need to do is open p.p We'll add our open AI API key and our mongodb connection string in these spots and then save it now let's open up the terminal and we're going to create a new python environment and let's activate the environment and we'll install all of the requirements now we're ready to vectorize our data we'll load it trans transform it embed it and store it and again we're going to use the AT&T Wikipedia site as our data source but there's no lack of sources of data it could be slack YouTube git Excel PDFs markdown files Etc and Lang chain provides a growing list of Integrations for these sources so let's take a look at the vectorize dopy file the first thing that we're doing is loading our data and here we're using the webbase loader that we're importing from Lang chain and in this loader we are pointing to the at TNT Wikipedia site and we can also load multiple sources like we're doing here with the Bank of America Wikipedia page as well and next we'll transform the data by splitting it now it needs to be split into smaller chunks so that we can easily find relevant portions based on our search query now for this example we'll use the recommended recursive character text splitter which again we are importing up here from Lang chain make this formatting look a little bit better here we have our chunk size of 1,000 so that's 1,000 characters we have an overlap of zero and our separators this is going to attempt to split on paragraphs with the two new line characters the one new line character and then sentences and words so if a paragraph doesn't fit into 1,000 characters it's going to truncate at the next word that it can find that will fit within that chunk size of under 1,000 characters and you can tune the chunk size to your liking smaller numbers will lead to more documents and vice versa on next is the embedding embedding is the process of creating Vector representations of data using an llm and there are many options to choose from such as open Ai and hugging face and Lang chain provides a standard interface for interacting with all of these so here we're going to use the popular open aai embeddings now again be sure you have your openai API key set in the pam. py file and now that we have the embedder loaded let's store our vectors and data now you will need a vector database to store your embeddings and fortunately for you mongodb is perfect for storing vector embeddings and even better the folks at Lang chain have a mongodb atlas module that will do all of the heavy lifting for you again don't forget to also add your mongod Atlas connection string to the pam. py file so here we're going to connect our client get our collection we'll go ahead and clear out the collection if there's anything there and then this is the Mong Atlas Vector search method from Lang chain that we are importing at the top here so in in this we're going to pass our docs our embeddings our collection and our index name and we only need to run this file once per data source so let's go ahead and run it so back in the terminal let's run python vectorize all right it split it into 288 documents let's go over to Mong Tobe Atlas and check it out so in our database let's go to browse Collections and we can see that a database search DB was created a collection search collection ction and we have all of these documents each of these has the original text Chunk along with the vector embeddings for each one along with some other metadata we see these embeddings are an array of 1 1536 vectors now in order to query this data using Vector search next we need to set up a vector search index on our Vector embeddings field but while we're here in Atlas let's go up to Atlas search and let's create a search index index we're going to choose Atlas Vector search Json editor and next for the index name I'm going to name it vsearch index then be sure to select the collection and we'll set some parameters the path is going to be embedding that's the field where our embeddings are and the number of Dimensions is 1536 this is specific to the embedding model of the llm that we're using and then for similarity we're going to set that to co all right and then let's click next and create that search index and now we're ready to retrieve and query our data using Vector search so let's take a look at the query. py file so this is a CLI based query so we're going to parse our arguments first and we can ask it questions within the CLI here's some sample questions that we could ask first thing it's going to do is print your question and the question that we asked then it's going to connect to our client and get our collection and then initialize our Vector store so again we're using mongod Tob Atlas Vector search from Lang chain passing it our collection and creating a new open AI embeddings for the query we have to vectorize the query in order to compare it with the vectors that we already have in our database we'll print another new line and then we'll run the actual query here now we can run a search using methods like similarity search or maximum marginal reference search which we have here and this would return the relevant slice of data which in our case would be an entire paragraph but we could continue to harness the power of the llm to contextually compress the response so that it more directly tries to answer our question so we'll run it again through the contextual compression retriever which is coming in at the top here from Lang chain and then we'll print that response as well so let's run the query and see what we get go back in our terminal let's run our query so we'll say python query uh pi and then our question is going to be who started AT&T all right let's make this a little bit bigger and we can see here our question was who started AT&T and the semantic search Returns the most relevant chunk of text but then passing that through the AI the ai's response is only the relevant pieces of text from that chunk that most directly answer the question so I hope you see how easy it is to get semantic search working in mongodb using Lang chain now if you prefer to read a written version of this tutorial is linked in the video description below give mongodb Vector search a try in your next AI project and let us know what you think if this video was helpful give it a like And subscribe for more great mongodb content ngodb