Vector Quantization

Install the required libraries.

Run the following command to install the MongoDB Node.js Driver. This operation might take a few minutes to complete.

npm install mongodb

You must install Node.js v6.11 or later driver. If necessary, you can also install libraries from your embedding model provider. For example, to generate float32, int8, and int1 embeddings by using Cohere as demonstrated in this page, install Cohere:

npm install cohere-ai dotenv
npm show cohere-ai version

Set the environment variables in your terminal.

1. To access the embedding model provider for generating and converting embeddings, set the environment variable for the embedding model provider's API key, if necessary.

   For using embeddings from Cohere, set up the COHERE_API_KEY environment variable.

   export COHERE_API_KEY="<COHERE-API-KEY>"

   If you don't set the environment variable, replace the <COHERE-API-KEY> in the sample code with the API key before running the code.

2. To access Atlas cluster, set the MONGODB_URI environment variable.

   export MONGODB_URI="<CONNECTION-STRING>"

   Your connection string should be in the following format:

   mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

   If you don't set the environment variable, replace the <CONNECTION-STRING> in the sample code with your connection string before running the code.

Generate the vector embeddings for your data.

1. Create a file named get-embeddings.js to generate float32, int8, and int1 vector embeddings by using Cohere's embed API.

   touch get-embeddings.js

2. Copy and paste the following code in the get-embeddings.js file.

   This code does the following:

   • Generates float32, int8, and int1 embeddings for the given data by using Cohere's embed-english-v3.0 embedding model.

   • Stores the float, int8, and int1 embeddings in fields named float, int8, and ubinary respectively.

   • Creates a file named embeddings.json and saves the embeddings in the file.

   get-embeddings.js

   1
   // Use 'require' for modules in a Node.js environment
   2
   const { CohereClient } = require('cohere-ai');
   3
   const { writeFile } = require('fs/promises');
   4
   dd:queueMicrotask
   5
   // Retrieve API key from environment variables or default placeholder
   6
   const apiKey = process.env.COHERE_API_KEY || '<COHERE-API-KEY>';
   7
   8
   if (!apiKey) {
   9
     throw new Error('API key not found. Please set COHERE_API_KEY in your environment.');
   10
   }
   11
   12
   // Instantiate the CohereClient with the API key
   13
   const cohere = new CohereClient({ token: apiKey });
   14
   15
   async function main() {
   16
     try {
   17
       // Data to embed
   18
       const data = [
   19
         "The Great Wall of China is visible from space.",
   20
         "The Eiffel Tower was completed in Paris in 1889.",
   21
         "Mount Everest is the highest peak on Earth at 8,848m.",
   22
         "Shakespeare wrote 37 plays and 154 sonnets during his lifetime.",
   23
         "The Mona Lisa was painted by Leonardo da Vinci.",
   24
       ];
   25
   26
       // Fetch embeddings for the data using the cohere API
   27
       const response = await cohere.v2.embed({
   28
         model: 'embed-english-v3.0',
   29
         inputType: 'search_document', 
   30
         texts: data,
   31
         embeddingTypes: ['float', 'int8', 'ubinary'], 
   32
       });
   33
   34
       // Extract embeddings from the API response
   35
       const { float, int8, ubinary } = response.embeddings;
   36
   37
       // Map the embeddings to the text data
   38
       const embeddingsData = data.map((text, index) => ({
   39
         text,
   40
         embeddings: {
   41
           float: float[index],
   42
           int8: int8[index],
   43
           ubinary: ubinary[index],
   44
         },
   45
       }));
   46
   47
       // Write the embeddings data to a JSON file
   48
       await writeFile('embeddings.json', JSON.stringify(embeddingsData, null, 2));
   49
       console.log('Embeddings saved to embeddings.json');
   50
     } catch (error) {
   51
       console.error('Error fetching embeddings:', error);
   52
     }
   53
   }
   54
   55
   // Execute the main function
   56
   main();

3. Replace the <COHERE_API_KEY> placeholder if you didn't set your API Key for Cohere as an environment variable and then save the file.

4. Run the code to generate embeddings.

   node get-embeddings.js

   HIDE OUTPUT

    Embeddings saved to embeddings.json

5. Verify the generated embeddings in the generated embeddings.json file.

Convert the vector embeddings to binData vectors.

1. Create a file named convert-embeddings.js to convert the float32, int8, and int1 vector embeddings from Cohere to BSON binData vectors by using the MongoDB Node.js driver.

   touch convert-embeddings.js

2. Copy and paste the following code in the convert-embeddings.js file.

   This code does the following:

   • Generates BSON binData vectors for the float32, int8, and int1 embeddings.

   • Appends the float32, int8, and ubinary BSON binData vectors to the embeddings.json file.

   convert-embeddings.js

   1
   const fs = require('fs/promises');
   2
   const { BSON } = require('mongodb');
   3
   const { Binary } = BSON;
   4
   5
   async function main() {
   6
     try {
   7
       // Read and parse the contents of 'embeddings.json' file
   8
       const fileContent = await fs.readFile('embeddings.json', 'utf8');
   9
       const embeddingsData = JSON.parse(fileContent);
   10
   11
       // Map the embeddings data to add BSON binary representations with subtype 9
   12
       const convertEmbeddingsData = embeddingsData.map(({ text, embeddings }) => {
   13
         // Create Binary for Float32Array with manual subtype 9
   14
         const bsonFloat32 = Binary.fromFloat32Array(new Float32Array(embeddings.float));
   15
   16
         // Create Binary for Int8Array with subtype 9
   17
         const bsonInt8 = Binary.fromInt8Array(new Int8Array(embeddings.int8));
   18
   19
         // Create Binary for PackedBits (Uint8Array) with subtype 9
   20
         const bsonPackedBits = Binary.fromPackedBits(new Uint8Array(embeddings.ubinary));
   21
   22
         return {
   23
           text,
   24
           embeddings: {
   25
             float: embeddings.float, // Original float data
   26
             int8: embeddings.int8, // Original int8 data
   27
             ubinary: embeddings.ubinary, // Original packed bits data
   28
           },
   29
           bsonEmbeddings: {
   30
             float32: bsonFloat32,
   31
             int8: bsonInt8,
   32
             packedBits: bsonPackedBits,
   33
           },
   34
         };
   35
       });
   36
   37
       // Serialize the updated data to EJSON for BSON compatibility
   38
       const ejsonSerializedData = BSON.EJSON.stringify(convertEmbeddingsData, null, null, { relaxed: false });
   39
   40
       // Write the serialized data to 'embeddings.json'
   41
       await fs.writeFile('embeddings.json', ejsonSerializedData);
   42
       console.log('Embeddings with BSON vectors have been saved to embeddings.json');
   43
     } catch (error) {
   44
       console.error('Error processing embeddings:', error);
   45
     }
   46
   }
   47
   48
   main();

3. Run the program to generate the BSON binData vectors.

   node convert-embeddings.js

   HIDE OUTPUT

   Embeddings with BSON vectors have been saved to embeddings.json

4. Verify the generated BSON embeddings in the embeddings.json file.

Connect to the Atlas cluster and upload the data to a collection.

1. Create a file named upload-data.js to connect to the Atlas cluster and create a collection in a database for the data in the embeddings.json file.

   touch upload-data.js

2. Copy and paste the following code in the upload-data.js file.

   This code does the following:

   • Connects to your Atlas cluster and creates a namespace with the database and collection name that you specify.

   • Uploads the data including the embeddings in the embeddings.json file to the specified namespace.

   upload-data.js

   1
   const fs = require('fs/promises'); // Use fs/promises for asynchronous operations
   2
   const { MongoClient, BSON } = require('mongodb'); // Import from the 'mongodb' package
   3
   4
   const { Binary } = BSON; // Ensure the Binary class is imported correctly
   5
   6
   async function main() {
   7
       const MONGODB_URI = process.env.MONGODB_URI || "<CONNECTION-STRING>";
   8
       const DB_NAME = "<DB-NAME>";
   9
       const COLLECTION_NAME = "<COLLECTION-NAME>";
   10
   11
       let client;
   12
       try {
   13
           client = new MongoClient(MONGODB_URI);
   14
           await client.connect();
   15
           console.log("Connected to MongoDB");
   16
   17
           const db = client.db(DB_NAME);
   18
           const collection = db.collection(COLLECTION_NAME);
   19
   20
           // Read and parse the contents of 'embeddings.json' file using EJSON
   21
           const fileContent = await fs.readFile('embeddings.json', 'utf8');
   22
           const embeddingsData = BSON.EJSON.parse(fileContent);
   23
   24
           // Map embeddings data to recreate BSON binary representations with the correct subtype
   25
           const documents = embeddingsData.map(({ text, bsonEmbeddings }) => {
   26
               return {
   27
                   text,
   28
                   bsonEmbeddings: {
   29
                       float32: bsonEmbeddings.float32,
   30
                       int8: bsonEmbeddings.int8,
   31
                       int1: bsonEmbeddings.packedBits
   32
                   }
   33
               };
   34
           });
   35
   36
           const result = await collection.insertMany(documents);
   37
           console.log(`Inserted ${result.insertedCount} documents into MongoDB`);
   38
   39
       } catch (error) {
   40
           console.error('Error storing embeddings in MongoDB:', error);
   41
       } finally {
   42
           if (client) {
   43
               await client.close();
   44
           }
   45
       }
   46
   }
   47
   48
   // Run the store function
   49
   main();

3. Replace the following settings and save the file.

   <CONNECTION-STRING>	Connection string to connect to the Atlas cluster where you want to create the database and collection. Replace this value only if you didn't set the MONGODB_URI environment variable.
   <DB-NAME>	Name of the database where you want to create the collection.
   <COLLECTION-NAME>	Name of the collection where you want to store the generated embeddings.

4. Run the following command to upload the data.

   node upload-data.js

5. Verify that the documents exist in the collection on your Atlas cluster.

Create the Atlas Vector Search index on the collection.

1. Create a file named create-index.js to define an Atlas Vector Search index on the collection.

   touch create-index.js

2. Copy and paste the following code to create the index in the create-index.js file.

   The code does the following:

   • Connects to the Atlas cluster and creates an index with the specified name for the specified namespace.

   • Indexes the bsonEmbeddings.float32 and bsonEmbeddings.int8 fields as vector type that uses the dotProduct similarity function, and the bsonEmbeddings.int1 field also as vector type that uses the euclidean function.

   create-index.js

   1
   const { MongoClient } = require("mongodb");
   2
   const { setTimeout } = require("timers/promises"); // Import from timers/promises
   3
   4
   // Connect to your Atlas deployment
   5
   const uri = process.env.MONGODB_URI || "<CONNECTION-STRING>";
   6
   7
   const client = new MongoClient(uri);
   8
   9
   async function main() {
   10
     try {
   11
       const database = client.db("<DB-NAME>");
   12
       const collection = database.collection("<COLLECTION-NAME>");
   13
   14
       // Define your Atlas Vector Search index
   15
       const index = {
   16
         name: "<INDEX-NAME>",
   17
         type: "vectorSearch",
   18
         definition: {
   19
           fields: [
   20
             {
   21
               type: "vector",
   22
               numDimensions: 1024,
   23
               path: "bsonEmbeddings.float32",
   24
               similarity: "dotProduct",
   25
             },
   26
             {
   27
               type: "vector",
   28
               numDimensions: 1024,
   29
               path: "bsonEmbeddings.int8",
   30
               similarity: "dotProduct",
   31
             },
   32
             {
   33
               type: "vector",
   34
               numDimensions: 1024,
   35
               path: "bsonEmbeddings.int1",
   36
               similarity: "euclidean",
   37
             },
   38
           ],
   39
         },
   40
       };
   41
   42
       // Run the helper method
   43
       const result = await collection.createSearchIndex(index);
   44
       console.log(`New search index named ${result} is building.`);
   45
   46
       // Wait for the index to be ready to query
   47
       console.log("Polling to check if the index is ready. This may take up to a minute.");
   48
       let isQueryable = false;
   49
   50
       // Use filtered search for index readiness
   51
       while (!isQueryable) {
   52
         const [indexData] = await collection.listSearchIndexes(index.name).toArray();
   53
   54
         if (indexData) {
   55
           isQueryable = indexData.queryable;
   56
           if (!isQueryable) {
   57
             await setTimeout(5000); // Wait for 5 seconds before checking again
   58
           }
   59
         } else {
   60
           // Handle the case where the index might not be found
   61
           console.log(`Index ${index.name} not found.`);
   62
           await setTimeout(5000); // Wait for 5 seconds before checking again
   63
         }
   64
       }
   65
   66
       console.log(`${result} is ready for querying.`);
   67
     } catch (error) {
   68
       console.error("Error:", error);
   69
     } finally {
   70
       await client.close();
   71
     }
   72
   }
   73
   74
   main().catch((err) => {
   75
     console.error("Unhandled error:", err);
   76
   });

3. Replace the following settings and save the file.

   <CONNECTION-STRING>	Connection string to connect to the Atlas cluster where you want to create the index. Replace this value only if you didn't set the MONGODB_URI environment variable.
   <DB-NAME>	Name of the database where you want to create the collection.
   <COLLECTION-NAME>	Name of the collection where you want to store the generated embeddings.
   <INDEX-NAME>	Name of the index for the collection.

4. Run the following command to create the index.

   node create-index.js

Generate the embeddings for the query text.

1. Create a file named get-query-embedding.js.

   touch get-query-embeddings.js

2. Copy and paste the code in the get-query-embedding.js file.

   The sample code does the following:

   • Generates float32, int8, and int1 embeddings for the query text by using Cohere.

   • Converts the generated embeddings to BSON binData vectors by using PyMongo.

   • Saves the generated embeddings to a file named query-embeddings.json.

   get-query-embedding.js

   1
   const { CohereClient } = require('cohere-ai');
   2
   const { BSON } = require('mongodb');
   3
   const { writeFile } = require('fs/promises');
   4
   const dotenv = require('dotenv');
   5
   const process = require('process');
   6
   7
   // Load environment variables
   8
   dotenv.config();
   9
   10
   const { Binary } = BSON;
   11
   12
   // Get the API key from environment variables or set the key here
   13
   const apiKey = process.env.COHERE_API_KEY || '<COHERE-API-KEY>';
   14
   15
   if (!apiKey) {
   16
     throw new Error('API key not found. Provide the COHERE_API_KEY.');
   17
   }
   18
   19
   // Initialize CohereClient
   20
   const cohere = new CohereClient({ token: apiKey });
   21
   22
   async function main(queryText) {
   23
     try {
   24
       if (typeof queryText !== 'string' || queryText.trim() === '') {
   25
         throw new Error('Invalid query text. It must be a non-empty string.');
   26
       }
   27
   28
       const data = [queryText];
   29
   30
       // Request embeddings from the Cohere API
   31
       const response = await cohere.v2.embed({
   32
         model: 'embed-english-v3.0',
   33
         inputType: 'search_query',
   34
         texts: data,
   35
         embeddingTypes: ['float', 'int8', 'ubinary'], // Request all required embedding types
   36
       });
   37
   38
       if (!response.embeddings) {
   39
         throw new Error('Embeddings not found in the API response.');
   40
       }
   41
   42
       const { float, int8, ubinary } = response.embeddings;
   43
   44
       const updatedEmbeddingsData = data.map((text, index) => {
   45
         // Create the BSON Binary objects using VECTOR_TYPE for all embedding types
   46
         const float32Binary = Binary.fromFloat32Array(new Float32Array(float[index])); // VECTOR_TYPE.FLOAT32
   47
         const int8Binary = Binary.fromInt8Array(new Int8Array(int8[index])); // VECTOR_TYPE.INT8
   48
         const packedBitsBinary = Binary.fromPackedBits(new Uint8Array(ubinary[index])); // VECTOR_TYPE.PACKED_BIT
   49
   50
         return {
   51
           text,
   52
           embeddings: {
   53
             float: float[index],
   54
             int8: int8[index],
   55
             ubinary: ubinary[index],
   56
           },
   57
           bsonEmbeddings: {
   58
             float32: float32Binary,
   59
             int8: int8Binary,
   60
             int1: packedBitsBinary,
   61
           },
   62
         };
   63
       });
   64
   65
       // Serialize the embeddings using BSON EJSON for BSON compatibility
   66
       const outputFileName = 'query-embeddings.json';
   67
       const ejsonSerializedData = BSON.EJSON.stringify(updatedEmbeddingsData, null, null, { relaxed: false });
   68
       await writeFile(outputFileName, ejsonSerializedData);
   69
       console.log(`Embeddings with BSON data have been saved to ${outputFileName}`);
   70
     } catch (error) {
   71
       console.error('Error processing query text:', error);
   72
     }
   73
   }
   74
   75
   // Main function that takes a query string
   76
   (async () => {
   77
     const queryText = "<QUERY-TEXT>"; // Replace with your actual query text
   78
     await main(queryText);
   79
   })();

3. Replace the following settings and save the file.

   <COHERE-API-KEY>	Your API Key for Cohere. Only replace this value if you didn't set the environment variable.
   <QUERY-TEXT>	Your query text. For this tutorial, use science fact.

4. Run the code to generate the embeddings for the query text.

   node get-query-embeddings.js

   HIDE OUTPUT

   Embeddings with BSON vectors have been saved to query-embeddings.json

Run an Atlas Vector Search query.

1. Create a file named run-query.js.

   touch run-query.js

2. Copy and paste the following sample $vectorSearch query in the run-query.js file.

   The sample query does the following:

   • Connects to your Atlas cluster and runs the $vectorSearch query against the bsonEmbeddings.float32, bsonEmbeddings.int8, and bsonEmbeddings.int1 fields in the specified collection by using the embeddings in the query-embeddings.json file.

   • Prints the results from Float32, Int8, and Packed Binary (Int1) embeddings to the console.

   run-query.js

   1
   const { MongoClient } = require('mongodb');
   2
   const fs = require('fs/promises');
   3
   const { BSON } = require('bson'); // Use BSON's functionality for EJSON parsing
   4
   const dotenv = require('dotenv');
   5
   6
   dotenv.config();
   7
   8
   // MongoDB connection details
   9
   const mongoUri = process.env.MONGODB_URI || '<CONNECTION-STRING>';
   10
   const dbName = '<DB-NAME>'; // Update with your actual database name
   11
   const collectionName = '<COLLECTION-NAME>'; // Update with your actual collection name
   12
   13
   // Indices and paths should match your MongoDB vector search configuration
   14
   const VECTOR_INDEX_NAME = '<INDEX-NAME>'; // Replace with your actual index name
   15
   const NUM_CANDIDATES = 5; // Number of candidate documents for the search
   16
   const LIMIT = 2; // Limit for the number of documents to return
   17
   18
   // Fields in the collection that contain the BSON query vectors
   19
   const FIELDS = [
   20
     { path: 'float32', subtype: 9 }, // Ensure that the path and custom subtype match
   21
     { path: 'int8', subtype: 9 },    // Use the custom subtype if needed
   22
     { path: 'int1', subtype: 9 } // Use the same custom subtype
   23
   ];
   24
   25
   26
   // Function to read BSON vectors from JSON and run vector search
   27
   async function main() {
   28
     // Initialize MongoDB client
   29
     const client = new MongoClient(mongoUri);
   30
   31
     try {
   32
       await client.connect();
   33
       console.log("Connected to MongoDB");
   34
   35
       const db = client.db(dbName);
   36
       const collection = db.collection(collectionName);
   37
   38
       // Load query embeddings from JSON file using EJSON parsing
   39
       const fileContent = await fs.readFile('query-embeddings.json', 'utf8');
   40
       const embeddingsData = BSON.EJSON.parse(fileContent);
   41
   42
       // Define and run the query for each embedding type
   43
       const results = {};
   44
   45
       for (const fieldInfo of FIELDS) {
   46
         const { path, subtype } = fieldInfo;
   47
         const bsonBinary = embeddingsData[0]?.bsonEmbeddings?.[path];
   48
         
   49
         if (!bsonBinary) {
   50
           console.warn(`BSON embedding for ${path} not found in the JSON.`);
   51
           continue;
   52
         }
   53
   54
         const bsonQueryVector = bsonBinary; // Directly use BSON Binary object
   55
   56
         const pipeline = [
   57
           {
   58
             $vectorSearch: {
   59
               index: VECTOR_INDEX_NAME,
   60
               path: `bsonEmbeddings.${path}`,
   61
               queryVector: bsonQueryVector,
   62
               numCandidates: NUM_CANDIDATES,
   63
               limit: LIMIT,
   64
             }
   65
           },
   66
           {
   67
             $project: {
   68
               _id: 0,
   69
               text: 1, // Adjust projection fields as necessary to match your document structure
   70
               score: { $meta: 'vectorSearchScore' }
   71
             }
   72
           }
   73
         ];
   74
   75
         results[path] = await collection.aggregate(pipeline).toArray();
   76
       }
   77
   78
       return results;
   79
     } catch (error) {
   80
       console.error('Error during vector search:', error);
   81
     } finally {
   82
       await client.close();
   83
     }
   84
   }
   85
   86
   // Main execution block
   87
   (async () => {
   88
     try {
   89
       const results = await main();
   90
   91
       if (results) {
   92
         console.log("Results from Float32 embeddings:");
   93
         console.table(results.float32 || []);
   94
         console.log("--------------------------------------------------------------------------");
   95
   96
         console.log("Results from Int8 embeddings:");
   97
         console.table(results.int8 || []);
   98
         console.log("--------------------------------------------------------------------------");
   99
   100
         console.log("Results from Packed Binary (PackedBits) embeddings:");
   101
         console.table(results.int1 || []);
   102
       }
   103
     } catch (error) {
   104
       console.error('Error executing main function:', error);
   105
     }
   106
   })();

3. Replace the following settings and save the run-query.js file.

   <CONNECTION-STRING>	Connection string to connect to the Atlas cluster where you want to run the query. Replace this value only if you didn't set the MONGODB_URI environment variable.
   <DB-NAME>	Name of the database which contains the collection.
   <COLLECTION-NAME>	Name of the collection that you want to query.
   <INDEX-NAME>	Name of the index for the collection.

4. Run the following command to execute the query.

   node run-query.js

   HIDE OUTPUT

   Connected to MongoDB
   Results from Float32 embeddings:
   ┌─────────┬─────────────────────────────────────────────────────────┬────────────────────┐
   │ (index) │                          text                           │       score        │
   ├─────────┼─────────────────────────────────────────────────────────┼────────────────────┤
   │    0    │ 'Mount Everest is the highest peak on Earth at 8,848m.' │ 0.6583383083343506 │
   │    1    │    'The Great Wall of China is visible from space.'     │ 0.6536108255386353 │
   └─────────┴─────────────────────────────────────────────────────────┴────────────────────┘
   --------------------------------------------------------------------------
   Results from Int8 embeddings:
   ┌─────────┬─────────────────────────────────────────────────────────┬────────────────────┐
   │ (index) │                          text                           │       score        │
   ├─────────┼─────────────────────────────────────────────────────────┼────────────────────┤
   │    0    │ 'Mount Everest is the highest peak on Earth at 8,848m.' │ 0.5149773359298706 │
   │    1    │    'The Great Wall of China is visible from space.'     │ 0.5146723985671997 │
   └─────────┴─────────────────────────────────────────────────────────┴────────────────────┘
   --------------------------------------------------------------------------
   Results from Packed Binary (PackedBits) embeddings:
   ┌─────────┬─────────────────────────────────────────────────────────┬─────────────┐
   │ (index) │                          text                           │    score    │
   ├─────────┼─────────────────────────────────────────────────────────┼─────────────┤
   │    0    │ 'Mount Everest is the highest peak on Earth at 8,848m.' │ 0.642578125 │
   │    1    │    'The Great Wall of China is visible from space.'     │ 0.61328125  │
   └─────────┴─────────────────────────────────────────────────────────┴─────────────┘

Install the required libraries.

Run the following command to install the MongoDB Node.js Driver. This operation might take a few minutes to complete.

npm install mongodb

You must install Node.js v6.11 or later driver. If necessary, you can also install libraries from your embedding model provider. For example, to generate float32, int8, and int1 embeddings by using Cohere as demonstrated in this page, install Cohere:

npm install cohere-ai dotenv
npm show cohere-ai version

Set the environment variables in your terminal.

1. To access the embedding model provider for generating and converting embeddings, set the environment variable for the embedding model provider's API key, if necessary.

   For using embeddings from Cohere, set up the COHERE_API_KEY environment variable.

   export COHERE_API_KEY="<COHERE-API-KEY>"

   If you don't set the environment variable, replace the <COHERE-API-KEY> in the sample code with the API key before running the code.

2. To access Atlas cluster, set the MONGODB_URI environment variable.

   export MONGODB_URI="<CONNECTION-STRING>"

   Your connection string should be in the following format:

   mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

   If you don't set the environment variable, replace the <CONNECTION-STRING> in the sample code with your connection string before running the code.

Fetch the data from your Atlas cluster.

1. Create a file named get-data.js.

   touch get-data.js

2. Copy and paste the following sample code to fetch the data from the sample_airbnb.listingsAndReviews namespace in your Atlas cluster.

   The sample code does the following:

   • Connects to your Atlas cluster and finds documents with the summary field.

   • Creates a file named subset.json to which it writes the data from the collection.

   get-data.js

   1
   const { MongoClient } = require('mongodb');
   2
   const fs = require('fs'); // Import the fs module for file system operations
   3
   4
   async function main() {
   5
       // Replace with your Atlas connection string
   6
       const uri = process.env.MONGODB_URI || '<CONNECTION-STRING>';
   7
   8
       // Create a new MongoClient instance
   9
       const client = new MongoClient(uri);
   10
   11
       try {
   12
       // Connect to your Atlas cluster
   13
       await client.connect();
   14
   15
       // Specify the database and collection
   16
       const db = client.db('sample_airbnb');
   17
       const collection = db.collection('listingsAndReviews');
   18
   19
       // Filter to exclude null or empty summary fields
   20
       const filter = { summary: { $nin: [null, ''] } };
   21
   22
       // Get a subset of documents in the collection
   23
       const documentsCursor = collection.find(filter).limit(50);
   24
   25
       // Convert the cursor to an array to get the documents
   26
       const documents = await documentsCursor.toArray();
   27
   28
       // Log the documents to verify their content
   29
       console.log('Documents retrieved:', documents);
   30
   31
       // Write the documents to a local file called "subset.json"
   32
       const outputFilePath = './subset.json';
   33
       fs.writeFileSync(outputFilePath, JSON.stringify(documents, null, 2), 'utf-8');
   34
   35
       console.log(`Subset of documents written to: ${outputFilePath}`);
   36
       } catch (error) {
   37
       console.error('An error occurred:', error);
   38
       } finally {
   39
       // Ensure the client is closed when finished
   40
       await client.close();
   41
       }
   42
   }
   43
   44
   main().catch(console.error);

3. Replace the <CONNECTION-STRING> placeholder if you didn't set the environmental variable for your Atlas connection string and then save the file.

4. Run the following command to fetch the data:

   node get-data.js

   HIDE OUTPUT

   Subset of documents written to: ./subset.json

Generate the vector embeddings for your data.

If you already have float32, int8, or int1 vector embeddings in your collection, skip this step.

1. Create a file named get-embeddings.js to generate float32, int8, and int1 vector embeddings by using Cohere's embed API.

   touch get-embeddings.js

2. Copy and paste the following code in the get-embeddings.js file.

   This code does the following:

   • Generates float32, int8, and int1 embeddings for the given data by using Cohere's embed-english-v3.0 embedding model.

   • Stores the float32, int8, and int1 embeddings in fields named float, int8, and ubinary respectively.

   • Creates a file named embeddings.json and saves the embeddings in the file.

   get-embeddings.js

   1
   // Import necessary modules using the CommonJS syntax
   2
   const { CohereClient } = require('cohere-ai');
   3
   const { readFile, writeFile } = require('fs/promises');
   4
   5
   // Retrieve the API key from environment variables or provide a placeholder
   6
   const apiKey = process.env.COHERE_API_KEY || '<COHERE-API-KEY>';
   7
   8
   if (!apiKey || apiKey === '<COHERE-API-KEY>') {
   9
     throw new Error('API key not found. Please set COHERE_API_KEY in your environment.');
   10
   }
   11
   12
   // Initialize the Cohere client with the API key
   13
   const cohere = new CohereClient({ token: apiKey });
   14
   15
   async function main() {
   16
     try {
   17
       // Read and parse the contents of 'subset.json'
   18
       const subsetData = await readFile('subset.json', 'utf-8');
   19
       const documents = JSON.parse(subsetData);
   20
   21
       // Extract the 'summary' fields that are non-empty strings
   22
       const data = documents
   23
         .map(doc => doc.summary)
   24
         .filter(summary => typeof summary === 'string' && summary.length > 0);
   25
   26
       if (data.length === 0) {
   27
         throw new Error('No valid summary texts available in the data.');
   28
       }
   29
   30
       // Request embeddings from the Cohere API
   31
       const response = await cohere.v2.embed({
   32
         model: 'embed-english-v3.0',
   33
         inputType: 'search_document',
   34
         texts: data,
   35
         embeddingTypes: ['float', 'int8', 'ubinary'],
   36
       });
   37
   38
       // Extract embeddings from the API response
   39
       const { float, int8, ubinary } = response.embeddings;
   40
   41
       // Structure the embeddings data
   42
       const embeddingsData = data.map((text, index) => ({
   43
         text,
   44
         embeddings: {
   45
           float: float[index],
   46
           int8: int8[index],
   47
           ubinary: ubinary[index],
   48
         },
   49
       }));
   50
   51
       // Write the embeddings data to 'embeddings.json'
   52
       await writeFile('embeddings.json', JSON.stringify(embeddingsData, null, 2));
   53
       console.log('Embeddings saved to embeddings.json');
   54
     } catch (error) {
   55
       console.error('Error fetching embeddings:', error);
   56
     }
   57
   }
   58
   59
   // Execute the main function
   60
   main();

3. If you didn't set the environmental variable for your Cohere API Key, replace the <COHERE-API-KEY> placeholder and save the file.

4. Run the code to generate the embeddings.

   node get-embeddings.js

   HIDE OUTPUT

    Embeddings saved to embeddings.json

5. Verify the generated embeddings by opening the generated embeddings.json file.

Convert the vector embeddings to binData vectors.

1. Create a file named convert-embeddings.js to convert the float32, int8, and int1 vector embeddings from Cohere to BSON binData vectors.

   touch convert-embeddings.js

2. Copy and paste the following code in the convert-embeddings.js file.

   This code does the following:

   • Generates BSON binData vectors for the float32, int8, and int1 embeddings.

   • Appends the float32, int8, and ubinary BSON binData vectors to the embeddings.json file.

   convert-embeddings.js

   1
   const fs = require('fs/promises');
   2
   const { BSON } = require('mongodb');
   3
   const { Binary } = BSON;
   4
   5
   async function main() {
   6
     try {
   7
       // Read and parse the contents of 'embeddings.json' file
   8
       const fileContent = await fs.readFile('embeddings.json', 'utf8');
   9
       const embeddingsData = JSON.parse(fileContent);
   10
   11
       // Map the embeddings data to add BSON binary representations with subtype 9
   12
       const convertEmbeddingsData = embeddingsData.map(({ text, embeddings }) => {
   13
         // Create Binary for Float32Array with manual subtype 9
   14
         const bsonFloat32 = Binary.fromFloat32Array(new Float32Array(embeddings.float));
   15
   16
         // Create Binary for Int8Array with subtype 9
   17
         const bsonInt8 = Binary.fromInt8Array(new Int8Array(embeddings.int8));
   18
   19
         // Create Binary for PackedBits (Uint8Array) with subtype 9
   20
         const bsonPackedBits = Binary.fromPackedBits(new Uint8Array(embeddings.ubinary));
   21
   22
         return {
   23
           text,
   24
           embeddings: {
   25
             float: embeddings.float, // Original float data
   26
             int8: embeddings.int8, // Original int8 data
   27
             ubinary: embeddings.ubinary, // Original packed bits data
   28
           },
   29
           bsonEmbeddings: {
   30
             float32: bsonFloat32,
   31
             int8: bsonInt8,
   32
             packedBits: bsonPackedBits,
   33
           },
   34
         };
   35
       });
   36
   37
       // Serialize the updated data to EJSON for BSON compatibility
   38
       const ejsonSerializedData = BSON.EJSON.stringify(convertEmbeddingsData, null, null, { relaxed: false });
   39
   40
       // Write the serialized data to 'embeddings.json'
   41
       await fs.writeFile('embeddings.json', ejsonSerializedData);
   42
       console.log('Embeddings with BSON vectors have been saved to embeddings.json');
   43
     } catch (error) {
   44
       console.error('Error processing embeddings:', error);
   45
     }
   46
   }
   47
   48
   main();

3. Run the program to generate the BSON binData vectors.

   node convert-embeddings.js

   HIDE OUTPUT

   Embeddings with BSON vectors have been saved to embeddings.json

4. Verify the generated BSON embeddings in the embeddings.json file.

Connect to the Atlas cluster and upload the data to the namespace.

1. Create a file named upload-data.js to connect to the Atlas cluster and upload the data to the sample_airbnb.listingsAndReviews namespace.

   touch upload-data.js

2. Copy and paste the following code in the upload-data.js file.

   This code does the following:

   • Connects to your Atlas cluster and creates a namespace with the database and collection name that you specify.

   • Uploads the data including the embeddings into the sample_airbnb.listingsAndReviews namespace.

   upload-data.js

   1
   const fs = require('fs/promises'); // Use fs/promises for asynchronous operations
   2
   const { MongoClient, BSON } = require('mongodb'); // Import from the 'mongodb' package
   3
   const { EJSON, Binary } = require('bson'); // Import EJSON and Binary from bson
   4
   5
   async function main() {
   6
     const MONGODB_URI = process.env.MONGODB_URI || "<CONNECTION-STRING>";
   7
     const DB_NAME = "sample_airbnb";
   8
     const COLLECTION_NAME = "listingsAndReviews";
   9
   10
     let client;
   11
     try {
   12
       // Connect to MongoDB
   13
       client = new MongoClient(MONGODB_URI);
   14
       await client.connect();
   15
       console.log("Connected to MongoDB");
   16
   17
       // Access database and collection
   18
       const db = client.db(DB_NAME);
   19
       const collection = db.collection(COLLECTION_NAME);
   20
   21
       // Load embeddings from JSON using EJSON.parse
   22
       const fileContent = await fs.readFile('embeddings.json', 'utf8');
   23
       const embeddingsData = EJSON.parse(fileContent); // Use EJSON.parse
   24
   25
       // Map embeddings data to recreate BSON binary representations
   26
       const documents = embeddingsData.map(({ text, bsonEmbeddings }) => {
   27
         return {
   28
           summary: text,
   29
           bsonEmbeddings: {
   30
             float32: bsonEmbeddings.float32,
   31
             int8: bsonEmbeddings.int8,
   32
             int1: bsonEmbeddings.packedBits
   33
           }
   34
         };
   35
       });
   36
   37
       // Iterate over documents and upsert each into the MongoDB collection
   38
       for (const doc of documents) {
   39
         const filter = { summary: doc.summary };
   40
         const update = { $set: doc };
   41
   42
         // Update the document with the BSON binary data
   43
         const result = await collection.updateOne(filter, update, { upsert: true });
   44
         if (result.matchedCount > 0) {
   45
           console.log(`Updated document with summary: ${doc.summary}`);
   46
         } else {
   47
           console.log(`Inserted new document with summary: ${doc.summary}`);
   48
         }
   49
       }
   50
   51
       console.log("Embeddings stored in MongoDB successfully.");
   52
     } catch (error) {
   53
       console.error('Error storing embeddings in MongoDB:', error);
   54
     } finally {
   55
       if (client) {
   56
         await client.close();
   57
       }
   58
     }
   59
   }
   60
   61
   // Run the main function to load the data
   62
   main();

3. Replace the <CONNECTION-STRING> placeholder if you didn't set the environmental variable for your Atlas connection string and then save the file.

4. Run the following command to upload the data.

   node upload-data.js

   HIDE OUTPUT

   Connected to MongoDB
   Updated document with text: ...
   ...
   Embeddings stored in MongoDB successfully.

5. Verify by logging into your Atlas cluster and checking the namespace in the Data Explorer.

Create the Atlas Vector Search index on the collection.

1. Create a file named create-index.js.

   touch create-index.js

2. Copy and paste the following code to create the index in the create-index.js file.

   The code does the following:

   • Connects to the Atlas cluster and creates an index with the specified name for the specified namespace.

   • Indexes the bsonEmbeddings.float32 and bsonEmbeddings.int8 fields as vector type by using the dotProduct similarity function, and the bsonEmbeddings.int1 field also as vector type by using the euclidean function.

   create-index.js

   1
   const { MongoClient } = require("mongodb");
   2
   const { setTimeout } = require("timers/promises"); // Import from timers/promises
   3
   4
   // Connect to your Atlas deployment
   5
   const uri = process.env.MONGODB_URI || "<CONNECTION-STRING>";
   6
   7
   const client = new MongoClient(uri);
   8
   9
   async function main() {
   10
     try {
   11
       const database = client.db("<DB-NAME>");
   12
       const collection = database.collection("<COLLECTION-NAME>");
   13
   14
       // Define your Atlas Vector Search index
   15
       const index = {
   16
         name: "<INDEX-NAME>",
   17
         type: "vectorSearch",
   18
         definition: {
   19
           fields: [
   20
             {
   21
               type: "vector",
   22
               numDimensions: 1024,
   23
               path: "bsonEmbeddings.float32",
   24
               similarity: "dotProduct",
   25
             },
   26
             {
   27
               type: "vector",
   28
               numDimensions: 1024,
   29
               path: "bsonEmbeddings.int8",
   30
               similarity: "dotProduct",
   31
             },
   32
             {
   33
               type: "vector",
   34
               numDimensions: 1024,
   35
               path: "bsonEmbeddings.int1",
   36
               similarity: "euclidean",
   37
             },
   38
           ],
   39
         },
   40
       };
   41
   42
       // Run the helper method
   43
       const result = await collection.createSearchIndex(index);
   44
       console.log(`New search index named ${result} is building.`);
   45
   46
       // Wait for the index to be ready to query
   47
       console.log("Polling to check if the index is ready. This may take up to a minute.");
   48
       let isQueryable = false;
   49
   50
       // Use filtered search for index readiness
   51
       while (!isQueryable) {
   52
         const [indexData] = await collection.listSearchIndexes(index.name).toArray();
   53
   54
         if (indexData) {
   55
           isQueryable = indexData.queryable;
   56
           if (!isQueryable) {
   57
             await setTimeout(5000); // Wait for 5 seconds before checking again
   58
           }
   59
         } else {
   60
           // Handle the case where the index might not be found
   61
           console.log(`Index ${index.name} not found.`);
   62
           await setTimeout(5000); // Wait for 5 seconds before checking again
   63
         }
   64
       }
   65
   66
       console.log(`${result} is ready for querying.`);
   67
     } catch (error) {
   68
       console.error("Error:", error);
   69
     } finally {
   70
       await client.close();
   71
     }
   72
   }
   73
   74
   main().catch((err) => {
   75
     console.error("Unhandled error:", err);
   76
   });

3. Replace the following settings and save the file.

   <CONNECTION-STRING>	Connection string to connect to your Atlas cluster that you want to create the database and collection. Replace this value only if you didn't set the MONGODB_URI environment variable.
   <DB-NAME>	Name of the collection, which is sample_airbnb.
   <COLLECTION-NAME>	Name of the collection, which is listingsAndReviews.
   <INDEX-NAME>	Name of the index for the collection.

4. Run the following command to create the index.

   node create-index.js

   HIDE OUTPUT

   New search index named vector_index is building.
   Polling to check if the index is ready. This may take up to a minute.
   <INDEX-NAME> is ready for querying.

Generate the embeddings for the query text.

1. Create a file named get-query-embeddings.js.

   touch get-query-embeddings.js

2. Copy and paste the code in the get-query-embedding.js file.

   The sample code does the following:

   • Generates float32, int8, and int1 embeddings for the query text by using Cohere.

   • Converts the generated embeddings to BSON binData vectors by using PyMongo.

   • Saves the generated embeddings to a file named query-embeddings.json.

   get-query-embedding.js

   1
   const { CohereClient } = require('cohere-ai');
   2
   const { BSON } = require('mongodb');
   3
   const { writeFile } = require('fs/promises');
   4
   const dotenv = require('dotenv');
   5
   const process = require('process');
   6
   7
   // Load environment variables
   8
   dotenv.config();
   9
   10
   const { Binary } = BSON;
   11
   12
   // Get the API key from environment variables or set the key here
   13
   const apiKey = process.env.COHERE_API_KEY || '<COHERE-API-KEY>';
   14
   15
   if (!apiKey) {
   16
     throw new Error('API key not found. Provide the COHERE_API_KEY.');
   17
   }
   18
   19
   // Initialize CohereClient
   20
   const cohere = new CohereClient({ token: apiKey });
   21
   22
   async function main(queryText) {
   23
     try {
   24
       if (typeof queryText !== 'string' || queryText.trim() === '') {
   25
         throw new Error('Invalid query text. It must be a non-empty string.');
   26
       }
   27
   28
       const data = [queryText];
   29
   30
       // Request embeddings from the Cohere API
   31
       const response = await cohere.v2.embed({
   32
         model: 'embed-english-v3.0',
   33
         inputType: 'search_query',
   34
         texts: data,
   35
         embeddingTypes: ['float', 'int8', 'ubinary'], // Request all required embedding types
   36
       });
   37
   38
       if (!response.embeddings) {
   39
         throw new Error('Embeddings not found in the API response.');
   40
       }
   41
   42
       const { float, int8, ubinary } = response.embeddings;
   43
   44
       const updatedEmbeddingsData = data.map((text, index) => {
   45
         // Create the BSON Binary objects using VECTOR_TYPE for all embedding types
   46
         const float32Binary = Binary.fromFloat32Array(new Float32Array(float[index])); // VECTOR_TYPE.FLOAT32
   47
         const int8Binary = Binary.fromInt8Array(new Int8Array(int8[index])); // VECTOR_TYPE.INT8
   48
         const packedBitsBinary = Binary.fromPackedBits(new Uint8Array(ubinary[index])); // VECTOR_TYPE.PACKED_BIT
   49
   50
         return {
   51
           text,
   52
           embeddings: {
   53
             float: float[index],
   54
             int8: int8[index],
   55
             ubinary: ubinary[index],
   56
           },
   57
           bsonEmbeddings: {
   58
             float32: float32Binary,
   59
             int8: int8Binary,
   60
             int1: packedBitsBinary,
   61
           },
   62
         };
   63
       });
   64
   65
       // Serialize the embeddings using BSON EJSON for BSON compatibility
   66
       const outputFileName = 'query-embeddings.json';
   67
       const ejsonSerializedData = BSON.EJSON.stringify(updatedEmbeddingsData, null, null, { relaxed: false });
   68
       await writeFile(outputFileName, ejsonSerializedData);
   69
       console.log(`Embeddings with BSON data have been saved to ${outputFileName}`);
   70
     } catch (error) {
   71
       console.error('Error processing query text:', error);
   72
     }
   73
   }
   74
   75
   // Main function that takes a query string
   76
   (async () => {
   77
     const queryText = "<QUERY-TEXT>"; // Replace with your actual query text
   78
     await main(queryText);
   79
   })();

3. Replace the following settings and save the file.

   <COHERE-API-KEY>	Your API Key for Cohere. Only replace this value if you didn't set the key as an environment variable.
   <QUERY-TEXT>	Your query text. For this example, use ocean view.

4. Run the code to generate the embeddings for the query text.

   node get-query-embeddings.js

   HIDE OUTPUT

   Embeddings with BSON vectors have been saved to query-embeddings.json

Run an Atlas Vector Search query.

1. Create a file named run-query.js.

   touch run-query.js

2. Copy and paste the following sample $vectorSearch query in the run-query.js file.

   The sample query does the following:

   • Connects to your Atlas cluster and runs the $vectorSearch query against the bsonEmbeddings.float32, bsonEmbeddings.int8, and bsonEmbeddings.int1 fields in the sample_airbnb.listingsAndReviews namespace by using the embeddings in the query-embeddings.json file.

   • Prints the results from Float32, Int8, and Packed Binary (Int1) embeddings to the console.

   run-query.js

   1
   const { MongoClient } = require('mongodb');
   2
   const fs = require('fs/promises');
   3
   const { BSON } = require('bson'); // Use BSON's functionality for EJSON parsing
   4
   const dotenv = require('dotenv');
   5
   6
   dotenv.config();
   7
   8
   // MongoDB connection details
   9
   const mongoUri = process.env.MONGODB_URI || '<CONNECTION-STRING>';
   10
   const dbName = 'sample_airbnb'; // Update with your actual database name
   11
   const collectionName = 'listingsAndReviews'; // Update with your actual collection name
   12
   13
   // Indices and paths should match your MongoDB vector search configuration
   14
   const VECTOR_INDEX_NAME = '<INDEX-NAME>'; // Replace with your actual index name
   15
   const NUM_CANDIDATES = 20; // Number of candidate documents for the search
   16
   const LIMIT = 5; // Limit for the number of documents to return
   17
   18
   // Fields in the collection that contain the BSON query vectors
   19
   const FIELDS = [
   20
     { path: 'float32', subtype: 9 }, // Ensure that the path and custom subtype match
   21
     { path: 'int8', subtype: 9 },    // Use the custom subtype if needed
   22
     { path: 'int1', subtype: 9 } // Use the same custom subtype
   23
   ];
   24
   25
   26
   // Function to read BSON vectors from JSON and run vector search
   27
   async function main() {
   28
     // Initialize MongoDB client
   29
     const client = new MongoClient(mongoUri);
   30
   31
     try {
   32
       await client.connect();
   33
       console.log("Connected to MongoDB");
   34
   35
       const db = client.db(dbName);
   36
       const collection = db.collection(collectionName);
   37
   38
       // Load query embeddings from JSON file using EJSON parsing
   39
       const fileContent = await fs.readFile('query-embeddings.json', 'utf8');
   40
       const embeddingsData = BSON.EJSON.parse(fileContent);
   41
   42
       // Define and run the query for each embedding type
   43
       const results = {};
   44
   45
       for (const fieldInfo of FIELDS) {
   46
         const { path, subtype } = fieldInfo;
   47
         const bsonBinary = embeddingsData[0]?.bsonEmbeddings?.[path];
   48
         
   49
         if (!bsonBinary) {
   50
           console.warn(`BSON embedding for ${path} not found in the JSON.`);
   51
           continue;
   52
         }
   53
   54
         const bsonQueryVector = bsonBinary; // Directly use BSON Binary object
   55
   56
         const pipeline = [
   57
           {
   58
             $vectorSearch: {
   59
               index: VECTOR_INDEX_NAME,
   60
               path: `bsonEmbeddings.${path}`,
   61
               queryVector: bsonQueryVector,
   62
               numCandidates: NUM_CANDIDATES,
   63
               limit: LIMIT,
   64
             }
   65
           },
   66
           {
   67
             $project: {
   68
               _id: 0,
   69
               name: 1,
   70
               summary: 1, // Adjust projection fields as necessary to match your document structure
   71
               score: { $meta: 'vectorSearchScore' }
   72
             }
   73
           }
   74
         ];
   75
   76
         results[path] = await collection.aggregate(pipeline).toArray();
   77
       }
   78
   79
       return results;
   80
     } catch (error) {
   81
       console.error('Error during vector search:', error);
   82
     } finally {
   83
       await client.close();
   84
     }
   85
   }
   86
   87
   // Main execution block
   88
   (async () => {
   89
       try {
   90
         const results = await main();
   91
     
   92
         if (results) {
   93
           console.log("Results from Float32 embeddings:");
   94
           (results.float32 || []).forEach((result, index) => {
   95
             console.log(`Result ${index + 1}:`, result);
   96
           });
   97
     
   98
           console.log("Results from Int8 embeddings:");
   99
           (results.int8 || []).forEach((result, index) => {
   100
             console.log(`Result ${index + 1}:`, result);
   101
           });
   102
     
   103
           console.log("Results from Packed Binary (PackedBits) embeddings:");
   104
           (results.int1 || []).forEach((result, index) => {
   105
             console.log(`Result ${index + 1}:`, result);
   106
           });
   107
         }
   108
       } catch (error) {
   109
         console.error('Error executing main function:', error);
   110
       }
   111
     })();
   112
     

3. Replace the following settings and save the run-query.js file.

   <CONNECTION-STRING>	Connection string to connect to your Atlas cluster that you want to create the database and collection. Replace this value if you didn't set the MONGODB_URI environment variable.
   <INDEX-NAME>	Name of the index for the collection.

4. Run the query.

   To execute the query, run the following command:

   node run-query.js

   HIDE OUTPUT

   Connected to MongoDB
   Results from Float32 embeddings:
   Result 1: {
   name: 'Makaha Valley Paradise with OceanView',
   summary: "A beautiful and comfortable 1 Bedroom Air Conditioned Condo in Makaha Valley - stunning Ocean & Mountain views All the amenities of home, suited for longer stays. Full kitchen & large bathroom.  Several gas BBQ's for all guests to use & a large heated pool surrounded by reclining chairs to sunbathe.  The Ocean you see in the pictures is not even a mile away, known as the famous Makaha Surfing Beach. Golfing, hiking,snorkeling  paddle boarding, surfing are all just minutes from the front door.",
   score: 0.7278661131858826
   }
   Result 2: {
   name: 'Ocean View Waikiki Marina w/prkg',
   summary: "A short distance from Honolulu's billion dollar mall, and the same distance to Waikiki. Parking included. A great location that work perfectly for business, education, or simple visit. Experience Yacht Harbor views and 5 Star Hilton Hawaiian Village.",
   score: 0.688639760017395
   }
   Result 3: {
   name: 'A Casa Alegre é um apartamento T1.',
   summary: 'Para 2 pessoas. Vista de mar a 150 mts. Prédio com 2 elevadores. Tem: - quarto com roupeiro e cama de casal (colchão magnetizado); - cozinha: placa de discos, exaustor, frigorifico, micro-ondas e torradeira; casa de banho completa; - sala e varanda.',
   score: 0.6831139326095581
   }
   Result 4: {
   name: 'Your spot in Copacabana',
   summary: 'Having a large airy living room. The apartment is well divided. Fully furnished and cozy. The building has a 24h doorman and camera services in the corridors. It is very well located, close to the beach, restaurants, pubs and several shops and supermarkets. And it offers a good mobility being close to the subway.',
   score: 0.6802051663398743
   }
   Result 5: {
   name: 'LAHAINA, MAUI! RESORT/CONDO BEACHFRONT!! SLEEPS 4!',
   summary: 'THIS IS A VERY SPACIOUS 1 BEDROOM FULL CONDO (SLEEPS 4) AT THE BEAUTIFUL VALLEY ISLE RESORT ON THE BEACH IN LAHAINA, MAUI!! YOU WILL LOVE THE PERFECT LOCATION OF THIS VERY NICE HIGH RISE! ALSO THIS SPACIOUS FULL CONDO, FULL KITCHEN, BIG BALCONY!!',
   score: 0.6779564619064331
   }
   Results from Int8 embeddings:
   Result 1: {
   name: 'Makaha Valley Paradise with OceanView',
   summary: "A beautiful and comfortable 1 Bedroom Air Conditioned Condo in Makaha Valley - stunning Ocean & Mountain views All the amenities of home, suited for longer stays. Full kitchen & large bathroom.  Several gas BBQ's for all guests to use & a large heated pool surrounded by reclining chairs to sunbathe.  The Ocean you see in the pictures is not even a mile away, known as the famous Makaha Surfing Beach. Golfing, hiking,snorkeling  paddle boarding, surfing are all just minutes from the front door.",
   score: 0.5215557217597961
   }
   Result 2: {
   name: 'Ocean View Waikiki Marina w/prkg',
   summary: "A short distance from Honolulu's billion dollar mall, and the same distance to Waikiki. Parking included. A great location that work perfectly for business, education, or simple visit. Experience Yacht Harbor views and 5 Star Hilton Hawaiian Village.",
   score: 0.5179016590118408
   }
   Result 3: {
   name: 'A Casa Alegre é um apartamento T1.',
   summary: 'Para 2 pessoas. Vista de mar a 150 mts. Prédio com 2 elevadores. Tem: - quarto com roupeiro e cama de casal (colchão magnetizado); - cozinha: placa de discos, exaustor, frigorifico, micro-ondas e torradeira; casa de banho completa; - sala e varanda.',
   score: 0.5173280239105225
   }
   Result 4: {
   name: 'Your spot in Copacabana',
   summary: 'Having a large airy living room. The apartment is well divided. Fully furnished and cozy. The building has a 24h doorman and camera services in the corridors. It is very well located, close to the beach, restaurants, pubs and several shops and supermarkets. And it offers a good mobility being close to the subway.',
   score: 0.5170232057571411
   }
   Result 5: {
   name: 'LAHAINA, MAUI! RESORT/CONDO BEACHFRONT!! SLEEPS 4!',
   summary: 'THIS IS A VERY SPACIOUS 1 BEDROOM FULL CONDO (SLEEPS 4) AT THE BEAUTIFUL VALLEY ISLE RESORT ON THE BEACH IN LAHAINA, MAUI!! YOU WILL LOVE THE PERFECT LOCATION OF THIS VERY NICE HIGH RISE! ALSO THIS SPACIOUS FULL CONDO, FULL KITCHEN, BIG BALCONY!!',
   score: 0.5168724060058594
   }
   Results from Packed Binary (PackedBits) embeddings:
   Result 1: {
   name: 'Makaha Valley Paradise with OceanView',
   summary: "A beautiful and comfortable 1 Bedroom Air Conditioned Condo in Makaha Valley - stunning Ocean & Mountain views All the amenities of home, suited for longer stays. Full kitchen & large bathroom.  Several gas BBQ's for all guests to use & a large heated pool surrounded by reclining chairs to sunbathe.  The Ocean you see in the pictures is not even a mile away, known as the famous Makaha Surfing Beach. Golfing, hiking,snorkeling  paddle boarding, surfing are all just minutes from the front door.",
   score: 0.6591796875
   }
   Result 2: {
   name: 'Ocean View Waikiki Marina w/prkg',
   summary: "A short distance from Honolulu's billion dollar mall, and the same distance to Waikiki. Parking included. A great location that work perfectly for business, education, or simple visit. Experience Yacht Harbor views and 5 Star Hilton Hawaiian Village.",
   score: 0.6337890625
   }
   Result 3: {
   name: 'A Casa Alegre é um apartamento T1.',
   summary: 'Para 2 pessoas. Vista de mar a 150 mts. Prédio com 2 elevadores. Tem: - quarto com roupeiro e cama de casal (colchão magnetizado); - cozinha: placa de discos, exaustor, frigorifico, micro-ondas e torradeira; casa de banho completa; - sala e varanda.',
   score: 0.62890625
   }
   Result 4: {
   name: 'LAHAINA, MAUI! RESORT/CONDO BEACHFRONT!! SLEEPS 4!',
   summary: 'THIS IS A VERY SPACIOUS 1 BEDROOM FULL CONDO (SLEEPS 4) AT THE BEAUTIFUL VALLEY ISLE RESORT ON THE BEACH IN LAHAINA, MAUI!! YOU WILL LOVE THE PERFECT LOCATION OF THIS VERY NICE HIGH RISE! ALSO THIS SPACIOUS FULL CONDO, FULL KITCHEN, BIG BALCONY!!',
   score: 0.6279296875
   }
   Result 5: {
   name: 'Be Happy in Porto',
   summary: 'Be Happy Apartment is an amazing space. Renovated and comfortable apartment, located in a building dating from the nineteenth century in one of the most emblematic streets of the Porto city "Rua do Almada".  Be Happy Apartment is located in the city center, able you to visit the historic center only by foot, being very close of majority points of interesting of the Porto City. Be Happy Apartment is located close of central Station MetroTrindade.',
   score: 0.619140625
   }

   Your results might be different because the generated embeddings can vary depending on your environment.

Install the required libraries.

Run the following command to install the PyMongo Driver. If necessary, you can also install libraries from your embedding model provider. This operation might take a few minutes to complete.

pip install pymongo

You must install PyMongo v4.10 or later driver.

pip install --quiet --upgrade pymongo cohere

Load the data for which you want to generate BSON vectors in your notebook.

data = [
   "The Great Wall of China is visible from space.",
   "The Eiffel Tower was completed in Paris in 1889.",
   "Mount Everest is the highest peak on Earth at 8,848m.",
   "Shakespeare wrote 37 plays and 154 sonnets during his lifetime.",
   "The Mona Lisa was painted by Leonardo da Vinci.",
]

(Conditional) Generate embeddings from your data.

This step is required if you haven't yet generated embeddings from your data. If you've already generated embeddings, skip this step. To learn more about generating embeddings from your data, see How to Create Vector Embeddings.

import os
import cohere
# Specify your Cohere API key
os.environ["COHERE_API_KEY"] = "<COHERE-API-KEY>"
cohere_client = cohere.Client(os.environ["COHERE_API_KEY"])
# Generate embeddings using the embed-english-v3.0 model
generated_embeddings = cohere_client.embed(
   texts=data,
   model="embed-english-v3.0",
   input_type="search_document",
   embedding_types=["float", "int8", "ubinary"]
).embeddings
float32_embeddings = generated_embeddings.float
int8_embeddings = generated_embeddings.int8
int1_embeddings = generated_embeddings.ubinary

Generate the BSON vectors from your embeddings.

You can use the PyMongo driver to convert your native vector embedding to BSON vectors.

from bson.binary import Binary, BinaryVectorDtype
def generate_bson_vector(vector, vector_dtype):
   return Binary.from_vector(vector, vector_dtype)
# For all vectors in your collection, generate BSON vectors of float32, int8, and int1 embeddings
bson_float32_embeddings = []
bson_int8_embeddings = []
bson_int1_embeddings = []
for i, (f32_emb, int8_emb, int1_emb) in enumerate(zip(float32_embeddings, int8_embeddings, int1_embeddings)):
   bson_float32_embeddings.append(generate_bson_vector(f32_emb, BinaryVectorDtype.FLOAT32))
   bson_int8_embeddings.append(generate_bson_vector(int8_emb, BinaryVectorDtype.INT8))
   bson_int1_embeddings.append(generate_bson_vector(int1_emb, BinaryVectorDtype.PACKED_BIT))

Create documents with the BSON vector embeddings.

If you already have the BSON vector embeddings inside of documents in your collection, skip this step.

# Specify the field names for the float32, int8, and int1 embeddings
float32_field = "<FIELD-NAME-FOR-FLOAT32-TYPE>"
int8_field = "<FIELD-NAME-FOR-INT8-TYPE>"
int1_field = "<FIELD-NAME-FOR-INT1-TYPE>"
# Define function to create documents with BSON vector embeddings
def create_docs_with_bson_vector_embeddings(bson_float32_embeddings, bson_int8_embeddings, bson_int1_embeddings, data):
  docs = []
  for i, (bson_f32_emb, bson_int8_emb, bson_int1_emb, text) in enumerate(zip(bson_float32_embeddings, bson_int8_embeddings, bson_int1_embeddings, data)):
     doc = {
          "_id": i,
          "data": text,
          float32_field: bson_f32_emb,
          int8_field: bson_int8_emb,
          int1_field: bson_int1_emb
     }
     docs.append(doc)
  return docs
# Create the documents
documents = create_docs_with_bson_vector_embeddings(bson_float32_embeddings, bson_int8_embeddings, bson_int1_embeddings, data)

Load your data into your Atlas cluster.

You can load your data from the Atlas UI and programmatically. To learn how to load your data from the Atlas UI, see Insert Your Data. The following steps and associated examples demonstrate how to load your data programmatically by using the PyMongo driver.

1. Connect to your Atlas cluster.

   Placeholder	Valid Value
   <ATLAS-CONNECTION-STRING>	Atlas connection string. To learn more, see Connect via Drivers.

   Example

   import pymongo
   mongo_client = pymongo.MongoClient("<ATLAS-CONNECTION-STRING>")
   if not MONGO_URI:
     print("MONGO_URI not set in environment variables")

2. Load the data into your Atlas cluster.

   Placeholder	Valid Value
   <DB-NAME>	Name of the database.
   <COLLECTION-NAME>	Name of the collection in the specified database.

   Example

   # Insert documents into a new database and collection
   db = mongo_client["<DB-NAME>"]
   collection_name = "<COLLECTION-NAME>"
   db.create_collection(collection_name)
   collection = db[collection_name]
   collection.insert_many(documents)

Create the Atlas Vector Search index on the collection.

You can create Atlas Vector Search indexes by using the Atlas UI, Atlas CLI, Atlas Administration API, and MongoDB drivers. To learn more, see How to Index Fields for Vector Search.

from pymongo.operations import SearchIndexModel
import time
# Define and create the vector search index
index_name = "<INDEX-NAME>"
search_index_model = SearchIndexModel(
  definition={
    "fields": [
      {
        "type": "vector",
        "path": float32_field,
        "similarity": "dotProduct",
        "numDimensions": 1024
      },
      {
        "type": "vector",
        "path": int8_field,
        "similarity": "dotProduct",
        "numDimensions": 1024
      },
      {
        "type": "vector",
        "path": int1_field,
        "similarity": "euclidean",
        "numDimensions": 1024
      }
    ]
  },
  name=index_name,
  type="vectorSearch"
)
result = collection.create_search_index(model=search_index_model)
print("New search index named " + result + " is building.")
# Wait for initial sync to complete
print("Polling to check if the index is ready. This may take up to a minute.")
predicate=None
if predicate is None:
  predicate = lambda index: index.get("queryable") is True
while True:
  indices = list(collection.list_search_indexes(index_name))
  if len(indices) and predicate(indices[0]):
    break
  time.sleep(5)
print(result + " is ready for querying.")

Define a function to run the Atlas Vector Search queries.

The function to run Atlas Vector Search queries must perform the following actions:

• Convert the query text to a BSON vector.

• Define the pipeline for the Atlas Vector Search query.

# Define a function to run a vector search query
def run_vector_search(query_text, collection, path):
  query_text_embeddings = cohere_client.embed(
    texts=[query_text],
    model="embed-english-v3.0",
    input_type="search_query",
    embedding_types=["float", "int8", "ubinary"]
  ).embeddings
  if path == float32_field:
    query_vector = query_text_embeddings.float[0]
    vector_dtype = BinaryVectorDtype.FLOAT32
  elif path == int8_field:
    query_vector = query_text_embeddings.int8[0]
    vector_dtype = BinaryVectorDtype.INT8
  elif path == int1_field:
    query_vector = query_text_embeddings.ubinary[0]
    vector_dtype = BinaryVectorDtype.PACKED_BIT
  bson_query_vector = generate_bson_vector(query_vector, vector_dtype)
  pipeline = [
    {
      '$vectorSearch': {
        'index': index_name,
        'path': path,
        'queryVector': bson_query_vector,
        'numCandidates': <NUMBER-OF-CANDIDATES-TO-CONSIDER>, # for example, 5
        'limit': <NUMBER-OF-DOCUMENTS-TO-RETURN> # for example, 2
       }
     },
     {
       '$project': {
         '_id': 0,
         'data': 1,
         'score': { '$meta': 'vectorSearchScore' }
        }
     }
  ]
  return collection.aggregate(pipeline)