Local atlas search index not working in aggregate

Aaron_Waidmann · 2024-03-25T15:10:18.978Z

I deployed atlas locally using this docker-compose:

version: "3.9"
name: test-database
services:
  database:
    image: mongodb/atlas
    privileged: true
    command: |
      /bin/bash -c "atlas deployments setup --type local --port 27017 --bindIpAll --username admin --password admin --force && tail -f /dev/null"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "27017:27017"

I then ran the following snippet to create a search index, insert some data and then query that data using a search index.

const {MongoClient, ObjectId} = require("mongodb");

async function testSearchIndexes() {
    const mongo = new MongoClient("<URL>");
    await mongo.connect();
    const db = mongo.db("test-database");

    const testColl = db.collection("test-collection");


    await testColl.createSearchIndex({
        definition: {
            mappings: {
                dynamic: true,
            },
        }
    });


    await testColl.insertMany([{
        _id: new ObjectId(),
        name: "Test value"
    }, {
        _id: new ObjectId(),
        name: "Other value"
    }, {
        _id: new ObjectId(),
        name: "Other value 2"
    }, {
        _id: new ObjectId(),
        name: "Test value 2"
    }, {
        _id: new ObjectId(),
        name: "Test value 3"
    }]);

    await new Promise((resolve) => setTimeout(resolve, 10000));
    
    const result = await testColl.aggregate([
        {
            $search: {
                text: {
                    query: "Test",
                    path: {
                        wildcard: '*',
                    },
                },
            }
        }
    ]).toArray();

    console.log(result);

    await mongo.close();
}

testSearchIndexes().catch(console.error);

However, this returns 0 documents. When I run the same aggregate on compass afterwards I get 3 documents as expected and I also get these 3 documents when rerunning the snippet. This is relevant to us, as this behaviour makes it very hard to run our E2E Tests.

Massimiliano_Marcon · 2024-03-25T16:10:23.170Z

10 seconds should be enough to build the local search index, but you could try something like this:

const { MongoClient, ObjectId } = require("mongodb");

async function testSearchIndexes() {
  const mongo = new MongoClient(
    "mongodb://admin:s3cr3t@127.0.0.1:27017?directConnection=true"
  );
  await mongo.connect();
  const db = mongo.db("test-database");

  await db.createCollection("test-collection");

  const testColl = db.collection("test-collection");

  await testColl.createSearchIndex({
    definition: {
      mappings: {
        dynamic: true,
      },
    },
  });

  await testColl.insertMany([
    {
      _id: new ObjectId(),
      name: "Test value",
    },
    {
      _id: new ObjectId(),
      name: "Other value",
    },
    {
      _id: new ObjectId(),
      name: "Other value 2",
    },
    {
      _id: new ObjectId(),
      name: "Test value 2",
    },
    {
      _id: new ObjectId(),
      name: "Test value 3",
    },
  ]);

  while ((await testColl.listSearchIndexes().next()).status !== "READY") {
    console.log("Waiting for search index to be ready...");
    await new Promise((resolve) => setTimeout(resolve, 1000));
  }

  console.log("Search index is ready!");

  const result = await testColl
    .aggregate([
      {
        $search: {
          text: {
            query: "Test",
            path: {
              wildcard: "*",
            },
          },
        },
      },
    ])
    .toArray();

  console.log(result);

  await mongo.close();
}

testSearchIndexes().catch(console.error);

Note that I added a call to await db.createCollection("test-collection"); as you need to make sure the collection exists before creating a search index.

Aaron_Waidmann · 2024-03-26T08:47:01.579Z

Thanks for the reply! Unfortunately I already was checking for the search index status before (but didn’t post it for a minimal example) so this didn’t change anything. Were you able to reproduce this behaviour or did the $search query return the documents for you?

Massimiliano_Marcon · 2024-03-26T10:06:54.781Z

The $search query returns the documents for me.

Here is exactly what I am doing: MongoDB Atlas Search Local Development · GitHub

Aaron_Waidmann · 2024-03-26T10:29:44.105Z

It’s really strange, I can’t find any changes in your setup compared to mine, but when I use yours the search index works 90% of the time. When it does work it always prints “Waiting for search index to be ready…” once and if it doesn’t work, the search index seems to be marked as READY from the getgo.

Massimiliano_Marcon · 2024-03-26T12:50:53.408Z

Ok, 90% of the time is definitely weird. Let me think a bit about what could be going wrong.