Mongodb-atlas-local vector search index goes missing

agibson · 2024-10-02T07:36:31.975Z

Just recently switched over to using the new mongodb-atlas-local docker container for local dev on our dev team due to some other issues with the old cli based docker image. We have a script that we run to initialize both our python dependencies (unrelated) as well as other local docker container dependencies for local development. However, when we run this script, it does bring up the mongo db with the persistent data from before. Here’s the problem: EVERYTIME we run the script on various machines, it stands the docker container up but does not keep the vector search index I setup prior. What would cause this? Is there another volume I need to add to persist the vector search index?

My script contains this:

cd .\local-dependencies
docker compose down
docker compose up -d

Here’s my docker compose:

services:
  mongo-atlas:
    image: mongodb/mongodb-atlas-local:7.0.14 #NOTE: This should ideally match the version in mongodb.com on the prod server.
    # privileged: true
    restart: always
    volumes:
      # Fixes issues with rebooting and not finding the key file.
      # https://www.mongodb.com/community/forums/t/persisting-data-with-mongodb-mongodb-atlas-local/286926/8
      - mongodb_config:/data/configdb
      - mongodb_data:/data/db
    ports:
      - 27017:27017
    environment:
      - MONGODB_INITDB_ROOT_USERNAME=root
      - MONGODB_INITDB_ROOT_PASSWORD=root

volumes:
  mongodb_config:
  mongodb_data:

To remedy (or workaround, actually), I tried to update my python code to use pymongo 4.7 and call create_search_index but that always results in a search index with a status of “DOES_NOT_EXIST”. I’d rather just set the index once, and have it persist past machine reboots and docker compose down && docker compose up -d calls.

agibson · 2024-10-02T19:39:12.233Z

In the meantime of this post being approved, I solved this issue…
If you look in that /data/mongot folder when you’re running, you can see the vector search index is set in one of those files. When you restart the container, it disappears. So the fix here is to persist that folder so that it doesn’t die between restarts.

services:
  mongo-atlas:
    image: mongodb/mongodb-atlas-local:7.0.14 #NOTE: This should ideally match the version in mongodb.com on the prod server.
    # privileged: true
    restart: always
    volumes:
      # Fixes issues with rebooting and not finding the key file.
      # https://www.mongodb.com/community/forums/t/persisting-data-with-mongodb-mongodb-atlas-local/286926/8
      - mongodb_config:/data/configdb
      - mongodb_data:/data/db
      # Fix issues with rebooting and not persisting the vector search index
      - mongodb_mongot:/data/mongot
    ports:
      - 27017:27017
    environment:
      - MONGODB_INITDB_ROOT_USERNAME=root
      - MONGODB_INITDB_ROOT_PASSWORD=root

volumes:
  mongodb_config:
  mongodb_data:
  mongodb_mongot:

Looks like the docs (https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker) need to be updated.