Creating a vector index in DocumentDB with pymongo's create_index` function fails with an "Index already exists with different options" error, even when the index options remain unchanged.

Amitoj_Singh1 · 2024-08-13T10:47:57.704Z

Vector Index Issue in DocumentDB

I am trying to create a vector index for my DocumentDB. According to pymongo's documentation, you are supposed to use create_index to do so. Whenever I drop all my indexes using drop_indexes before initially creating or recreating my vector index, all works well. However, when I remove the drop_indexes and try to create an index (on the pre-existing one), I get an error.

According to the documentation, the create_index function shouldn't create an index if a pre-existing one already exists. The function will check if the index exists and if the arguments passed in are the same, if they are, the index doesn't get created and no error is thrown. This is the functionality I expect and need for my use case. However, although all settings remain the same for the vector index I'm creating, additional calls afterwards for creating the index are "going through" and actually trying to be created when nothing of the index changes.

Here is the error I receive:

Index already exists with different options, full error: {'ok': 0.0, 'operationTime': Timestamp(1723494808, 1), 'code': 85, 'errmsg': 'Index already exists with different options'}

Here is a simplified example to illustrate the issue:


# Initial index creation
col.create_index(
    [("embedding_key", "vector")],
    vectorOptions={
        "type": "hnsw",
        "similarity": "cosine",
        "dimensions": 128,
        "m": 16,
        "efConstruction": 64,
    },
    name="my_vss_index",
)
Attempting to create the same index again later (this isn’t my specific use case but this does mimic my case effectively)
col.create_index(

[(“embedding_key”, “vector”)],

vectorOptions={

“type”: “hnsw”,

“similarity”: “cosine”,

“dimensions”: 128,

“m”: 16,

“efConstruction”: 64,

},

name=“my_vss_index”,

)

On the other hand, creating an index with text search capability and the same use case does work as expected:




Initial text index creation
col.create_index([(“content”, “text”)], name=“my_memories_index”)
Attempting to create the same text index again later
col.create_index([(“content”, “text”)], name=“my_memories_index”)

I would appreciate any guidance on why the vector index behaves differently and how to ensure that create_index does not recreate an existing vector index if it already exists.

Thank you!

Mark_Smith · 2024-08-13T12:08:49.266Z

Hi @Amitoj_Singh1 - welcome to the community forums!

In MongoDB, a vector index is a type of search index. They’re created with collection.create_search_index() in PyMongo. Where did you find documentation saying that they’re created with create_index?

You’re right that create_index with the same arguments as an existing index will silently do nothing, but that’s not the case with create_search_index, unfortunately, where you’ll have to call collection.list_search_indexes() with the name of your index to check whether it already exists.

Hope this helps!

Mark

Amitoj_Singh1 · 2024-08-13T17:10:02.952Z

Thanks Mark! Excited to be here!

Amazon DocumentDB provides this as one of the code examples on how to setup DocumentDB and use it to do semantic search in Python. This is where I got the idea to use create_index and not create_search_index. Is what Amazon provides as an example a valid approach?

My next idea was to use collection.list_search_indexes() to search through indexes before creation, so I’m glad to know this is a recommended solution.

Thanks!