Search Index is automatically deleted once pod got deleted

Rahul_Upraity · 2024-10-04T10:18:14.315Z

Hi Team,
I am creating search index using reactive mongo template but I am facing strange issue. If I deleted the pod searchIndex also got deleted.
Is search index is tied to the lifecycle of microservice pod as other normal indexes are not deleted ?

In pod logs I can see one log regarding dropping of searchindex .

image1243×26 2.57 KB

Thanks

Benjamin_Flast · 2024-10-04T19:25:48.203Z

Hey @Rahul_Upraity , I’m not sure what’s going on here based on error message. Can you provide more info, or we could jump on a call if it’s easier?

Am I right in understanding this is a Kubernetes pod? What specifically is invoking the command?

Feel free to put some time on my calendar to troubleshoot: Clockwise

iamabs2001 · 2024-10-05T20:31:48.511Z

Hey Rahul can you describe more about tech & infra where is your mongodb service hosted is it inside pod or on atlas ?

Rahul_Upraity · 2024-10-07T08:44:04.594Z

Hi @iamabs2001 ,
Mongodb service hosted on Atlas.

cc: @Benjamin_Flast
Thanks

Rahul_Upraity · 2024-10-08T02:03:49.623Z

Hi @Benjamin_Flast ,
Sorry, I forgot to join the meeting as I forgot to put it in my calender.
I am putting one more if it is ok for you?

thanks

Mayank_Anand2 · 2024-10-08T16:06:19.388Z

Hi ,
The issue you’re facing where the search index gets deleted when the pod is deleted is unusual and does not align with typical MongoDB behavior. In MongoDB, indexes (including search indexes) are usually persisted at the database level and are not tied to the lifecycle of any specific microservice or pod. Here are a few things that might be causing this behavior and steps to troubleshoot:

Potential Causes:

Ephemeral Data Stores or Volumes:

If your MongoDB instance is running in an environment with ephemeral storage (e.g., a container with no persistent volumes), any time the pod or container is recreated, the data—including the indexes—might be lost.
Solution: Ensure that your MongoDB data is stored on a persistent volume (in Kubernetes, this would be a PersistentVolume and PersistentVolumeClaim). Verify that the data directory used by MongoDB is properly backed by persistent storage.

Search Index vs. Normal Index:

Search indexes in MongoDB (especially for features like Atlas Search or text-based search indexes) could be treated differently than normal B-tree indexes. In some cloud-hosted environments or specific configurations, these search indexes might be handled separately.
Check if there is a difference between how search indexes and normal indexes are being stored or replicated.

Pod-Scoped Processes:

If the pod or microservice you are using to interact with MongoDB is running some kind of cleanup or re-initialization logic that deletes and recreates the search index, this could explain why the search index disappears.
Solution: Check if there’s any custom code or automation in your pod setup that might be handling index creation and deletion. Look into the microservice lifecycle hooks, like startup or shutdown scripts.

MongoDB Replica Set Settings:

If your MongoDB is running as a replica set, it’s possible that the search index is being dropped due to a reconfiguration of the nodes or some synchronization issue. When a node (or the entire replica set) is shut down or restarted, it might drop non-replicated data like search indexes.
Solution: Investigate the behavior of your MongoDB replica set during pod restarts. Check the MongoDB logs to see if the search index is being explicitly dropped or if it’s getting dropped as a side effect of other actions.

Logs Related to Search Index:

Since you mentioned that you see logs about the search index being dropped, examine these logs closely. They might provide clues about why the index is being deleted. Look for keywords such as “index drop,” “cleanup,” or “rebuilding indexes.”

For Debugging:

Persistent Volume Check:

Ensure that MongoDB’s data directory is mapped to a persistent volume. In Kubernetes, this would involve checking the YAML definition of the pod and ensuring that PersistentVolumeClaims are properly configured.

Pod Lifecycle Hooks:

Check if any custom scripts are tied to the pod lifecycle, especially during shutdown or startup. Look for any scripts that might be affecting indexes.

MongoDB Logs:

Inspect the MongoDB logs for more details about why the search index is being dropped. Look for any errors or explicit DROP INDEX commands in the logs.

MongoDB Search Index Documentation:

Review the specific documentation for MongoDB’s search indexes. If you’re using MongoDB Atlas Search or a similar feature, ensure that you’re following best practices for index persistence.

Index Creation Automation:

If the issue persists, consider automating the re-creation of the search index at startup if it’s acceptable to your application. However, this is more of a workaround than a true solution.

By following these steps, you should be able to determine why the search index is being tied to the pod lifecycle and find a way to ensure that it’s properly persisted.

Thanks

steevej · 2024-10-09T12:32:02.169Z

@Mayank_Anand2, could you please update your post with the mention that you use some kind of Generative AI to generate your response? Just like you did in Performance Impact of 955 indexes in one database - #2 by Mayank_Anand2.

I think it is important for people to know. To me it is obvious it comes from AI since your 4 replies appears at the same time and uses the same nice formatting.

Mayank_Anand2 · 2024-10-24T07:35:39.698Z

Hi @steevej ,
I also faced this kind and during that some GenAI , Documentation helps a alot and even I have some friend worldwide who use to work on Mongo and some are mongo developer.
and yes i used AI to response , its because there are different type of people across and GenAI generated my response in good format along with some more troubleshoot which help community to solve their bug in faster way and also its more understandable to everyone.
Use AI in positive way , ultimately we developers have to fix the issue, irrespective of from where you take guidence
I fell happy if i can make some person for fixing their issues.
Will continue the same.

Thanks,
Mayank Anand

steevej · 2024-10-24T13:14:28.890Z

I do not condemn the use of GenAI.

I just want you to mention that you did use GenAI.

Transparency helps everyone even GenAI to prevent degeneration:

Vue.ai Blog – 2 Aug 23

The Degeneration Of Generative AI | Vue.ai

Degeneration of Generative AI, aided by model collapse is here. Any mishandling could kill the tech’s potential; expertise is absolutely essential.

Est. reading time: 5 minutes

casmi.northwestern.edu

Degenerative AI: The Risks of Training Systems on their own Data

Notes on the News: CASMI Director Kristian Hammond explain the risks of training systems on their own data.

Mayank_Anand2 · 2024-10-25T05:24:30.774Z

True.
It will helps a lot if you know the right way to use it
Thanks brother