I’m building an application to process a lot of information on each file and save them in mongo.
There are around 1 million files.
And in each file there can be from 100 to 400 grouped information.
So the logic on my rust code is loop every file.
In each file loop, i then iter and collect each grouped information in a tokio::spawn (save into mongo with update_one upsert true) into a vec which then use in future::try_join_all.
The mongo collections have indexes including the parameter used for update_one - find match.
Sometimes in the middle of this process, i get:
Error { kind: ConnectionPoolCleared { message: "Connection pool for localhost:27017 cleared because another operation failed w
ith: Kind: I/O error: timed out, labels: {}" }, labels: {"RetryableWriteError"}, wire_version: None, source: None }
and because of try_join_all, the loop gets terminated - which is what i want.
but the problem now is that i cannot pass this 1 particular file because the error above keeps on showing up.
what does that error means and how do I fix it?
Note: modifying the file as a workaround is not an option.
