Index Build: draining writes received during build

Arturs_Sosins · 2021-10-12T09:20:30.226Z

Hello,
we have a docker setup for the app and on each container start we try to create all indexes again, just in case, cause we would not know if it is first started or not, etc. But it does not matter.

Point is, sometimes indexing are stuck with:
“Index Build: draining writes received during build”

And they never stop.

It seems to happen with 4.4.4 and now after upgrading to 4.4.9 it also still seems to happen.
And it seems to happen only on Replica sets.
And it happens on collections with low write rates or even empty collections that have no writes.

Could it be related to the bugs fixed in 4.4.9? And we simply need to fix/clean all data?
Or this is something else?

kevinadi · 2021-10-13T05:29:14.451Z

Hi @Arturs_Sosins ,

This was a known issue previously (see SERVER-48516), where the main issue was that using auth, the node was not able to communicate with itself. A workaround was to create a keyfile as mentioned in this comment.

Having said that, this is supposed to be fixed in 4.4.4. If you have added the keyfile and it doesn’t seem to fix the issue, could you please post some more details:

What is the MongoDB version you’re running
What’s the output of rs.status() and rs.conf() from the replica set
What do you mean exactly by “sometimes it’s stuck”? Is this happening intermittently? Are there any pattern you can discern?

Best regards
Kevin

Arturs_Sosins · 2021-10-13T09:16:06.165Z

True, I forgot to mention that it is not the case with the node not being able to communicate with itself. We specifically checked that.

This thing is more random, and it happens on collections that already have the index. Afaik it never happened on collections that do not have the index.

That’s why my next idea is that maybe that bug with index uniqueness could be causing that. We have hundreds of client servers, and it has been happening on only a couple of those servers, but if it happens for a client, then it would be happening and repeating again in the future (but not always), only collections sometimes change.

Unfortunately, we could not establish any patterns. Versions vary from 4.4.2 to now 4.4.9.

The only way we can get around that is either kill the index building process or mongoexport and mongoimport with drop.

Any specific diagnostic data we could collect when that happens next time?

Daniel_Lewis2 · 2024-11-18T18:33:30.838Z

I’m sorry for resurrecting such an old thread, but was there ever a solution to this?

I’m also getting “Index Build: draining writes received during build” in my standalone database. I’ve rebooted it, added a keyfile, but nothing seems to work. It seems I can’t create new indexes, either, because of this.

Daniel_Lewis2 · 2024-11-18T19:18:32.590Z

Well, I guess I did something wrong. Because here was a crucial missing detail: my DB was a replica set, but then I switched it back to standalone (for temporary performance reasons). Maybe my switching it back interrupted some processing, or maybe MongoDB simply doesn’t switch back well. But I re-initiated it as a single-member replica set and now that annoying message went away and my indexes create successfully again.

chris · 2024-11-18T19:47:47.353Z

Yup, that is a pretty critical detail. Dropping the local database will have set you right also.