Intermittent MongoNetworkError: connect ETIMEDOUT 127.0.0.1:27017, connect ECONNREFUSED ::1:27017 under high load

David_Thomson · 2025-01-07T11:26:05.733Z

Hi

I’m having an issue where mongo appears to close and/or reject connections when under high CPU load. I have set socketTimeoutMS to 300000 and connectTimeoutMS to 300000 but the timeouts still occur.

I’m running mongo 6 on windows server 2022. At the time of the error there are 300 concurrent operations trying to update the database. The error occurs after about 10 seconds of high load.

The client reports an error like this:
2025-01-07 16:14:02+1300: Error in mapByChunks: MongoNetworkError: connect ETIMEDOUT 127.0.0.1:27017, connect ECONNREFUSED ::1:27017
2025-01-07 16:14:02+1300: at connectionFailureError (C:\Program Files (x86).…\node_modules\mongodb\src\cmap\connect.ts:470:14)
2025-01-07 16:14:02+1300: at Socket. (C:\Program Files (x86).…\node_modules\mongodb\src\cmap\connect.ts:372:38)
2025-01-07 16:14:02+1300: at Object.onceWrapper (node:events:634:26)
2025-01-07 16:14:02+1300: at Socket.emit (node:events:519:28)
2025-01-07 16:14:02+1300: at emitErrorNT (node:internal/streams/destroy:169:8)
2025-01-07 16:14:02+1300: at emitErrorCloseNT (node:internal/streams/destroy:128:3)
2025-01-07 16:14:02+1300: at processTicksAndRejections (node:internal/process/task_queues:82:21) {
2025-01-07 16:14:02+1300: connectionGeneration: 0,
2025-01-07 16:14:02+1300: [Symbol(errorLabels)]: Set(1) { ‘ResetPool’ },
2025-01-07 16:14:02+1300: [cause]: AggregateError:
2025-01-07 16:14:02+1300: at internalConnectMultiple (node:net:1117:18)
2025-01-07 16:14:02+1300: at afterConnectMultiple (node:net:1684:7) {
2025-01-07 16:14:02+1300: code: ‘ETIMEDOUT’,
2025-01-07 16:14:02+1300: [errors]: [ [Error], [Error] ]
2025-01-07 16:14:02+1300: }
2025-01-07 16:14:02+1300: }

Around the same time mongo reports a warning:
{“t”:{“$date”:“2025-01-07T16:14:03.489+13:00”},“s”:“W”, “c”:“NETWORK”, “id”:4615610, “ctx”:“conn147”,“msg”:“Failed to check socket connectivity”,“attr”:{“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Connection closed by peer”}}}
{“t”:{“$date”:“2025-01-07T16:14:03.489+13:00”},“s”:“I”, “c”:“-”, “id”:20883, “ctx”:“conn147”,“msg”:“Interrupted operation as its client disconnected”,“attr”:{“opId”:14311}}
{“t”:{“$date”:“2025-01-07T16:14:03.491+13:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn147”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:60456”,“uuid”:“a3391ed0-56c1-423c-b447-a9fdd3942ec9”,“connectionId”:147,“connectionCount”:203}}

The connection was created only 4 seconds before it was closed:
{“t”:{“$date”:“2025-01-07T16:10:37.119+13:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“127.0.0.1:60456”,“uuid”:“a3391ed0-56c1-423c-b447-a9fdd3942ec9”,“connectionId”:147,“connectionCount”:135}}
{“t”:{“$date”:“2025-01-07T16:10:37.120+13:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn147”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:60456”,“client”:“conn147”,“negotiatedCompressors”:,“doc”:{“application”:{“name”:“polling:9848”},“driver”:{“name”:“nodejs|Mongoose”,“version”:“6.8.0|8.6.1”},“platform”:“Node.js v20.13.0, LE”,“os”:{“name”:“win32”,“architecture”:“x64”,“version”:“10.0.20348”,“type”:“Windows_NT”}}}}

Is this expected behaviour, that connections will be closed when the CPU gets busy? Is there a way to control this behaviour?

Please let me know if you need any more information.

Thanks

David_Thomson · 2025-01-28T08:28:55.924Z

It turns out that we had inadvertently disabled IPv6 support. The “ipv6: true” line had been removed from the config. With this config removed, when under heavy load, the error messages occurred. After putting the configuration back, the error messages went away.