Number of connections spiked when consistent lambda functions invokes are stopped

Frank_Ye · 2024-10-02T07:36:39.880Z

In our design, we have lambda function to access MongoDB. The connection to MongoDB is shared by different lambda function invokes (as a form of a library). The lambda function invoke rate is about 100/second. The average execution time of Lambda function is about 30~30ms.
In normal working cases, the number of connections to MongoDB stays consistently within the range of 150~200 (we have other components that contribute to the number of connections too). However, we have do Lambda Function upgrade, a spike in number of connection to MongoDB is triggered. The spike can go as high as 2000~2500. After about 5-6 minutes (300s~360s), it starts to drop to normal level. Further tests shows it is the stop of Lambda Function invoke caused the spike (we run some tests to trigger Lambda Function invoke, after it becomes stable ,we suddenly stop the tests and Lamda Function invoke suddenlly stops), on the contrary of our expectation, the number of connections to MongoDB server spiked from 200 to 2000~2500.
Anyone has similar experience or simular observation about this behavior?

Frank_Ye · 2024-10-02T19:39:06.938Z

Never mind. I found the root cause of the issue. We have 7~8 lambda functions in total. Stopping the trigger of one lambda function also triggers the system automatic generated $disconnect event for another websocket event processing lambda function. The bulk arrival of these $disconnect events in a very short time triggers the ten fold spike of connections to MongoDB (from another websocket event processing lambda function).