Issues with Implementing a MongoDB Relay Proxy for Multiple Workers

Hello MongoDB Community,

I am currently working on a challenging project where I am trying to build a MongoDB relay service. The goal is to act as an intermediary between multiple worker applications and the MongoDB server, allowing all workers to connect through a single relay without exceeding MongoDB’s maximum connection limit.

Scenario Overview

1. Project Context:

• I have multiple worker applications (using mongoose) that need to connect to a single MongoDB server.
• Each worker application is expected to perform various read and write operations on the database.
• The problem is that each worker establishes a new connection with the MongoDB server, leading to connection spikes and eventual connection exhaustion.

2. Objective:

• I want to build a relay service that the workers can connect to, using the MongoDB URI of the relay, which will then forward their requests to the actual MongoDB server.
• The goal is to avoid creating new connections for each request and reduce the total number of connections to the MongoDB server to a manageable limit.

3. **Approaches Tried:**Approach 1: Direct Mongoose Proxy

• I attempted to use a single Mongoose client to handle all worker requests, where the incoming data was parsed as a JSON object, and a shared Mongoose connection executed the commands.
• Issue Faced: While this approach worked for structured JSON queries, it failed for other MongoDB protocol commands, especially during initial handshake requests. Parsing raw binary data became a significant challenge.Approach 2: Raw net.Socket Forwarding
• I used net.Socket to handle raw protocol data and forward it directly to the MongoDB server. The relay established a new net.Socket connection between each worker and the MongoDB server.
• Issue Faced: Each worker interaction resulted in a new MongoDB connection, causing a rapid increase in connections. This led to MongoDB server hitting the connection limit, resulting in ECONNREFUSED errors.

4. Current Problem:

• Whenever a worker establishes a connection to the relay, it sends a series of raw buffer data as part of the initial handshake. The relay service has difficulty handling this buffer and translating it into MongoDB client commands.
• Due to the raw buffer being passed, the relay ends up creating multiple MongoDB connections instead of reusing a single shared connection, leading to a connection flood.
• I am unable to parse and forward this data correctly without introducing significant changes in the worker codebase, which is not a feasible option.

5. Additional Details:

• The MongoDB relay service is written using Node.js and net.Socket.
• The worker applications use Mongoose for MongoDB interactions.
• The relay service should act as a pass-through layer for the worker applications, without modifying the underlying logic of how they communicate with the MongoDB server.

Request for Assistance:

I am looking for guidance on the following points:

1. Is there a built-in MongoDB client or protocol handling library that can better manage the raw buffer data and forward it correctly to the MongoDB server?
2. Are there any recommended approaches or libraries that would allow me to act as a MongoDB protocol proxy without introducing additional connections?
3. If anyone has faced similar issues, could you share how you managed to overcome the connection spike and protocol compatibility challenges?

I would greatly appreciate any advice or suggestions on how to handle this scenario, especially without modifying the existing worker codebase or introducing new connections for every worker request.

Thank you in advance for your help!