Connecting to MongoDB Atlas from Google Cloud Functions

Atef_Sawaed · 2021-01-25T19:48:52.003Z

I am connecting to MongoDB Atlas with Google Cloud Functions (Serverless). I see that there is a high number of active connections on the server that is hosting my replica while I am the only user currently running the functions. (My app is still in development)
I found the following article that solves the issue for AWS Lambda functions (https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda) But I am not able to find the Google Cloud equivalent. does anyone knows how can I reuse my connection between Cloud Function instances?

Pavel_Duchovny · 2021-01-26T08:59:59.423Z

Hi @Atef_Sawaed,

Welcome to MongoDB community.

I have not used google functions against Atlas but I wonder if global variables could help here the same way as the concept of handlers work in AWS lambda:

Google Cloud

Tips & Tricks | Cloud Functions Documentation | ...

Try to define the MongoDB client as a global variable.

Thanks
Pavel

Atef_Sawaed · 2021-01-26T09:45:21.496Z

Hi Pavel,

Yes, global variables are supported in GC Functions and I’ve already used them. Here is my implementation:

var client;

const getClient = async () => {
	if (client && client.isConnected()) {
		console.log("MONGODB CLIENT ALREADY CONNECTED!");
	} else
		try {
			client = await MongoClient.connect(url, {
				useNewUrlParser: true,
				useUnifiedTopology: true,
			});
			console.log("MONGODB CLIENT RECONNECTED!");
		} catch (e) {
			throw e;
		}

	return client;
};

exports.getPlacesFromDB = functions.https.onCall((data, context) => {
    var limitOption = 10;
    var query = {};
    var sortOption = {};
    return getClient().then((client) => {
        const db = client.db("database");
        const collection = db.collection("collection");
        
        var _cursor = collection.find(query).skip(skips).limit(limitOption).sort(sortOption);
        return _cursor.map(({ _id, ...d }) => ({ _id: _id.toString(), ...d })).toArray().then((result) => {
            // Returning the message to the client.
            return { text: 'Retrieved data!', data: result };
          })
          .catch((error) => {
              functions.logger.log("error:", error.message);
              throw new functions.https.HttpsError('cannot get data!', error.message, error);
            });
        }).catch((error) => {
          functions.logger.log("error:", error.message);
          throw new functions.https.HttpsError('no connection', error.message, error);
        });
});

In the AWS article I shared, they are suggesting to disable callbackWaitsForEmptyEventLoop in the Context object of the Lambda function. I am trying to see how can this be done in GCP?

Thanks!

Pavel_Duchovny · 2021-01-26T10:00:19.138Z

@Atef_Sawaed,

Maybe background functions?

Google Cloud

Write event-driven functions | Cloud Functions Documentation...

Atef_Sawaed · 2021-01-26T12:17:31.451Z

@Pavel_Duchovny,

How can background functions help me here? Can you elaborate?

I Appreciate your support and quick responses!

Pavel_Duchovny · 2021-01-27T07:14:11.919Z

@Atef_Sawaed,

It was just a guess to allow usage of callback rather than freeze.

With global variable do you still see unexpected amount of connections?

Thanks
Pavel

Atef_Sawaed · 2021-01-27T18:01:11.495Z

Yes. I see ~40 active connections in my cluster while there is 1-2 daily active users.

Pavel_Duchovny · 2021-01-28T07:30:24.271Z

@Atef_Sawaed,

Usually there are different parties connecting to the Atlas nodes like the other nodes and our agents.

It is possible that those together with connection pools from the Google functions endup in 40 connections.

I will suggest you to contact our support for a more specific breakdown.

thanks,
Pavel

Benjamin_Bouvier · 2021-07-19T05:45:36.302Z

@Atef_Sawaed Any updates on this ? Could you find a proper solution in the end ?
Thanks in advance !

Axel_Vaindal · 2021-11-11T22:21:42.936Z

@Atef_Sawaed @Benjamin_Bouvier Any updates on this?
I’m having the same problem with a lot of Exit with code 16 errors in my cloud functions due to this.
Do you have a pattern to properly manage connections in Google Cloud Functions?

Thanks in advance

Pavel_Duchovny · 2021-11-12T06:09:49.361Z

Hi @Atef_Sawaed and @Axel_Vaindal

I didn’t find any official best practices.

But found some interesting articles

The Code Barbarian

Getting Started With Google Cloud Functions and MongoDB

Google Cloud

Tips & Tricks | Cloud Functions Documentation | ...

Hope you can use it

Ty

Very_Crazy_Dog · 2021-11-16T01:59:54.857Z

Not sure this is the root cause, but GCP Functions different from AWS Lambda in a way that GCP Functions allows one instance to handle multiple requests concurrently. A proper resource guard is therefore required to avoid creating multiple clients when concurrent requests arrive. @Atef_Sawaed’s example can be modified to something like below (WARNING: I haven’t tested the code):

const getClient = async () => {
	if (client && client.isConnected()) {
		console.log("MONGODB CLIENT ALREADY CONNECTED!");
	} else if (client instanceof Promise) {
		client = await client;
		console.log("MONGODB CLIENT RECONNECTED!");
	} else {
		// If client.isConnected() return false and arrived here, do we need explicit disconnect and release resource?
		client = MongoClient.connect(url, {
			useNewUrlParser: true,
			useUnifiedTopology: true,
		});
	}
	return client;
};

Stennie_X · 2022-01-24T22:48:02.531Z