Developing a Facebook Chatbot with AWS Lambda and MongoDB Atlas
This post is part of our
Road to re:Invent series
. In the weeks leading up to AWS re:Invent in Las Vegas this November, we'll be posting about a number of topics related to running MongoDB in the public cloud.
Introduction
While microservices have been the hot trend over the past couple of years, serverless architectures have been gaining momentum by providing a new way to build scalable, responsive and cost effective applications. Serverless computing frees developers from the traditional cost and effort of building applications by automatically provisioning servers and storage, maintaining infrastructure, upgrading software, and only charging for consumed resources. More insight into
serverless computing can be found in this whitepaper
.
Amazon’s serverless computing platform, AWS Lambda, lets you run code without provisioning and running servers.
MongoDB Atlas
is Hosted MongoDB as a Service.
MongoDB Atlas
provides all the features of the database without the heavy operational lifting. Developers no longer need to worry about operational tasks such as provisioning, configuration, patching, upgrades, backups, and failure recovery. In addition,
MongoDB Atlas
offers elastic scalability, either by scaling up on a range of instance sizes or scaling out with automatic sharding, all with no application downtime. Together, AWS Lambda and
MongoDB Atlas
allow developers to spend more time developing code and less time managing the infrastructure.
Learn how to easily integrate an AWS Lambda Node.js function with a MongoDB database in this tutorial.
To demonstrate the power of serverless computing and managed database as a service, I’ll use this blog post to show you how to develop a Facebook chatbot that responds to weather requests and stores the message information in
MongoDB Atlas
.
Setting Up MongoDB Atlas
MongoDB Atlas provides multiple size options for instances. Within an instance class, there is also the ability to customize storage capacity and storage speed, as well as to use encrypted storage volumes. The number of virtual CPUs (vCPUs) – where a vCPU is a shared physical core or one hyperthread – increases as the instance class grows larger.
The M10, M20, and M30 instances are excellent for development and testing purposes, but for production it is recommended to use instances higher than M30. The
base options
for instances are:
M0 - Variable RAM, 512 MB Storage
M10 – 2 GB RAM, 10 GB Storage, 1 vCPU
M20 – 4 GB RAM, 20 GB Storage, 2 vCPUs
M30 – 8 GB RAM, 40 GB Storage, 2 vCPUs
M40 – 16 GB RAM, 80 GB Storage, 4 vCPUs
M50 – 32 GB RAM, 160 GB Storage, 8 vCPUs
M60 – 64 GB RAM, 320 GB Storage, 16 vCPUs
M100 – 160 GB RAM, 1000 GB Storage, 40 vCPUs
Register with
MongoDB Atlas
and use the intuitive user interface to select the instance size, region, and features you need.
Connecting MongoDB Atlas to AWS Lambda
Important note
: VPC Peering is not available with
MongoDB Atlas
free tier (M0). If you use an M0 cluster, allow any IP to connect to your M0 cluster and switch directly to the
Set up AWS Lambda
section.
MongoDB Atlas
enables
VPC
(Virtual Private Cloud) peering, which allows you to easily create a private networking connection between your application servers and backend database. Traffic is routed between the VPCs using private IP addresses. Instances in either VPC can communicate with each other as if they are within the same network. Note,
VPC peering
requires that both VPCs be in the same region. Below is an architecture diagram of how to connect
MongoDB Atlas
to AWS Lambda and route traffic to the Internet.
Figure 1: AWS Peering Architecture Architecture
For our example, a Network Address Translation (NAT) Gateway and Internet Gateway (IGW) is needed as the Lambda function will require internet access to query data from the Yahoo weather API. The Yahoo weather API will be used to query real-time weather data from the chatbot. The Lambda function we will create resides in the private subnet of our VPC. Because the subnet is private, the IP addresses assigned to the Lambda function cannot be used in public. To solve this issue, a
NAT Gateway
can be used to translate private IP addresses to public, and vice versa. An IGW is also needed to provide access to the internet.
The first step is to set up an Elastic IP address, which will be the static IP address of your Lambda functions to the outside world. Go to
Services->VPC->Elastic IPs
, and allocate a new Elastic IP address.
Next we will create a new VPC, which you will attach to your Lambda function.
Go to
Services->VPC->Start VPC Wizard
.
After clicking VPC wizard, select
VPC with Public and Private Subnets
.
Let’s configure our VPC. Give the VPC a name (e.g., “Chatbot App VPC”), select an IP CIDR block, choose an Availability Zone, and select the Elastic IP you created in the previous step. Note, the IP CIDR that you select for your VPC, must not overlap with the Atlas IP CIDR. Click
Create VPC
to set up your VPC. The AWS VPC wizard will automatically set up the NAT and IGW.
You should see the VPC you created in the VPC dashboard.
Go to the
Subnets
tab to see if your private and public subnets have been set up correctly.
Click on the Private Subnet and go to the
Route Table
tab in the lower window. You should see the NAT gateway set to 0.0.0.0/0, which means that messages sent to IPs outside of the private subnet will be routed to the NAT gateway.
Next, let's check the public subnet to see if it’s configured correctly. Select
Public subnet
and the
Route Table
tab in the lower window. You should see 0.0.0.0/0 connected to your IGW. The IGW will enable outside internet traffic to be routed to your Lambda functions.
Now, the final step is initiating a VPC peering connection between MongoDB Atlas and your Lambda VPC. Log in to MongoDB Atlas, and go to
Clusters->Security->Peering->New Peering Connection
.
After successfully initiating the peering connection, you will see the
Status
of the peering connection as
Waiting for Approval
.
Go back to AWS and select
Services->VPC->Peering Connections
. Select the VPC peering connection. You should see the connection request pending. Go to
Actions
and select
Accept Request
.
Once the request is accepted, you should see the connection status as
active
.
We will now verify that the routing is set up correctly. Go to the Route Table of the Private Subnet in the VPC you just set up. In this example, it is
rtb-58911e3e
. You will need to modify the
Main Route Table
(see Figure 1) to add the VPC Peering connection. This will allow traffic to be routed to
MongoDB Atlas
.
Go to the
Routes
tab and select
Edit->Add another route
. In the
Destination
field, add your Atlas CIDR block, which you can find in the
Clusters->Security
tab of the MongoDB Atlas web console:
Click in the
Target
field. A dropdown list will appear, where you should see the peering connection you just created. Select it and click
Save
.
Now that the VPC peering connection is established between the
MongoDB Atlas
and AWS Lambda VPCs, let’s set up our AWS Lambda function.
Set Up AWS Lambda
Now that our
MongoDB Atlas
cluster is connected to AWS Lambda, let’s develop our Lambda function. Go to
Services->Lambda->Create Lambda Function
. Select your runtime environment (here it’s Node.js 4.3), and select the hello-world starter function.
Select
API Gateway
in the box next to the Lambda symbol and click
Next
.
Create your API name, select
dev
as the deployment stage, and
Open
as the security. Then click
Next
.
In the next step, make these changes to the following fields:
Name
: Provide a name for your function – for example,
lambda-messenger-chatbot
Handler
: Leave as is (index.handler)
Role
: Create a basic execution role and use it (or use an existing role that has permissions to execute Lambda functions)
Timeout
: Change to 10 seconds. This is not necessary but will give the Lambda function more time to spin up its container on initialization (if needed)
VPC
: Select the VPC you created in the previous step
Subnet
: Select the private subnet for the VPC (don’t worry about adding other subnets for now)
Security Groups
: the default security group is fine for now
Press
Next
, review and create your new Lambda function.
In the code editor of your Lambda function, paste the following code snippet and press the
Save
button:
'use strict';
var VERIFY_TOKEN = "mongodb_atlas_token";
exports.handler = (event, context, callback) => {
var method = event.context["http-method"];
// process GET request
if(method === "GET"){
var queryParams = event.params.querystring;
var rVerifyToken = queryParams['hub.verify_token']
if (rVerifyToken === VERIFY_TOKEN) {
var challenge = queryParams['hub.challenge']
callback(null, parseInt(challenge))
}else{
callback(null, 'Error, wrong validation token');
}
}
};
This is the piece of code we'll need later on to set up the Facebook webhook to our Lambda function.
Set Up AWS API Gateway
Next, we will need to set up the API gateway for our Lambda function. The API gateway will let you create, manage, and host a RESTful API to expose your Lambda functions to Facebook messenger. The API gateway acts as an abstraction layer to map application requests to the format your integration endpoint is expecting to receive. For our example, the endpoint will be our Lambda function.
Go to
Services->API Gateway->[your Lambda function]->Resources->ANY
.
Click on
Integration Request
. This will configure the
API Gateway
to properly integrate Facebook with your backend application (AWS Lambda). We will set the integration endpoint to
lambda-messenger-bot
, which is the name I chose for our Lambda function.
Uncheck
Use Lambda Proxy Integration
and navigate to the
Body Mapping Templates
section.
Select
When there are no templates defined
as the
Request body passthrough
option and add a new template called
application/json
. Don't select any value in the
Generate template
section, add the code below and press
Save
:
## See http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
## This template will pass through all parameters including path, querystring, header, stage variables, and context through to the integration endpoint via the body/payload
#set($allParams = $input.params())
{
"body-json" : $input.json('$'),
"params" : {
#foreach($type in $allParams.keySet())
#set($params = $allParams.get($type))
"$type" : {
#foreach($paramName in $params.keySet())
"$paramName" : "$util.escapeJavaScript($params.get($paramName))"
#if($foreach.hasNext),#end
#end
}
#if($foreach.hasNext),#end
#end
},
"stage-variables" : {
#foreach($key in $stageVariables.keySet())
"$key" : "$util.escapeJavaScript($stageVariables.get($key))"
#if($foreach.hasNext),#end
#end
},
"context" : {
"account-id" : "$context.identity.accountId",
"api-id" : "$context.apiId",
"api-key" : "$context.identity.apiKey",
"authorizer-principal-id" : "$context.authorizer.principalId",
"caller" : "$context.identity.caller",
"cognito-authentication-provider" : "$context.identity.cognitoAuthenticationProvider",
"cognito-authentication-type" : "$context.identity.cognitoAuthenticationType",
"cognito-identity-id" : "$context.identity.cognitoIdentityId",
"cognito-identity-pool-id" : "$context.identity.cognitoIdentityPoolId",
"http-method" : "$context.httpMethod",
"stage" : "$context.stage",
"source-ip" : "$context.identity.sourceIp",
"user" : "$context.identity.user",
"user-agent" : "$context.identity.userAgent",
"user-arn" : "$context.identity.userArn",
"request-id" : "$context.requestId",
"resource-id" : "$context.resourceId",
"resource-path" : "$context.resourcePath"
}
}
The mapping template will structure the Facebook response in the desired format specified by the
application/json
template. The Lambda function will then extract information from the response and return the required output to the chatbot user. For more information on AWS mapping templates, see the
AWS documentation
.
Go back to
Services->API Gateway->[your Lambda function]->Resources->ANY
and select
Method Request
. In the Settings section, make sure
NONE
is selected in the
Authorization
dropdown list. If not, change it
NONE
and press the small Update button.
Go back to the
Actions
button for your API gateway and select
Deploy API
to make your API gateway accessible by the internet. Your API gateway is ready to go.
Set Up Facebook Messenger
Facebook makes it possible to use Facebook Messenger as the user interface for your chatbot. For our chatbot example, we will use Messenger as the UI. To create a Facebook page and Facebook app, go to the
Facebook App Getting Started Guide
to set up your Facebook components.
To connect your Facebook App to AWS Lambda you will need to go back to your API gateway. Go to your Lambda function and find the API endpoint URL (obscured in the picture below).
Go back to your Facebook App page and in the
Add Product
page, click on the
Get Started
button next to the
Messenger
section. Scroll down and in the
Webhooks
section, press the
Setup webhooks
button. A
New Page Subscription
page window should pop up. Enter your API endpoint URL in the
Callback URL
text box and in the
Verify Token
text box, enter a token name that you will use in your Lambda verification code (e.g.
mongodb_atlas_token
). As the Facebook docs explain, your code should look for the Verify Token and respond with the challenge sent in the verification request. Last, select the
messages
and
messaging_postbacks
subscription fields.
Press the
Verify and Save
button to start the validation process. If everything went well, the
Webhooks
section should show up again and you should see a
Complete
confirmation in green:
In the
Webhooks
section, click on
Select a Page
to select a page you already created. If you don't have any page on Facebook yet, you will first need to
create a Facebook page
. Once you have selected an existing page and press the
Subscribe
button.
Scroll up and in the
Token Generation
section, select the same page you selected above to generate a page token.
The first time you want to complete that action, Facebook might pop up a consent page to request your approval to grant your Facebook application some necessary page-related permissions. Press the
Continue as [your name]
button and the
OK
button to approve these permissions. Facebook generates a page token which you should copy and paste into a separate document. We will need it when we complete the configuration of our Lambda function.
Connect Facebook Messenger UI to AWS Lambda Function
We will now connect the Facebook Messenger UI to AWS Lambda and begin sending weather queries through the chatbot. Below is the
index.js
code for our Lambda function. The
index.js
file will be packaged into a compressed archive file later on and loaded to our AWS Lambda function.
"use strict";
var assert = require("assert");
var https = require("https");
var request = require("request");
var MongoClient = require("mongodb").MongoClient;
var facebookPageToken = process.env["PAGE_TOKEN"];
var VERIFY_TOKEN = "mongodb_atlas_token";
var mongoDbUri = process.env["MONGODB_ATLAS_CLUSTER_URI"];
let cachedDb = null;
exports.handler = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
var httpMethod;
if (event.context != undefined) {
httpMethod = event.context["http-method"];
} else {
//used to test with lambda-local
httpMethod = "PUT";
}
// process GET request (for Facebook validation)
if (httpMethod === "GET") {
console.log("In Get if loop");
var queryParams = event.params.querystring;
var rVerifyToken = queryParams["hub.verify_token"];
if (rVerifyToken === VERIFY_TOKEN) {
var challenge = queryParams["hub.challenge"];
callback(null, parseInt(challenge));
} else {
callback(null, "Error, wrong validation token");
}
} else {
// process POST request (Facebook chat messages)
var messageEntries = event["body-json"].entry;
console.log("message entries are " + JSON.stringify(messageEntries));
for (var entryIndex in messageEntries) {
var messageEntry = messageEntries[entryIndex].messaging;
for (var messageIndex in messageEntry) {
var messageEnvelope = messageEntry[messageIndex];
var sender = messageEnvelope.sender.id;
if (messageEnvelope.message && messageEnvelope.message.text) {
var onlyStoreinAtlas = false;
if (
messageEnvelope.message.is_echo &&
messageEnvelope.message.is_echo == true
) {
console.log("only store in Atlas");
onlyStoreinAtlas = true;
}
if (!onlyStoreinAtlas) {
var location = messageEnvelope.message.text;
var weatherEndpoint =
"https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22" +
location +
"%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";
request(
{
url: weatherEndpoint,
json: true
},
function(error, response, body) {
try {
var condition = body.query.results.channel.item.condition;
var response =
"Today's temperature in " +
location +
" is " +
condition.temp +
". The weather is " +
condition.text +
".";
console.log(
"The response to send to Facebook is: " + response
);
sendTextMessage(sender, response);
storeInMongoDB(messageEnvelope, callback);
} catch (err) {
console.error(
"error while sending a text message or storing in MongoDB: ",
err
);
sendTextMessage(sender, "There was an error.");
}
}
);
} else {
storeInMongoDB(messageEnvelope, callback);
}
} else {
process.exit();
}
}
}
}
};
function sendTextMessage(senderFbId, text) {
var json = {
recipient: { id: senderFbId },
message: { text: text }
};
var body = JSON.stringify(json);
var path = "/v2.6/me/messages?access_token=" + facebookPageToken;
var options = {
host: "graph.facebook.com",
path: path,
method: "POST",
headers: { "Content-Type": "application/json" }
};
var callback = function(response) {
var str = "";
response.on("data", function(chunk) {
str += chunk;
});
response.on("end", function() {});
};
var req = https.request(options, callback);
req.on("error", function(e) {
console.log("problem with request: " + e);
});
req.write(body);
req.end();
}
function storeInMongoDB(messageEnvelope, callback) {
if (cachedDb && cachedDb.serverConfig.isConnected()) {
sendToAtlas(cachedDb, messageEnvelope, callback);
} else {
console.log(
=> connecting to database ${mongoDbUri}
);
MongoClient.connect(mongoDbUri, function(err, db) {
assert.equal(null, err);
cachedDb = db;
sendToAtlas(db, messageEnvelope, callback);
});
}
}
function sendToAtlas(db, message, callback) {
db.collection("records").insertOne({
facebook: {
messageEnvelope: message
}
}, function(err, result) {
if (err != null) {
console.error("an error occurred in sendToAtlas", err);
callback(null, JSON.stringify(err));
} else {
var message =
Inserted a message into Atlas with id: ${result.insertedId}
;
console.log(message);
callback(null, message);
}
});
}
We are passing the
MongoDB Atlas
connection string (or URI) and Facebook page token as environment variables so we'll configure them in our Lambda function later on.
For now, clone
this GitHub repository
and open the
README file
to find the instructions to deploy and complete the configuration of your Lambda function.
Save your Lambda function and navigate to your Facebook Page chat window to verify that your function works as expected. Bring up the Messenger window and enter the name of a city of your choice (such as
New York
,
Paris
or
Mumbai
).
Store Message History in MongoDB Atlas
AWS Lambda functions are stateless; thus, if you require data persistence with your application you will need to store that data in a database. For our chatbot, we will save message information (text, senderID, recipientID) to
MongoDB Atlas
(if you look at the code carefully, you will notice that the response with the weather information comes back to the Lambda function and is also stored in MongoDB Atlas).
Before writing data to the database, we will first need to connect to
MongoDB Atlas
. Note that this code is already included in the
index.js
file.
function storeInMongoDB(messageEnvelope, callback) {
if (cachedDb && cachedDb.serverConfig.isConnected()) {
sendToAtlas(cachedDb, messageEnvelope, callback);
} else {
console.log(`=> connecting to database ${mongoDbUri}`);
MongoClient.connect(mongoDbUri, function(err, db) {
assert.equal(null, err);
cachedDb = db;
sendToAtlas(db, messageEnvelope, callback);
});
}
}
sendToAtlas
will write chatbot message information to your
MongoDB Atlas
cluster.
function sendToAtlas(db, message, callback) {
db.collection("records").insertOne({
facebook: {
messageEnvelope: message
}
}, function(err, result) {
if (err != null) {
console.error("an error occurred in sendToAtlas", err);
callback(null, JSON.stringify(err));
} else {
var message = `Inserted a message into Atlas with id: ${result.insertedId}`;
console.log(message);
callback(null, message);
}
});
}
Note that the
storeInMongoDB
and
sendToAtlas
methods implement MongoDB's recommended
performance optimizations for AWS Lambda and MongoDB Atlas
, including not closing the database connection so that it can be reused in subsequent calls to the Lambda function.
The Lambda input contains the message text, timestamp, senderID and recipientID, all of which will be written to your
MongoDB Atlas
cluster. Here is a sample document as stored in MongoDB:
{
"_id": ObjectId("58124a83c976d50001f5faaa"),
"facebook": {
"message": {
"sender": {
"id": "1158763944211613"
},
"recipient": {
"id": "129293977535005"
},
"timestamp": 1477593723519,
"message": {
"mid": "mid.1477593723519:81a0d4ea34",
"seq": 420,
"text": "San Francisco"
}
}
}
}
If you'd like to see the documents as they are stored in your MongoDB Atlas database, download
MongoDB Compass
,
connect to your Atlas cluster
and visualize the documents in your
fbchats
collection:
Note that we're storing both the message as typed by the user, as well as the response sent back by our Lambda function (which comes back to the Lambda function as noted above).
Using MongoDB Atlas with other AWS Services
In this blog, we demonstrated how to build a Facebook chatbot, using
MongoDB Atlas
and AWS Lambda.
MongoDB Atlas
can also be used as the persistent data store with many other AWS services, such as Elastic Beanstalk and Kinesis. To learn more about developing an application with AWS Elastic Beanstalk and MongoDB Atlas, read
Develop & Deploy a Node.js App to AWS Elastic Beanstalk & MongoDB Atlas
.
To learn how to orchestrate Lambda functions and build serverless workflows, read
Integrating MongoDB Atlas, Twilio, and AWS Simple Email Service with AWS Step Functions
.
For information on developing an application with AWS Kinesis and
MongoDB Atlas
, read
Processing Data Streams with Amazon Kinesis and MongoDB Atlas
.
To learn how to use your favorite language or framework with MongoDB Atlas, read
Using MongoDB Atlas From Your Favorite Language or Framework
.
About the Author - Raphael Londner
Raphael Londner is a Principal Developer Advocate at MongoDB, focused on cloud technologies such as Amazon Web Services, Microsoft Azure and Google Cloud Engine. Previously he was a developer advocate at Okta as well as a startup entrepreneur in the identity management space. You can follow him on Twitter at
@rlondner
.
November 18, 2016