MongoDB

Aastha Mahendru

Software Engineer

As the volumes of data managed by organizations grow exponentially, effective log management becomes critical for maintaining the performance and security of database platforms. 

Automation and configuration of log management through infrastructure as code (IaC) tools can enable users to automatically push mongod, mongos, and audit logs directly to their S3 buckets instead of manually downloading zipped log files or creating any custom solutions to continuously pull logs from Atlas.

To enhance developer agility, we’re introducing a new capability that enables developer teams to push logs to an Amazon S3 bucket, providing a continuous, scalable, cost-effective solution for log storage and analysis.

This guide will walk you through how to set up push-based logging to Amazon S3 in Atlas through our [HashiCorp Terraform MongoDB Atlas provider.](https://registry.terraform.io/providers/mongodb/mongodbatlas/latest) Let’s get started! 

## Prerequisites

- Create a [MongoDB Atlas](https://mongodb.prakticum-team.ru/cloud/atlas/register) account. 
- Create an [AWS](https://aws.amazon.com/free/) account.
- Install [HashiCorp Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) on your terminal or sign up for a free [Terraform Cloud](https://app.terraform.io/public/signup/account) account.
- Create [MongoDB Atlas programmatic API keys](https://mongodb.prakticum-team.ru/docs/atlas/configure-api-access/#grant-programmatic-access-to-service) and [associate them with Terraform. ](https://mongodb.prakticum-team.ru/developer/products/atlas/deploy-mongodb-atlas-terraform-aws/)
- [Configure your AWS credentials](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#provider-configuration) to be used with Terraform.
- Select an IDE of your choice. For this tutorial, we will be using [VS Code](https://code.visualstudio.com/).

You can enable the [push-based log export](https://mongodb.prakticum-team.ru/docs/atlas/push-logs/#std-label-mongodb-logs-push) feature on a project level in your Atlas organization after you have authorized your Atlas project to access the S3 bucket in your AWS account. This means all clusters in your Atlas project will be automatically configured to push logs to the S3 bucket.

So to start off, let’s give Atlas what it needs to get the appropriate permissions!

*Note that if you prefer to manage IAM roles and permissions via UI or other tools, you can accomplish this by following the appropriate steps outlined in **[Set Up Unified AWS Access](https://mongodb.prakticum-team.ru/docs/atlas/security/set-up-unified-aws-access/#set-up-unified-aws-access)**, skipping* **Step #2**, and configuring the appropriate roles in your Terraform configuration.* 

## Configure the Hashicorp Terraform MongoDB Atlas and AWS providers

In this tutorial, we will use Terraform to manage all operations so you don’t have to worry about jumping between your AWS Management Console, your MongoDB Atlas account, and your Terraform configuration files. This requires you to first specify the [HashiCorp Terraform MongoDB Atlas provider](https://registry.terraform.io/providers/mongodb/mongodbatlas/latest) as well as [Hashicorp Terraform AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest) versions in a **versions.tf** file as below:

```
terraform {
 required_providers {
   mongodbatlas = {
     source  = "mongodb/mongodbatlas"
     version = ">= 1.16.0"
   }
   aws = {
     source  = "hashicorp/aws"
     version = ">= 5.0"
   }
 }
 required_version = ">= 1.0"
}
```

Note: Before deploying anything, be sure to store the MongoDB Atlas programmatic API keys you created as part of the prerequisites as [environment variables](https://developer.hashicorp.com/terraform/cli/config/environment-variables), and additionally, ensure to have configured your credentials for the AWS provider. 

We will now create a **variables.tf** file for declaring Terraform variables and a **terraform.tfvars** file for defining variable values. These files are typically created within the root directory of your Terraform project.

In **variables.tf**, we will define the identifier of your Atlas organization, the name of your new Atlas project, and the name of the new S3 bucket that the logs will be pushed to:

```
# Atlas Organization ID
variable "atlas_org_id" {
 type        = string
 description = "Atlas Organization ID"
}

# Atlas Project Name
variable "atlas_project_name" {
 type        = string
 description = "Atlas Organization ID"
}

# Unique AWS S3 Bucket Name
variable "s3_bucket_name" {
 type        = string
 description = "AWS S3 Bucket Name for logs"
}
```

In the **terraform.tfvars** file, we will configure the values for the above variables:

```
atlas_org_id = "<UPDATE WITH YOUR ORG ID>"
atlas_project_name = "<UPDATE WITH A NAME FOR PROJECT>"
s3_bucket_name = "<UPDATE WITH A UNIQUE NAME FOR YOUR S3 BUCKET>"
```

We can now run `terraform init` in the terminal. This will initialize Terraform and download Terraform MongoDB Atlas and AWS providers.

![The screenshot shows that Terraform and the providers have been successfully initialized.][1]

## Set up AWS access for MongoDB Atlas

To set up unified AWS access, you must give [Organization Owner](https://mongodb.prakticum-team.ru/docs/atlas/reference/user-roles/#mongodb-authrole-Organization-Owner) or [Project Owner](https://mongodb.prakticum-team.ru/docs/atlas/reference/user-roles/#mongodb-authrole-Project-Owner) access to the Atlas project.

From here, set up cloud provider access in MongoDB Atlas. You can do this with the `mongodbatlas_cloud_provider_access_setup` resource which returns an `atlas_aws_account_arn` and `atlas_assumed_role_external_id`.

At this point, your ***.tf** configuration file should look similar to this:

```
resource "mongodbatlas_project" "exampleProject" {
 name   = var.atlas_project_name
 org_id = var.atlas_org_id
}

resource "mongodbatlas_cloud_provider_access_setup" "setupOnly" {
 project_id    = mongodbatlas_project.exampleProject.id
 provider_name = "AWS"
}
```

*If you are curious to learn more about this authorization mechanism, visit the* [*AWS docs*](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)*.*

Next, create an IAM role with the `assume_role_policy` in Atlas configured with the Atlas AWS account from the previous step as the principal.

```
resource "aws_iam_role" "atlasRole" {
 name                 = "atlas-role"
 max_session_duration = 43200

 assume_role_policy = jsonencode({
   "Version" : "2012-10-17",
   "Statement" : [
     {
       "Effect" : "Allow",
       "Principal" : {
         "AWS" : "${mongodbatlas_cloud_provider_access_setup.setupOnly.aws_config[0].atlas_aws_account_arn}"
       },
       "Action" : "sts:AssumeRole",
       "Condition" : {
         "StringEquals" : {
           "sts:ExternalId" : "${mongodbatlas_cloud_provider_access_setup.setupOnly.aws_config[0].atlas_assumed_role_external_id}"
         }
       }
     }
   ]
 })
}

resource "aws_iam_role_policy" "atlasRolePolicy" {
 name = "atlas-role-policy"
 role = aws_iam_role.atlasRole.id

 policy = jsonencode({
   "Version" : "2012-10-17",
   "Statement" : [
     {
       "Effect" : "Allow",
       "Action" : "s3:*",
       "Resource" : "*"
     }
   ]
 })
}
```

Finally, use `mongodbatlas_cloud_provider_access_authorization` resource to authorize and configure the new IAM Assumed Role ARN. 

*Tip: For now, we have only authorized this role for S3 actions but once successful, you can expand the policy and use the role_id value when configuring other Atlas services that use AWS,* *such as* [*Data Federation*](https://mongodb.prakticum-team.ru/docs/atlas/data-federation/#std-label-atlas-data-federation) *and* [*encryption at Rest*](https://mongodb.prakticum-team.ru/docs/atlas/security-aws-kms/#std-label-security-aws-kms)*.*

```
resource "mongodbatlas_cloud_provider_access_authorization" "authRole" {
 project_id = mongodbatlas_project.exampleProject.id
 role_id    = mongodbatlas_cloud_provider_access_setup.setupOnly.role_id

 aws {
   iam_assumed_role_arn = aws_iam_role.atlasRole.arn
 }
}
```

## Configure an Amazon S3 bucket 

Now that you have given Atlas the authorization to Amazon S3 service, we will create a new S3 bucket (or use an existing bucket) and configure an `aws_iam_role_policy` resource to allow the previously created `atlasRole` to perform operations required for pushing the logs to this bucket:

```
resource "aws_s3_bucket" "atlasLogBucket" {
 bucket        = var.s3_bucket_name
 force_destroy = true      // required for destroying as Atlas creates a test folder in the bucket when push-based log export is initially created
}

resource "aws_iam_role_policy" "atlasLogBucketPolicy" {
 name = "atlas-log-bucket-policy"
 role = aws_iam_role.atlasRole.id

 policy = jsonencode({
   "Version" : "2012-10-17",
   "Statement" : [
     {
       "Effect" : "Allow",
       "Action" : [
         "s3:ListBucket",
         "s3:PutObject",
         "s3:GetObject",
         "s3:GetBucketLocation"
       ],
       "Resource" : [
         "${aws_s3_bucket.atlasLogBucket.arn}",
         "${aws_s3_bucket.atlasLogBucket.arn}/*"
       ]
     }
   ]
 })
}
```

Let’s deploy everything so far by running the below commands from the terminal: 

```
terraform plan
terraform apply
```

With that, you have configured your S3 bucket!

## Enable push-based log export

Coming back to where it all started, you can now very easily enable push-based log export configuration for your Atlas project. All you need is the name of the S3 bucket and the `role_id` from the `mongodbatlas_cloud_provider_access_authorization` resource. Add the below to your ***.tf** file:

```
resource "mongodbatlas_push_based_log_export" "logExport" {
 project_id  = mongodbatlas_project.exampleProject.id
 bucket_name = aws_s3_bucket.atlasLogBucket.bucket
 iam_role_id = mongodbatlas_cloud_provider_access_authorization.authRole.role_id
 prefix_path = "push-based-log-test"
}
```

To deploy again, run from the terminal: 

```
terraform plan
terraform apply
```

If your deployment was successful, you should be greeted with “Apply complete!” 

![Terraform in terminal showcasing deployment][2]

You can verify the configuration by fetching objects in the S3 bucket. You can do this by updating your Terraform configuration to include the `aws_s3_objects` data source and specify the bucket name and the prefix path like this:

```
data "aws_s3_objects" "atlasLogObjects" {
 bucket = aws_s3_bucket.atlasLogBucket.bucket
 prefix = mongodbatlas_push_based_log_export.logExport.prefix_path
}
```

List the objects in the bucket by creating an **outputs.tf** file and referencing the `keys` parameter as below:

```
output "log_bucket_keys" {
 value = data.aws_s3_objects.atlasLogObjects.keys
}
```

Run `terraform plan` followed by `terraform apply` in the terminal. You should now see an “atlas-test” object created in your S3 bucket in the outputs:

![Terraform in terminal showing objects created in Amazon S3 bucket as outputs][3]

Atlas creates this test object to verify that the IAM role configured has write access to this S3 bucket for the purposes of push-based log exporting. It is safe to delete this file once the log export configuration is successful.

Now that you have successfully configured push-based log export to your S3 bucket, as soon as a cluster is deployed in your Atlas project, you should see the logs in your configured S3 bucket. Happy logging!

## All done

Congratulations! You now have everything that you need to start pushing your MongoDB Atlas database logs to your Amazon S3 bucket.

The HashiCorp Terraform Atlas provider is open-sourced under the [Mozilla Public License v2.0](https://github.com/mongodb/terraform-provider-mongodbatlas?tab=MPL-2.0-1-ov-file) and we welcome community contributions. To learn more, see our [contributing guidelines](https://github.com/mongodb/terraform-provider-mongodbatlas/blob/master/CONTRIBUTING.md). 

The fastest way to get started is to create a MongoDB Atlas account from the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?sr=0-2&ref_=beagle&applicationId=AWSMPContessa). To learn more about the Terraform provider, check out our [documentation](https://registry.terraform.io/providers/mongodb/mongodbatlas/latest/docs), [tutorials](https://mongodb.prakticum-team.ru/developer/search/?s=Terraform&sortMode=0), [solution brief](https://mongodb.prakticum-team.ru/collateral/build-resilient-applications-with-mongodb-atlas-and-hashicorp-terraform), or [get started](https://account.mongodb.com/account/login?signedOut=true&n=https%3A%2F%2Fcloud.mongodb.com%2Fgo%3Fl%3Dhttps%3A%2F%2Fcloud.mongodb.com%2Fv2%2F%3Cproject%3E%23%2FdataFederation%26_ga%3D2.121590028.567858162.1702935684-1592895210.1668436484%26_gac%3D1.90809704.1702935684.CjwKCAiA-P-rBhBEEiwAQEXhH1-PuNDJyW2Z4QzJDKtcrNGmvK965-IyQbQ-bitpEZxRCffLHhX-KRoC1HsQAvD_BwE) directly.

Go build with MongoDB Atlas and the HashiCorp Terraform Atlas provider today! 


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt719e8a5da077f3b7/668c54ae94a816a0f907e508/1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltfb254491bf60da59/668c54c4534bb9564426eab1/2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd3faa714bbb7a907/668c54d794a8162a6907e50e/3.png

Aastha Mahendru

Content by Aastha Mahendru

Streamlining Log Management to Amazon S3 Using Atlas Push-based Log Exports With HashiCorp Terraform