How is Atlas Online Archive Implemented?

Greg_Harabedian · 2023-05-31T14:34:17.437Z

I’ve been looking into the Online Archive capability within Atlas. I understand from a user perspective how to set it up and how it works, but not from an architecture/implementation standpoint. When one sets up an “online archive” what is actually happening behind the scenes? Is it just using Atlas Data Lake capabilities to query data that is stored in an S3 bucket? If so, what tier of S3 Storage Class is being used? How is data protected in the S3 bucket and do we have direct access to that S3 bucket outside of Atlas?

Kushagra_Kesav · 2023-06-01T13:28:14.125Z

Hey @Greg_Harabedian,

Thank you for reaching out to the MongoDB Community forums!

Greg_Harabedian:

When one sets up an “online archive” what is actually happening behind the scenes?

MongoDB Atlas Online Archive is a feature of the MongoDB Cloud Data Platform. It allows you to set a rule to automatically archive data off of your Atlas cluster to fully-managed cloud object storage that is optimized for analytical queries. It reformats, creates partition indexes, and partitions data as it is ingested, creating an isolated workload ready to support large and complex analytical queries.

When you set up an Online Archive rule MongoDB Atlas configures a capability that runs on a schedule and safely moves data out of your cluster when it reaches qualification based on the rule configured.

Greg_Harabedian:

Is it just using Atlas Data Lake capabilities to query data that is stored in an S3 bucket?

Under the hood, it is using a storage service that is built on top of S3 and other technologies to store and manage the archival data. This data is then queried through the same federated query capability present in Data Federation.

Greg_Harabedian:

If so, what tier of S3 Storage Class is being used?

It incorporates more storage technologies beyond S3, making it inaccurate to think about it in terms of just S3 Storage Class tiers. We optimize our storage infrastructure to meet various requirements, prioritizing performance, cost, and durability. Categorizing the storage solely based on S3 Storage Class tiers would not accurately represent the diverse capabilities MongoDB Atlas employ.

Greg_Harabedian:

How is data protected in the S3 bucket

MongoDB Atlas encrypts your archived data using Amazon’s server-side encryption S3-managed keys (SSE-S3) for archived data.

Greg_Harabedian:

do we have direct access to that S3 bucket outside of Atlas?

As of now you cannot directly access the S3 bucket outside of MongoDB Atlas.

I hope it answers your questions. In case of any additional questions or concerns, please feel free to reach out.

Best regards,
Kushagra

system · 2023-07-17T08:31:09.174Z

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.