Migrating your data from DynamoDB to MongoDB Atlas

Jay Gordon
January 5, 2018 | Updated: May 24, 2021

There may be a number of reasons you are looking to migrate from DynamoDB to MongoDB Atlas. While DynamoDB may be a good choice for a set of specific use cases, many developers prefer solutions that reduce the need for client-side code or additional technologies as requirements become more sophisticated. They may also want to work with open source technologies or require some degree of deployment flexibility. In this post, we are going to explore a few reasons why you might consider using MongoDB Atlas over DynamoDB, and then look at how you would go about migrating a pre-existing workload.

Get building, faster...

MongoDB Atlas is the best way to consume MongoDB and get access to a developer-friendly experience, delivered as a service on AWS. MongoDB’s query language is incredibly rich and allows you to execute complex queries natively while avoiding the overhead of moving data between operational and analytical engines. With Atlas, you can use MongoDB’s native query language to perform anything from searches on single keys or ranges, faceted searches, graph traversals, and geospatial queries through to complex aggregations, JOINs, and subqueries - without the need to use additional add-on services or integrations.

For example, MongoDB’s aggregation pipeline is a powerful tool for performing analytics and statistical analysis in real-time and generating pre-aggregated reports for dashboarding.

Additionally, MongoDB will give you a few extra things which I think are pretty cool:

  • Document Size - MongoDB handles documents up to 16MB in size, natively. In DynamoDB, you’re limited to 400KB per item, including the name and any local secondary indexes. For anything bigger, AWS suggests that you split storage between DynamoDB and S3.
  • Deployment flexibility - Using MongoDB Atlas, you are able to deploy fully managed MongoDB to AWS, Google Cloud Platform, or Microsoft Azure. If you decide that you no longer want to run your database in the cloud, MongoDB can be run in nearly any environment on any hardware so self-managing is also an option.
  • MongoDB has an idiomatic driver set, providing native language access to the database in dozens of programming languages.
  • MongoDB Atlas provides a queryable backup method for restoring your data at the document level, without requiring a full restoration of your database.
  • MongoDB Atlas provides you with over 100 different instance metrics, for rich native alerting and monitoring.
  • Atlas will assist you in finding the right indexes thanks to Performance Advisor. The Performance Advisor utility is on all the time, helping you make certain that your queries are efficient and fast.

Getting Started

In this tutorial, we'll take a basic data set from an existing DynamoDB table and migrate it to MongoDB Atlas. We'll use a free, M0 cluster so you can do this as well at no cost while you evaluate the benefits of MongoDB Atlas.

This blog post makes a couple of assumptions:

You've installed MongoDB on the computer you'll be importing the data from (we need the mongoimport tool which is included with MongoDB) You've signed up for a MongoDB Atlas account (the M0 instance is free and fine for this demonstration)

To begin, we'll review our table in AWS:

This table contains data on movies including the year they were released, the title of the film, with other information about the film contained in a subdocument.

We want to take this basic set of data and bring it into MongoDB Atlas for a better method of querying, indexing, and managing our data long term.

First, ensure your application has stopped your writes if you are in production to prevent new entries into your database. You'll likely want to create a temporary landing page and disable new connections to your DynamoDB. Once you've completed this, navigate to your table in your AWS panel.

Click "Actions" at the top after you've selected your table. Find the "Export to .csv" option and click it.

Now you'll have a CSV export of your data from DynamoDB, let's take a quick look:

  
$ more ~/Downloads/Movies.csv
"year (N)","title (S)","info (M)"
"1944","Arsenic and Old Lace","{    ""actors"" : { ""L"" : [        { ""S"" : ""Cary Grant"" },        { ""S"" : ""Priscilla Lane"" },        { ""S"" : ""Raymond Massey"" }      ]    },    ""directors"" : { ""L"" : [        { ""S"" : ""Frank Capra"" }      ]    },    ""genres"" : { ""L"" : [        { ""S"" : ""Comedy"" },        { ""S"" : ""Crime"" },        { ""S"" : ""Romance"" },        { ""S"" : ""Thriller"" }      ]    },    ""image_url"" : { ""S"" : ""http://ia.media-imdb.com/images/M/MV5BMTI3NTYyMDA0NV5BMl5BanBnXkFtZTcwMjEwMTMzMQ@@._V1_SX400_.jpg"" },    ""plot"" : { ""S"" : ""A drama critic learns on his wedding day that his beloved maiden aunts are homicidal maniacs, and that insanity runs in his family."" },    ""rank"" : { ""N"" : ""4025"" },    ""rating"" : { ""N"" : ""8"" },    ""release_date"" : { ""S"" : ""1944-09-01T00:00:00Z"" },    ""running_time_secs"" : { ""N"" : ""7080"" }  }"

Looks good, let's go ahead and start using MongoDB's open source tools to import this to our Atlas cluster.

Import your data Let's start moving our data into MongoDB Atlas. First, launch a new free M0 cluster (M0s are great for demos but you’ll want to pick a different tier if you are going into production). Once you have a new M0 cluster, you can then whitelist your local IP address so that you may access your Atlas cluster.

Next, you'll want to use the mongoimport utility to upload the contents of Movies.csv to Atlas. I'll provide my connection string, which I can get right from my Atlas control panel so that mongoimport can begin importing our data:

Now I can enter this into my mongoimport command along with some other important options:

  
mongoimport --uri "mongodb://admin:PASSWORD@demoabc-shard-00-00-a7nzr.mongodb.net:27017,demoabc-shard-00-01-a7nzr.mongodb.net:27017,demoabc-shard-00-02-a7nzr.mongodb.net:27017/test?ssl=true&replicaSet=demoabc-shard-0&authSource=admin"  --collection movies --file ~/Downloads/Movies.csv  --type csv --headerline
2017-11-20T14:02:45.612-0500  imported 100 documents

Now that our documents are uploaded, we can log into Atlas with MongoDB Compass and review our data:

This is cool but I want to do something a little bit more advanced. Luckily, MongoDB’s aggregation pipeline will give us the power to do so

We'll need to connect to our shell here; I can press the "CONNECT" button within my Atlas cluster’s overview panel and find the connection instructions for the shell.

Once I am logged in, I can start playing with some different aggregations; here is a basic one that tells us the total number of movies released in 1944 in our collection: 

  
db.movies.aggregate(

  // Pipeline
  [
    // Stage 1
    {
      $match: { year : 1944 }
    },

    // Stage 2
    {
      $group: { _id: null, count: { $sum: 1 } }
    },
  ]
);

With DynamoDB, we would have had to connect our database cluster to Amazon EMR, adding cost, complexity, and latency.

You can configure backups, ensure network security and configure additional user roles for your data all from the MongoDB Atlas UI.

Sign up for MongoDB Atlas today and start building better apps faster.

← Previous

What’s New in MongoDB 3.6. Part 4 – Avoid Lock-In, Run Anywhere

Welcome to part 4 of our MongoDB 3.6 blog series. In part 1 we took a look at the new capabilities designed specifically to help developers build apps faster, including change streams, retryable writes, developer tools, and fully expressive array manipulation In part 2 , we dived into the world of DevOps and distributed systems management, exploring Ops Manager, schema governance, and compression In part 3 we covered what’s new for developers, data scientists, and business analysts with the new SQL-based Connector for BI, richer in-database analytics and aggregations, and the new recommended driver for R In our final part 4, we’ll look at all of the new goodness in our MongoDB Atlas fully managed database service available on AWS, Azure, and GCP, including Cross-region replication for globally distributed clusters, auto-scaling, and more. If you want to get the detail now on everything the new release offers, download the Guide to What's New in MongoDB 3.6 . Run Anywhere Many organizations are turning to the cloud to accelerate the speed of application development, deployment, and data discovery. Replatforming to the cloud gives them the ability to enable self-service IT, to elastically scale resources on demand, and to align costs to actual consumption. But they are also concerned about exposing the business to deeper levels of lock-in – this time from the APIs and services of the cloud providers themselves. Increasingly, users are demanding the freedom to run anywhere: private clouds in their own data center, in the public cloud, or in a hybrid model that combines the two. This flexibility is not available when they build on a cloud-proprietary database from a single vendor. Alternatively, the platform independence provided by MongoDB gives them the ability to respond to business or regulatory changes without incurring the complexity, risk, and time that comes from expensive database migrations whenever they need or want to transition to a new platform. MongoDB Atlas As a fully managed database service, MongoDB Atlas is the best way to run MongoDB in the public cloud. 2017 has already seen major evolutions in the Atlas service, with key highlights including: Expansion beyond Amazon Web Services (AWS) to offer Atlas on Google Cloud Platform (GCP) and Microsoft Azure. Achieving SOC2 Type 1 compliance. The launch of managed database clusters on a shared architecture, including the free M0 instances, and the M2s and M5s, which allow customers to jumpstart their projects for a low and predictable price. A live migration facility to move data from an existing MongoDB replica set into an Atlas cluster with minimal application impact. The addition of the Data Explorer and Real Time Performance Panel, now coming to Ops Manager, as discussed above. MongoDB 3.6 is available as a fully managed service on Atlas, along with important new features to support global applications, and with automated scalability and performance optimizations. Turnkey Global Distribution of Clusters with Cross-Region Replication MongoDB Atlas clusters can now span multiple regions offered by a cloud provider. This enables developers to build apps that maintain continuous availability in the event of geographic outages, and improve customer experience by locating data closer to users. When creating a cluster or modifying its configuration, two options are now available: Teams can now deploy a single MongoDB database across multiple regions supported by a cloud provider for improved availability guarantees. Reads and writes will default to a “preferred region” assuming that there are no active failure or failover conditions. The nearest read preference, discussed below, can be used to route queries to local replicas in a globally distributed cluster. Replica set members in additional regions will participate in the automated election and failover process if the primary member is affected by a local outage, and can become a primary in the unlikely event that the preferred region is offline. Read-only replica set members can be deployed in multiple regions, allowing teams to optimize their deployments to achieve reduced read latency for a global audience. Read preference – providing a mechanism to control how MongoDB routes read operations across members of a replica set – can be configured using the drivers. For example, the nearest read preference routes queries to replicas with the lowest network latency from the client, thus providing session locality by minimizing the effects of geographic latency. As the name suggests, read-only replica set members will not participate in the automated election and failover process, and can never be become a primary. Teams can activate both of the options outlined above in a single database to provide continuous availability and an optimal experience for their users. Figure 1: Globally distributed MongoDB Atlas cluster, providing resilience to regional outages and lower latency experiences for global apps Auto-Scaling Storage and Performance Optimization MongoDB Atlas now supports automatic scaling for the storage associated with a cluster, making it easier for you to manage capacity. Enabled by default, auto-scaling for storage detects when your disks hit 90% utilization and provisions additional storage such that your cluster reaches a disk utilization of 70% on AWS & GCP, or a maximum of 70% utilization on Azure. This automated process occurs without impact to your database or application availability. In addition to auto-storage scaling, the new Performance Advisor discussed earlier for Ops Manager is also available in MongoDB Atlas, providing you with always-on, data-driven insights into query behavior and index recommendations. A Cloud Database Platform for Development & Testing New enhancements to MongoDB Atlas make it the optimal cloud database for spinning up and running test and development environments efficiently. You can now pause your MongoDB Atlas cluster, perfect for use cases where only intermittent access to your data is required, such as development during business hours or temporary testing. While your database instances are stopped, you are charged for provisioned storage and backup storage, but not for instance hours. You can restart your MongoDB Atlas cluster at any time on demand; your cluster configuration will be the same as when you stopped it and public DNS hostnames are retained so no modifications to your connection string are required. MongoDB Atlas clusters can be stopped for up to 7 days. If you do not start your cluster after 7 days, Atlas will automatically start your cluster. Pausing and restarting your MongoDB clusters can be triggered in the MongoDB Atlas UI or via the REST API. Cross-project restores, introduced with Ops Manager 3.6, are also available in MongoDB Atlas, allowing users to restore to different MongoDB Atlas projects than the backup snapshot source. Next Steps That wraps up the final part of our what’s new blog series. I hope I’ve helped demonstrate how MongoDB 3.6 helps you move at the speed of your data . It enables new digital initiatives and modernized applications to be delivered to market faster, running reliably and securely at scale, and unlocking insights and intelligence ahead of your competitors. Change streams, retryable writes, causal consistency, greater query and update expressivity, and Compass Community help developers move faster. Ops Manager, schema validation, enhanced security, end to end compression, and user session management help operations teams scale faster. The MongoDB aggregation pipeline, Connector for BI, and the recommended R driver help analysts and data scientists unlock insights faster. And you have the freedom to run MongoDB anywhere – on-premises, public cloud, and as a service with MongoDB Atlas available on AWS, Azure, and GCP. If you want to get the detail now on everything the new release offers, download the Guide to What's New in MongoDB 3.6 . Alternatively, if you’d had enough of reading about it and want to get started now, then: Spin up MongoDB 3.6 on MongoDB Atlas . Download MongoDB 3.6 to evaluate the new release in your own environment. Sign up for our free 3.6 training from the MongoDB University.

December 27, 2017
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024