How to Get Mobile Data Sync Right with Mobile Backend as a Service (MBaaS)

Steve Jurczak
January 4, 2023

Twenty years ago, Watts Humphrey, known today as the "Father of Software Quality," declared that every business is a software business. While his insight now seems obvious, digital technology has evolved to where we can add to it: Every business is also a mobile business. According to Gartner, 75% of enterprise data will be generated and processed away from the central data center by 2025. And according to data.ai, 84% of enterprises attribute growth in productivity to mobile apps.

Today, mobile tech transforms every aspect of business. It enables the workforce through point-of-sale, inventory, service, and sales. It streamlines critical business processes like self-checkout and customer communications. And it powers essential work devices from telemetry to IoT to manufacturing.

The data businesses capture on mobile and edge devices can be used to improve operational efficiency, drive process improvements, and deliver richer, real-time app experiences. But all of this requires a solution for synchronizing mobile data with backend systems, where it can be combined with other historical data, analyzed, or fed into predictive intelligence algorithms to surface new insights and trigger other value-add activities. But syncing mobile data with backend systems can be hard for a number of reasons.

Mobile devices are constantly going in and out of coverage. When connections break and then resume, conflicts emerge between edits that were made on devices while offline and other data that's being processed on the backend. So conflict resolution becomes a crucial part of ensuring changes on the mobile device are captured on the backend in a way that ensures data integrity.

Sync and swim

Apps that are not designed with backend sync in mind can take a long time to load, are prone to crashing, and show stale information. When apps don’t deliver positive experiences, people stop trusting them — and stop using them.

On the other hand, an app with robust sync between a device’s local data store and the back end lets workers see live data across users and devices, allowing for real-time collaboration and decision-making. According to Deloitte, 70% of workers don’t sit at a desk every day, so the ability to sync data will increasingly drive business outcomes.

Indian startup, FloBiz, uses MongoDB Atlas Device Sync to handle the difficult job of keeping the mobile, desktop, and web apps in sync. This means even if multiple users were using the same account, going offline and online, there would be no issues, duplications or lost data.

Why data sync is difficult

A lot of organizations choose to build their own sync solutions. DIY solutions can go one of two ways: overly complex or oversimplified, resulting in sync that happens only a few times a day or in only one direction. It can be complicated and time-consuming for developers to write their own conflict-resolution code because building data sync the right way takes potentially thousands of lines of code. Developers frequently underestimate the challenge because it seems straightforward on the surface. They assume sync consists simply of the application making a request from the server, receiving some data, and using that data to update the app’s UI on the device. But when building for mobile devices this is a massive oversimplification.

Representation for how people perceive building sync versus what it is actually like; visualized by an iceberg. Above the surface is what people thing building sync is like, which is as follows; Step 1, applications request. Step 2, received data. Step 3, Update UI. Below the surface of the iceberg is what building sync is actually like, which is a complicated weave of questions and steps intertwined together. There are plenty of chances for the app to fail, which would result in a retry of the system.
Building data sync can be more complicated than people assume.

When developers attempt to build their own sync tool, they typically use RESTful APIs to connect the mobile app with the backend and exchange data between them. Mobile apps are often built more like web apps in the beginning. But once the need to handle offline scenarios arises, and because some functionality requires local persistence, then it becomes necessary to add a mobile database. Syncing with that mobile database then becomes a challenge. The exchange of data between the device and the back end gets complicated. It requires the developer to anticipate numerous offline use cases and write complex conflict-resolution code. It can be done, but it’s a time-consuming process that’s not guaranteed to solve all use cases. When data is requested, applications need to understand whether a network is available, and if not, whether the appropriate data is stored locally, leading to complex query, retry, and error handling logic.

The worst part about all this complexity is that it’s non-differentiating, meaning it doesn’t set the business apart from the competition. Users expect the functionality powered by data sync and won’t tolerate anything less.

An integrated, out-of-the-box solution

MongoDB's Atlas Device Sync combined with Realm is a mobile backend as a service (MBaaS) solution that enables developers to build offline-first applications that automatically refresh when a connection is reestablished. Local and edge data persistence is managed by Realm, a development platform designed for modern, data-driven applications. Developers use Realm to build mobile, web, desktop, and IoT apps. Realm is a fast and scalable alternative to SQLite and Core Data for client-side persistence. The bidirectional data synchronization service between Realm and MongoDB Atlas allows businesses to do more with their data at the edge by tapping into some of MongoDB’s more powerful data processing capabilities in the cloud. Complex synchronization problems such as conflict resolution are handled automatically by MongoDB’s built-in sync.

To learn more about the challenges of building real-time mobile apps that scale, with sample use cases about how thousands of businesses are handling it today, download our white paper, Building Real-time Mobile Apps that Scale.

← Previous

Introducing MongoDB Spark Connector Version 10.1

Today, MongoDB released version 10.1 of the MongoDB Spark Connector. In this post, we highlight key features of this new release. Microbatch streaming support The MongoDB Spark connection version 10 introduced support for Apache Structured Spark Streaming. In this initial release, continuous mode streaming was the only mode supported. In this 10.1 update, microbatch mode is now supported, enabling you to stream writes to destinations that currently do not support continuous mode streams, such as Amazon S3 storage. Increased control of write behavior When the Spark Connector issues a write, the default behavior is for an upsert to occur. This can cause problems in some scenarios in which you may not want an upsert, such as with time series collections. There is a new configuration parameter, upsertDocument , that, when set to false, will only issue insert statements on write. solar.write.format("mongodb").mode("append").option("database", "sensors").option("collection", "panels").option("upsertDocument", "false").save() In the above code snippet we are writing to the "panels" time series collection by setting the upsertDocument to false. Alternatively, you can set operationType to the value, “insert”. Setting this option will ignore any upsertDocument option set. Support for BSON types The data types supported in BSON are not exactly the same as those supported in a Spark dataframe. For example, Spark doesn't support ObjectId as a type specifically. To mitigate these scenarios where you need to leverage different BSON types, you can now set the new configuration values : spark.mongodb.read.outputExtendedJson=<true/false> spark.mongodb.write.convertJson=<true/false> This will enable you to effectively leverage BSON datatypes within your Spark application. Call to action Version 10.1 of the MongoDB Spark Connector continues to enhance the streaming capabilities with support for microbatch processing. This version also adds more granular support for writing to MongoDB supporting use cases like time series collections. For those users wanting to upgrade from the 3.x version but could not because of lack of BSON data type support, the 10.1 version now provides an option for using BSON data types. To learn more about the MongoDB Spark Connector check out the online documentation . You can download the latest version of the MongoDB Spark Connector from the maven repository .

January 4, 2023
Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024