Jim Walker

Microservices: Realizing the Benefits Without the Complexity

The microservice architecture has emerged as the preferred, modern approach for developers to build and deploy applications on the cloud. It can help you deliver more reliable applications, and address the scale and latency concerns for System Reliability Engineers (SREs) and operations. But microservices aren't without their hangups. For developers, microservices can lead to additional complexity and cognitive overhead, such as cross-service coordination, shared states across multiple services, and coding and testing failure logic across disconnected services. While the monolith was suboptimal for compute and scale efficiencies, the programming model was simple. So the question is, can we get the best of both worlds? In addition, how do we make the individual services easier to build and adapt to changing requirements? Since, at their core, microservices provide access to and perform operations on data, how do we architect services so that developers can easily work with data? How can we make it easier for developers to add new types of data and data sources and perform a wide variety of data operations without the complexity of managing caches and using multiple query languages (SQL, full-text and vector search, time series, geospatial, etc.) The development complexity associated with microservice architectures occurs at two levels: service orchestration and service data management. The diagram below depicts this complexity. At the orchestration level, a typical application may support tens or hundreds of processes, and each may have thousands or millions of executions. To make this work, services are often connected by a patchwork of queues. Developers spend quite a bit of time tracking and managing all the various workflows. The sheer scale necessitates a central mechanism to manage concurrent tasks and sharded databases to manage the state of millions of concurrent workflow instances. To add more complexity, each microservice is deployed using a set of data platforms including RDBMS, caches, search engines, and vector and NoSQL databases. Developers must work with multiple query languages, write code to keep data in sync among these platforms and write code to deal with edge cases when invariably data or indexes are not in sync. Finally, developer productivity is inhibited by the brittleness of RDBMS, which lacks flexibility when trying to incorporate new or changing data types. As a result, microservice applications often end up with complex architectures that are difficult to develop against and maintain in terms of both the individual microservices and the service orchestration. Realizing the benefits without the complexity One approach to address these microservice challenges is to combine two technologies: Temporal and MongoDB. Both give you the benefits of microservices while simplifying the implementation of service orchestration. Together, they allow developers to build services that can easily handle a wide variety of data, eliminate the need to code for complex infrastructure and reduce the likelihood of failure. They simplify the data model and your code. In one real-world example, open-source indexing company Mixpeek leverages the combination of MongoDB and Temporal to provide a platform enabling organizations to easily incorporate multi-modal data sources in AI applications. Mixpeek’s CEO Ethan Steininger states, “Temporal’s durable execution guarantees and MongoDB's flexible data model are core components of Mixpeek’s multimodal data processing and storage. Combined, they enable our users to run high volume ML on commodity hardware without worrying about dropped jobs.” MongoDB and Temporal: Build like a monolith with durable microservices Both MongoDB and Temporal were built by developers, for developers. They both use a code-first approach to solving the complex infrastructure needs of our modern applications within our application code and empower developers to be more productive. They are part of an emerging development stack that greatly simplifies data and all the cross-functional coordination we need in our cloud applications. Ultimately, the combination of these two developer-focused platforms allows you to simplify design, development, and testing of microservice-based applications. With the document model of MongoDB, you model data as real world objects and not tables, rows, and columns. With Temporal, you design your end-to-end service flows as workflows as described by domain experts without having to explicitly identify every edge case and exception (Temporal handles those implicitly). Temporal and MongoDB provide the same benefits that, when combined, multiply in value. You become more agile, as not only can everyone understand your code better, but you are no longer challenged by the cognitive overload of trying to coordinate, comprehend, and test a web of disconnected and complex services. Together, they allow us to reliably orchestrate business processes within apps that are all speaking the language of the data itself. Combining Temporal and MongoDB results in the simplified architecture shown below. Temporal enables orchestration to be implemented at a higher level of abstraction, eliminating much of the event management and queuing complexity. MongoDB, in turn, provides a single integrated data platform with a unified query language thereby eliminating much of the data management complexity. Let’s examine MongoDB and Temporal in more depth to better understand their capabilities and why they facilitate the rapid development of microservices-based applications. MongoDB: Simplifying microservice data MongoDB's features align well with the principles of microservices architectures. It reduces the need for niche databases and the associated costs of deploying and maintaining a complicated sprawl of data technologies. More explicitly, MongoDB delivers key benefits for microservice development: Flexible schema, flexible services: Unlike relational databases with rigid schemas, MongoDB's document model allows microservices to easily evolve as data requirements change. Distributed scale for data-heavy, distributed services: MongoDB scales horizontally by adding more partitions to distribute the load. This aligns with the modular nature of microservices, where individual services can scale based on their specific needs. Unified query language reduces microservice sprawl: MongoDB supports a diverse set of data operations without requiring multiple data platforms (caches, vector, and text search engines, time series, geospatial, etc.) Operational efficiency: MongoDB Atlas, the cloud-based version of MongoDB, simplifies managing databases for microservices. It handles provisioning, backups, and patching, freeing developers to focus on core responsibilities. Integrated developer data platform: The integrated developer data platform delivers an intuitive set of tools to build services that support mobile clients, real-time analytics, data visualization, and historical analysis across many service databases. With MongoDB, development teams use one interface for all their services and run it anywhere, even across clouds. Also, it provides a data foundation for your microservices that is highly available, scalable, and secure. It greatly simplifies microservices development so that you can focus on your business problems and not data. Temporal: Don't coordinate services, orchestrate them Temporal delivers an open-source, durable execution solution that removes the complexity of building scalable distributed microservices. It presents a development abstraction that preserves the complete application state so that in the case of a host or software failure, it can seamlessly migrate execution to another machine. This means you can develop applications as if failures—like network outages or server crashes—do not exist. Temporal handles these issues, allowing you to focus on implementing business logic rather than coding complex failure detection and recovery routines. Here's how Temporal simplifies application development: Durable workflows: Temporal maintains the state and progress of a defined workflow across multiple services, even in the face of server crashes, network partitions, and other types of failures. This durability ensures that your application logic can resume where it left off, making your overall application more resilient. Simplifies failure handling: Temporal abstracts away the complex error handling and retry logic that developers typically have to implement in distributed systems. This abstraction allows developers to focus on business logic rather than the intricacies of ensuring their end-to-end services can handle failures gracefully. Scale: Temporal applications are inherently scalable and capable of handling billions of workflow executions. Long-running services: Temporal supports long-running operations, from seconds to years, with the same level of reliability and scalability. By providing a platform that handles the complexities of distributed systems, Temporal allows developers to concentrate on implementing business logic in their services. This focus can lead to faster development times and more reliable applications, as developers are not bogged down by the intricacies of state management, retries, and error handling. The next generation of microservices development is here Developers want to code. They want to solve business problems. They do not want to be bogged down by the complexity of infrastructure failures. They want to model their apps and data so that it is aligned with the real-world entities and domains they are solving for. Using MongoDB and Temporal together solves these complexities. Together, they simplify design, development, and testing of microservice-based applications so that you can focus on business problems and deliver more features faster. Getting started with Temporal and MongoDB Atlas We can help you design the best architecture for your organization’s needs. Feel free to connect with your MongoDB and Temporal account teams or contact us to schedule a collaborative session and explore how Temporal and MongoDB can optimize your AI development process.