Atlas Stream Processing

Streaming data is generated continuously from a wide range of sources. IoT sensors, microservices, and mobile devices are all common sources of high-volume streams of data. The continuous nature of streaming data as well as its immutability make it unique from static data at rest in a database.

Stream processing is continuously ingesting and transforming event data from an event messaging platform (like Apache Kafka) to perform various functions. This could mean creating simple filters to remove unneeded data, performing aggregations to count or sum data as needed, creating stateful windows, and more. Stream processing can be a differentiating characteristic in event-driven applications, allowing for more reactive, responsive customer experiences.

Streaming data lives inside of event streaming platforms (like Apache Kafka), and these systems are essentially an immutable distributed log. Event data is published and consumed from event streaming platforms using APIs.

Developers need to use a stream processor to perform more advanced processing, such as stateful aggregations, window operations, mutations, and creating materialized views. These are similar to the operations one does when running queries on a database, except that stream processing continuously queries an endless stream of data. This area of streaming is an emergent technology with solutions such as Apache Flink and Spark Streaming quickly gaining traction.

With Atlas Stream Processing, MongoDB provides developers with a better way to process streams for use in their applications while leveraging the aggregation framework.

Stream processing is an increasingly critical component for building responsive, event-driven applications. By adding stream processing functionality as a native capability in MongoDB Atlas, we're helping more developers build innovative applications leveraging our multi-cloud developer data platform.

Stream processing happens continuously. In the context of building event-driven applications, stream processing enables reactive and compelling experiences like real-time notifications, personalization, route planning, and predictive maintenance.

Batch processing does not work on continuously produced data. Instead, batch processing works by gathering data over a specified period of time and then processing that static data as needed. An example of batch processing is a retail business collecting sales at the close of business each day for reporting purposes or updating inventory levels.

Atlas Stream Processing extends the aggregation pipeline with stages for processing continuous data streams. These stages combine with existing aggregation stages built into the default mongod process, enabling developers to perform many of the same operations on continuous data as they can perform on data at rest.

Yes. Atlas Stream Processing uses checkpoint documents to capture the state of a stream processor once the last operator of a stream processor completes. This is an important capability for restarting a processor after failure, without requiring significant re-processing of data.

Atlas Stream Processing supports the $validate operator in the MongoDB Query API. Using this operator, developers can ensure that data values and types conform to a schema, optionally routing data to a dead letter queue (DLQ) when it fails validation criteria.

As of August 2024, Atlas Stream Processing supports AWS across the U.S., Europe, and APAC. Support for other cloud providers and regions is coming soon.