Atlas Stream Processing is Now in Public Preview

Clark Gates-George and Joe Niemiec
February 13, 2024 | Updated: August 1, 2024
#Atlas Stream Processing

Update May 2, 2024: Atlas Stream Processing is now generally available. Read our blog to learn more.

This post is also available in: Deutsch, Français, Español, Português, Italiano, 한국인, 简体中文.

Today, we’re excited to announce that Atlas Stream Processing is now in public preview. Any developer on Atlas interested in giving it a try has full access. Learn more in our docs or get started today.

Listen to the MongoDB Podcast to learn about the Atlas Stream Processing public preview from Head of Streaming Products, Kenny Gorman.

Developers love the flexibility and ease of use of the document model, alongside the Query API, which allows them to work with data as code in MongoDB Atlas. With Atlas Stream Processing, we are bringing these same foundational principles to stream processing.

A report covering the topic published by S&P Global Market Intelligence 451 Research had this to say, “A unified approach to leveraging data for application development — the direction of travel for MongoDB — is particularly valuable in the context of stream processing where operational and development complexity has proven a significant barrier to adoption."

First announced at .local NYC 2023, Atlas Stream Processing is redefining the experience of aggregating and enriching streams of high velocity, rapidly changing event data, and unifying how to work with data in motion and at rest.

How are developers using the product so far? And what have we learned?

During the private preview, we saw thousands of development teams request access and we have gathered useful feedback from hundreds of engaged teams. One of those engaged teams is the marketing technology leader, Acoustic:

"At Acoustic, our key focus is to empower brands with behavioral insights that enable them to create engaging, personalized customer experiences. To do so, our Acoustic Connect platform must be able to efficiently process and manage millions of marketing, behavioral, and customer signals as they occur. With Atlas Stream Processing, our engineers can leverage the skills they already have from working with data in Atlas to process new data continuously, ensuring our customers have access to real-time customer insights."
John Riewerts, EVP, Engineering at Acoustic

Other interesting use cases include:

A leading global airline using complex aggregations to rapidly process maintenance and operations data, ensuring on-time flights for their thousands of daily customers,
A large manufacturer of energy equipment using Atlas Stream Processing to enable continuous monitoring of high-volume pump data to avoid outages and optimize their yields, and
An innovative enterprise SaaS provider leveraging the rich processing capabilities in Atlas Stream Processing to deliver timely and contextual in-product alerts to drive improved product engagement.

These are just a few of the many use-case examples that we’re seeing across industries. Beyond the use cases we’ve already seen, developers are giving us tons of insight into what they’d like to see us add to in the future.

In addition to enabling continuous processing of data in Atlas databases through change streams, it’s exciting to see developers using Atlas Stream Processing with their Kafka data hosted by valued partners like Confluent, Amazon MSK, Azure Event Hubs, and Redpanda. Our aim with developer data platform capabilities in Atlas has always been to make for a better experience across the key technologies relied on by developers.

What’s new in the public preview?

That brings us to what’s new. As we scale to more teams, we’re expanding functionality to include the most requested feedback gathered in our private preview. From the many pieces of feedback received, three common themes emerged:

Refining the developer experience
Expanding advanced features and functionality
Improving operations and security

Refining the developer experience

In private preview, we established the core of the developer experience that is essential to making Atlas Stream Processing a natural solution for development teams. And in public preview, we’re doubling down on this by making two additional enhancements:

VS Code integration
The MongoDB VS Code plugin has added support for connecting to Stream Processing instances. For developers already leveraging the plugin, teams can create and manage processors in a familiar development environment. This means less time switching between tools and more time building your applications!
Improved dead letter queue (DLQ) capabilities
DLQ support is a key element for powerful stream processing and in public preview, we’re expanding DLQ capabilities. DLQ messages will now display themselves when executing pipelines with sp.process() and when running .sample() on running processors, allowing for a more streamlined development experience that does not require setting up a target collection to act as a DLQ.

Expanding advanced features and functionality

Atlas Stream Processing already supported many of the key aggregation operators developers are familiar with in the Query API used with data at rest. We've now added powerful windowing capabilities and the ability to easily merge and emit data to an Atlas database or to a Kafka topic. Public preview will add even more functionality demanded by the most advanced teams relying on stream processing to deliver customer experiences:

$lookup
Developers can now enrich documents being processed in a stream processor with data from remote Atlas clusters, performing joins against fields from the document and the target collection.
Change streams pre- and post-imaging
Many developers are using Atlas Stream Processing to continuously process data in Atlas databases as a source through change streams. We have enhanced the change stream $source in public preview with support for pre-and post-images. This enables common use cases where developers need to calculate deltas between fields in documents as well as use cases requiring access to the full contents of a deleted document.
Conditional routing with dynamic expressions in merge and emit stages
Conditional routing lets developers use the value of fields in documents being processed in Atlas Stream Processing to dynamically send specific messages to different Atlas collections or Kafka topics. The $merge and $emit stages also now support the use of dynamic expressions. This makes it possible to use the Query API for use cases requiring the ability to fork messages to different collections or topics as needed.
Idle stream timeouts
Streams without advancing watermarks due to a lack of inbound data can now be configured to close after a period of time emitting the results of the windows. This can be critical for streaming sources that have inconsistent flows of data.

Improving operations and security

Finally, we have invested heavily over the past few months in improving other operational and security aspects of Atlas Stream Processing. A few of the highlights include:

Checkpointing
Atlas Stream Processing now performs checkpoints for saving a state while processing. Stream processors are continuously running processes, so whether due to a data issue or infrastructure failure, they require an intelligent recovery mechanism. Checkpoints make it easy to resume your stream processors from wherever data stopped being collected and processed.
Terraform provider support
Support for the creation of connections and stream processing instances (SPIs) is now available with Terraform. This allows for infrastructure to be authored as code for repeatable deployments.
Security roles
Atlas Stream Processing has added a project-level role, giving users just enough permission to perform their stream processing tasks. Stream processors can run under the context of a specific role, supporting a least privilege configuration.
Auditing
Atlas Stream Processing can now audit authentication attempts and actions within your Stream Processing Instance giving you insight into security-related events.
Kafka consumer group support
Stream processors in now use Kafka consumer groups for offset tracking. This allows users to easily change the position of the processor in the stream for operations and easily monitor for potential processor lag.

A final note on what’s new is that in public preview, we will begin charging for Atlas Stream Processing, using preview pricing (subject to change). You can learn more about pricing in our documentation.

Build your first stream processor today

Public preview is a huge step forward for us as we expand the developer data platform and enable more teams with a stream processing solution that simplifies the operational complexity of building reactive, responsive, event-driven applications, while also offering an improved developer experience.

We can’t wait to see what you build!

Login today or get started with the tutorial, view our resources, or follow the Learning Byte on MongoDB University.

← Previous

Atlas Stream Processing ahora está en vista previa pública.

Hoy, nos complace anunciar que Atlas Stream Processing ya está en versión preliminar pública. La vista previa pública abre el acceso a cualquier desarrollador en Atlas interesado en probarlo. Obtenga más información en nuestros docs o comienza hoy mismo. Escuche el podcast de MongoDB para obtener más información sobre la versión preliminar pública de Atlas Stream Processing del jefe de productos de transmisión, Kenny Gorman. A los desarrolladores les encanta la flexibilidad y facilidad de uso del modelo de documentos, junto con la API de consultas, que les permite trabajar con datos como código. Con Atlas Stream Processing, trasladamos estos mismos principios fundamentales al procesamiento de flujos. Anunciado por primera vez en .local NYC 2023 , Atlas Stream Processing está redefiniendo la experiencia de agregar y enriquecer flujos de datos de transmisiones de alta velocidad y que cambian con rapidez, y unificando cómo trabajar con datos en movimiento y en reposo. ¿Cómo usan el producto los desarrolladores hasta ahora? ¿Y qué aprendimos? Durante la vista previa privada, vimos que miles de equipos de desarrollo solicitan acceso y recopilamos comentarios útiles de cientos de equipos comprometidos. Entre los casos de uso se incluyen los siguientes: Una aerolínea líder mundial que utiliza agregaciones complejas para procesar rápidamente los datos de mantenimiento y operaciones, lo que garantiza la puntualidad de los vuelos para sus miles de clientes diarios Un gran fabricante de equipos de energía que utilizan Atlas Stream Processing para permitir el monitoreo continuo de datos de bombas de alto volumen para evitar interrupciones y optimizar sus rendimientos Un innovador proveedor de SaaS empresarial que aprovecha las capacidades de procesamiento enriquecidas en Atlas Stream Processing para ofrecer alertas oportunas y contextuales en el producto para impulsar una mejor interacción con el producto Estos son solo algunos de los muchos ejemplos de casos de uso que vemos en todas las industrias para Atlas Stream Processing. Más allá de los casos de uso que ya vimos, los desarrolladores nos dan toneladas de información sobre lo que les gustaría que agregáramos a Atlas Stream Processing en el futuro. Además de permitir el procesamiento continuo de los datos en las bases de datos de Atlas a través de change stream, fue impactante ver a los desarrolladores utilizar Atlas Stream Processing con sus datos de Kafka alojados por socios valiosos como Confluent , Amazon MSK , Azure Event Hubs y Redpanda . Nuestro objetivo con las capacidades de la plataforma de datos para desarrolladores siempre es lograr una mejor experiencia en todas las tecnologías clave en las que confían los desarrolladores. ¿Qué hay de nuevo en la vista previa pública? Eso nos trae a las novedades. Y a medida que escalamos a más equipos, expandimos la funcionalidad para incluir los comentarios más solicitados recopilados en nuestra vista previa privada. De los muchos comentarios recibidos, surgieron tres temas comunes: Cómo refinar la experiencia del desarrollador Ampliar las características y funcionalidades avanzadas, y Mejorar las operaciones y la seguridad Cómo refinar la experiencia del desarrollador En la vista previa privada, establecimos el núcleo de la experiencia del desarrollador esencial para hacer de Atlas Stream Processing una solución natural para los equipos de desarrollo. Y en la vista previa pública, publicamos esto haciendo 2 mejoras adicionales: Integración de VS Code El complemento MongoDB VS Code agregó soporte para conectarse a instancias de procesamiento de flujo. Para los desarrolladores que ya utilizan el complemento, esto permite a los equipos crear y gestionar procesadores en un entorno de desarrollo familiar. ¡Pase menos tiempo cambiando entre herramientas y más tiempo creando sus aplicaciones! Capacidades mejoradas de cola de mensajes fallidos (DLQ) La compatibilidad con DLQ es un elemento clave para un potente procesamiento de transmisiones y, en vista previa pública, estamos expandiendo las capacidades de DLQ en Atlas Stream Processing. Los mensajes de la cola de letras muertas ahora se mostrarán al ejecutar tuberías con sp.proceso() y al ejecutar .muestra() en procesadores en ejecución, lo que permite una experiencia de desarrollo más optimizada que no requiere configurar una collection objetivo para actuar como DLQ. Ampliación de características y funcionalidades avanzadas Atlas Stream Processing ya admitía muchos de los operadores de agregación clave con los que los desarrolladores están familiarizados en la API de consulta utilizada con datos en reposo, y agregaba potentes funciones de ventanas y la capacidad de fusionar y emitir datos fácilmente a la base de datos Atlas o a un tema Kafka. La vista previa pública agregará aún más funcionalidades exigidas por los equipos más avanzados que confían en el procesamiento de stream para ofrecer experiencias de cliente: $lookup Los desarrolladores ahora pueden enriquecer los documentos que se procesan en un procesador de transmisión con datos del Atlas cluster remoto, y realizar uniones con campos del documento y la collection de destino. change stream antes y después de la creación de imágenes Muchos desarrolladores están utilizando Atlas Stream Processing para procesar continuamente datos en bases de datos Atlas como fuente a través de change streams. Mejoramos el $source de change stream en vista previa pública con asistencia técnica para imágenes previas y posteriores. Esto permite casos de uso comunes en los que los desarrolladores necesitan calcular deltas entre campos en documentos, así como casos de uso que requieren acceso al contenido completo de un documento eliminado. Enrutamiento condicional con expresiones dinámicas en las etapas de fusión y emisión El enrutamiento condicional permite a los desarrolladores utilizar el valor de los campos de los documentos que se procesan en Atlas Stream Processing para enviar dinámicamente mensajes específicos a diferentes collections de Atlas o temas de Kafka. Las etapas $merge y $emit ahora también admiten el uso de expresiones dinámicas. Esto permite utilizar la API de consultas para casos de uso que requieren la capacidad de bifurcar mensajes a diferentes collections o temas, según sea necesario. Tiempos de espera de transmisión inactiva Las reproducciones sin avanzar en las marcas de agua debido a la falta de datos entrantes ahora se pueden configurar para cerrar después de un período de tiempo que emita los resultados de las ventanas. Esto puede ser fundamental para las fuentes de transmisión que tienen flujos de datos inconsistentes. Mejorar las operaciones y la seguridad Por último, invertimos mucho en los últimos meses en mejorar otros aspectos operativos y de seguridad de Atlas Stream Processing. Entre los aspectos destacados se incluyen los siguientes: Punto de control Atlas Stream Processing ahora realiza puntos de control para guardar el estado durante el procesamiento. Los procesadores de flujos son procesos en ejecución continua, por lo que, ya sea por un problema de datos o por un fallo de la infraestructura, requieren un mecanismo de recuperación inteligente. Los puntos de control facilitan la reanudación de sus procesadores de transmisión desde cualquier lugar donde los datos dejaron de recopilarse y procesarse. Asistencia técnica para proveedores de Terraform El soporte para la creación de Conexiones e Instancias de Procesamiento de Transmisión (SPI) ahora está disponible con Terraform. Esto permite crear la infraestructura como código para implementaciones repetibles. Funciones de seguridad Atlas Stream Processing agregó una función a nivel de proyecto, que otorga a los usuarios los permisos necesarios para realizar sus tareas de procesamiento de transmisiones. Los procesadores de transmisiones pueden ejecutarse bajo el contexto de una función específica, lo que permite una configuración de mínimos privilegios. Asistencia técnica del grupo de consumidores de Kafka Los procesadores de transmisión en Atlas Stream Processing ahora usan grupos de consumidores Kafka para el seguimiento compensatorio. Esto permite a los usuarios cambiar fácilmente la posición del procesador en la transmisión de operaciones y controlar fácilmente el posible retraso del procesador. Una nota final sobre las novedades es que, en la vista previa pública, comenzaremos a cobrar por Atlas Stream Processing, utilizando precios promocionales hasta nuestra versión de disponibilidad general. Puede obtener más información sobre los precios de Atlas Stream Processing en nuestra documentación. Crea tu primer procesador de transmisión hoy La vista previa pública es un gran avance para nosotros a medida que ampliamos la plataforma de datos para desarrolladores y permitimos más equipos con una solución de procesamiento de transmisiones que simplifica la complejidad operativa de crear aplicaciones reactivas, receptivas y basadas en eventos, al tiempo que también ofrece una mejor experiencia de desarrollador. ¡Estamos ansiosos por ver lo que desarrollas! Inicia sesión hoy mismo para comenzar u obtener más información sobre Atlas Stream Processing en nuestra documentación , recursos , tutoriales o Learning Byte en MongoDB University.

February 13, 2024

Next →

MongoDB Database Observability: Integrating with Monitoring Tools

This post is the final in a three-part series on leveraging database observability. Welcome back to our series on Leveraging Database Observability! Our previous post showcased a real-world use case highlighting how MongoDB Atlas’s observability tools effectively tackle database performance challenges. Whether you’re a developer, DBA, or DevOps engineer, our mission is to empower you to harness the full potential of your data through our observability suite . Integrating Atlas metrics with your central enterprise observability tools can simplify your operations. By seamlessly working with popular observability tools, our approach helps teams streamline workflows and enhance visibility across systems. Integrating MongoDB Atlas with third-party monitoring tools MongoDB’s developer data platform combines all essential data services for building modern applications within a unified experience. Our purpose-built observability tools for Atlas environments offer automatic monitoring and optimization, guiding diagnostics tailored specifically for MongoDB. Additionally, we extend Atlas metrics into your existing enterprise observability stack, enabling seamless integration without replacing your current tools. This creates a consolidated, single-pane view that unifies Atlas telemetry with other tech and application metrics, ensuring comprehensive visibility into both database and full-stack performance. This integration empowers you to monitor, receive alerts, and make data-driven decisions within your existing workflows, driving greater efficiency. Below is a quick guide to modifying integration settings through the Atlas UI and the popular integrations we support: Navigate to the Project Integrations page in Atlas. Choose the organization and project you want to configure from the navigation bar. On the Project Integrations page, select the third-party services you’d like to integrate. Configure the chosen services with the required API keys and regions. Critical integrations for your observability platform With Atlas’s Datadog and Prometheus integrations, you can send critical MongoDB metrics to these platforms, empowering detailed, real-time monitoring. Through Datadog , you can track database operation counts, query efficiency, and resource usage, ideal for pinpointing bottlenecks and managing resources. Similarly, Prometheus enables you to monitor essential metrics like query times, connection rates, and memory usage, supporting flexible tracking of database health and performance. Both integrations facilitate proactive detection of issues, alert configuration for resource thresholds, and a cohesive view of Atlas data when visualized in Grafana. Atlas’s integration with PagerDuty streamlines incident management by sending metrics like performance alerts, billing anomalies, and security events directly to PagerDuty. This integration records incidents automatically, notifies teams upon alerts, and supports two-way syncing, ensuring resolved alerts in Atlas are reflected in PagerDuty. It enables efficient incident response and resource allocation to maintain system stability. With Atlas integrations for Microsoft Teams and Slack, you can route key metrics—such as query latency, disk usage, and throughput—to these channels for timely updates. Teams can use these insights for real-time performance monitoring, incident response, and collaboration. Notifications through these platforms ensure your team stays informed on database performance, storage health, and user activity changes as they occur. Use case: Centralized observability with MongoDB Atlas, Datadog, and Slack Let’s walk through a hypothetical scenario for ShopSmart, an e-commerce company that leverages MongoDB Atlas to manage its product catalog and customer data. As traffic surges, the DevOps team faces challenges in monitoring application performance and database health effectively. To tackle these challenges, the team leverages MongoDB Atlas’ integration with Datadog and Slack, creating a powerful observability ecosystem. Integrating MongoDB Atlas with Datadog: The team pushes key MongoDB Atlas metrics into Datadog, such as query performance, connection counts, and Atlas Vector Search metrics. With Datadog, they can visualize these metrics and correlate overall MongoDB performance with their other applications. Out-of-the-box monitors and dedicated dashboards allow the team to track metrics like throughput, average read/write latency, and current connections. This visibility helps pinpoint bottlenecks in real time, ensuring optimal database performance and improving overall application responsiveness. Setting up alerts in Datadog: The team configures alerts for critical metrics like high query latency and increased error rates. When thresholds are breached, Datadog instantly notifies the team. This proactive approach allows the team to address potential performance issues before they impact customers. Integrating Datadog with Slack: To ensure fast communication, alerts are sent directly to the dedicated Slack channel, “ShopSmart-Alerts.” This integration fosters seamless collaboration, enabling the team to discuss and resolve issues in real-time. With these integrations, ShopSmart’s engineering team can monitor performance quickly and address issues efficiently. The unified observability approach enhances operational efficiency, improves the customer experience, and supports ShopSmart’s competitive edge in the e-commerce industry. By leveraging MongoDB Atlas, Datadog, and Slack, the team ensures scalable performance and drives continuous innovation. Conclusion MongoDB Atlas empowers developers and organizations to achieve unparalleled observability and control over their database environments. By seamlessly integrating with central enterprise observability tools, Atlas enhances your ability to monitor performance metrics and ensures you can do so within your existing workflows. This means you can focus on building modern applications confidently, knowing you have the insights and alerts necessary to maintain optimal performance. Embrace the power of MongoDB Atlas and transform your approach to database management—because your applications can thrive when your data is observable. And that wraps up our Leveraging Database Observability series! We hope you learned something new and found value in these discussions. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. To dive deeper and expand your knowledge, check out this learning byte for more insights on the MongoDB observability suite and how it can enhance your database performance.

November 14, 2024