Manage Connections

Each Stream Processing Workspace stores every configured connection to a streaming data source or sink in its connection registry. The procedures on this page walk you through creating and configuring your Atlas Stream Processing connections.

Prerequisites

To create and configure connections, you must have the following:

An Atlas project
An Atlas user with the Project Owner or the Project Stream Processing Owner role to manage a connection registry

Note
While both the Project Owner and Project Stream Processing Owner roles grant the privileges to create, view, edit, and delete connections in the connection registry, the Project Owner role also allows users to perform tasks such as creating database deployments, managing project access, and more.
See Project Roles to learn more about the differences between these roles.
An Atlas cluster

Cross Project Connections

Atlas Stream Processing supports reading from and writing to Atlas clusters in other projects. To enable this functionality, follow these steps:

In Atlas, go to the Organization Settings page.

If it's not already displayed, select your desired organization from the Organizations menu in the navigation bar.
In the sidebar, click Organization Settings.

The Organization Settings page displays.

Toggle Enable Atlas Stream Processing Across Projects on.

To create a cross-project connection, you must have either:

Org Stream Processing Owner privileges.
Stream Processing Project Owner privileges in both Atlas projects participating in the connection.

Considerations

When configuring Atlas Stream Processing connections, ensure you select a connection type suitable to your use case. Atlas Stream Processing supports the following providers:

Provider (See note)	Description	Connection Types
Kafka Broker	Configurable server that receives and stores event streams, organizing them into partitioned topics.	Sink and Source
Atlas Clusters	Cloud-hosted MongoDB server providing both permanent storage and event-driven Change Streams.	Sink and Source
AWSS3	Object storage container hosted on AWS.	Only Sink
AWS Kinesis	Cloud-hosted platform that receives and stores event streams, providing various integrated tools.	Sink and Source
Azure Blob Storage	Object storage container hosted on Azure.	Only Sink
Google Cloud Pub/Sub	Google Cloud-hosted service that supports asynchronous streaming.	Only Sink
HTTPS	HTTPS request to retrieve data through an API endpoint each time a document passes through. Atlas Stream Processing then uses this data to enrich the document that triggered the request.	Mid-pipeline Stage and Sink
External Function Connections	Atlas Stream Processing supports embedding calls to AWS Lambda resources within your Atlas Stream Processing pipeline. You can execute these calls synchronously or asynchronously, and you can pass responses from synchronous calls to subsequent stages in your pipeline. To learn more, see the Atlas Stream Processing External Functions documentation.	Mid-pipeline Stage and Sink

Note

AWS Kinesis and S3, Azure Blob Storage, and Google Cloud Pub/Sub connections require cloud provider access configuration regardless of the network access method you choose:

To set up Unified AWS Access for Kinesis and S3 connections, see Set Up Unified AWS Access.
To set up Azure Service Principal Access for Blob Storage connections, see Set Up and Manage Azure Service Principal Access.
To set up GCP Service Account Access for Pub/Sub connections, see Set Up and Manage Google Cloud Service Account Access.

Note

You cannot use time series collections as a source for Atlas Stream Processing. Time series collections do not support change streams.

Tip

See Time Series Collection Limitations for more information.

Add a Connection to the Connection Registry

Each Atlas Stream Processing connection type has distinct configuration requirements and procedures. To learn more, see Add an Atlas Stream Processing Connection.

View Connections in the Connection Registry

To view a connection to your stream processing workspace:

To list all connections from one stream processing workspace you specify using the Atlas CLI, run the following command:

atlas streams connections list [options]

To return the details for one stream processing connection you specify using the Atlas CLI, run the following command:

atlas streams connections describe <streamConnectionName> [options]

To learn more about the syntax and parameters for the previous commands, see the Atlas CLI documentation for atlas streams connections list and atlas streams connections describe.

Tip

See: Related Links

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Streaming Data heading.

The Stream Processing page displays.

Go to the Connection Registry page.

Locate the overview panel of the stream processing workspace containing the connection registry you want to view.
Click Manage.
Select the Connection Registry tab.

Atlas Stream Processing displays all of the connections available in your instance connection registry.

The Atlas Administration API provides different endpoints for retrieving one or all connections in a connection registry.

View One Connection in a Connection Registry

List All Connections in a Connection Registry

To list all available connections using mongosh, use the sp.listConnections() method. It returns a list of documents describing each connection in the stream processing workspace's connection registry. It has the following syntax:

sp.listConnections()

Modify a Connection from the Connection Registry

To modify a connection from a connection registry:

To modify the details of one connection on the stream processing workspace you specify using the Atlas CLI, run the following command:

atlas streams connections update <connectionName> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections update.

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Streaming Data heading.

The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing workspace that contains the connection registry that you want to modify and click Manage in the upper right.

Select the Connection Registry tab.

Modify the connection.

For the connection that you want to modify in the table, click the pencil icon in the Actions column.
You can edit a connection only if it is not currently in use by a stream processor. If you want to edit a connection in use, stop all stream processors that use the connection first.
Modify the fields you want to modify.
Click Save changes.

The Atlas Administration API provides an endpoint for editing a stream processing workspace.

Update One Stream Connection

Delete a Connection from the Connection Registry

To delete a connection from a connection registry:

To remove one connection from the stream processing workspace you specify using the Atlas CLI, run the following command:

atlas streams connections delete <connectionName> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections delete.

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Streaming Data heading.

The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing workspace that contains the connection registry that you want to modify
Click Manage in the upper right.
Select the Connection Registry tab.

Delete the connection.

For the connection you want to modify in the table, click the trash can icon in the Actions column of that entry.
When prompted, click Delete.

The Atlas Administration API provides an endpoint for editing a stream processing workspace.

Delete One Stream Connection

Test a Connection from the Connection Registry

To verify a connection from a connection registry is working correctly in the Atlas UI:

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Streaming Data heading.

The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing workspace that contains the connection registry that you want to modify.
Click Manage in the upper right.
Select the Connection Registry tab.

Test the connection.

For the connection you want to test in the table, click the link icon in the Actions column of that entry.
For Atlas Database connections, enter the Database Name and Collection Name to test the connection to a specific database and collection.
For Kafka connections, enter the Topic Name to test the connection to a specific topic.
For other connection types, no additional information is required.
Click Test Connection.

List Private Link Connections

The Atlas Administration API provides an endpoint for viewing all Private Link connections configured for Atlas Stream Processing within a specified project.

Return All Private Links

View One Private Link Connection

The Atlas Administration API provides an endpoint for viewing the details of one Private Link connection configured for Atlas Stream Processing within a specified project.

Return One Private Link

Delete a Private Link Connection

The Atlas Administration API provides an endpoint for deleting a Private Link connection configured for Atlas Stream Processing within a specified project.

Delete One Private Link

Back

Manage Stream Processing Workspaces

Add Public Connections