Add an S3 Private Link Connection

Set up Unified AWS Access.

Follow the procedure described in Set Up Unified AWS Access.

Ensure that you grant your IAM role ListAllMyBuckets and PutObject permissions. The following example policy grants the AWS principal these permissions for any S3 bucket:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect": "Allow",
         "Principal":{
            "AWS":"<yourAtlasAWSAccountArn>"
         },
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::*"
         ]
      }
   ]
}

Note the ARN value in Statement.Principal.AWS to use later in this procedure.

In Atlas, go to the Users page.

1. If it's not already displayed, select your desired organization from the Organizations menu in the navigation bar.

2. Click All Projects in the sidebar under the Identity & Access section, and select your desired project.

3. Click Project Identity & Access in the sidebar under the Security section.

The Users page displays.

Configure a Service Account.

Create a service account.

Grant this service account the following roles within your Atlas Stream Processing project:

• Project Owner

• Project Stream Processing Owner

Create the Private Link Connection.

The Atlas Administration API provides an endpoint to Create One Private Link

For an AWS S3 Private Link connection, set the following key-value pairs:

Use Return All Private Link Connections to check the state of your endpoint. When it reaches a DONE state, note the value of the _id field and proceed to the next step.

Create the S3 Connection.

The Atlas Administration API provides an endpoint to Create One Connection.

For an AWS S3 Private Link connection, set the following key-value pairs:

Set up Unified AWS Access.

Follow the procedure described in Set Up Unified AWS Access.

Ensure that you grant your IAM role ListAllMyBuckets and PutObject permissions. The following example policy grants the AWS principal these permissions for any S3 bucket:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect": "Allow",
         "Principal":{
            "AWS":"<yourAtlasAWSAccountArn>"
         },
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::*"
         ]
      }
   ]
}

Note the ARN value in Statement.Principal.AWS to use later in this procedure.

In Atlas, go to the Users page.

1. If it's not already displayed, select your desired organization from the Organizations menu in the navigation bar.

2. Click All Projects in the sidebar under the Identity & Access section, and select your desired project.

3. Click Project Identity & Access in the sidebar under the Security section.

The Users page displays.

Create an API key.

The Create One Connection API endpoint requires digest authorization when creating an S3 Connection. To support this, you must create an API Key.

1. In the Project Access Manager, select the Applications tab, then click API Keys.

2. Click Create API Key. Provide a short description for the key.

3. In the Project Permissions dropdown menu, select both the Project Stream Processing Owner and Project Owner roles. Click Next.

4. Save both the public and private keys to use later in this procedure.

Create the Private Link Connection.

To a Private Link endpoint for a stream processor connection using the Atlas CLI, run the following command:

atlas streams privateLinks create [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams privateLinks create.

In your configuration file, set the following key-value pairs:

Use atlas streams privateLinks list command to check the state of your endpoint. When it reaches a DONE state, note the value of the _id field and proceed to the next step.

Create the S3 Connection.

To create one connection for the stream processing workspace you specify using the Atlas CLI, run the following command:

atlas streams connections create [connectionName] [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections create.

In your configuration file, set the following key-value pairs:

In Atlas, go to the Database & Network Access page for your project.

1. If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.

2. If it's not already displayed, select your project from the Projects menu in the navigation bar.

3. In the sidebar, click Database & Network Access under the Security heading.

The Database & Network Access page displays.

Navigate to the Atlas Stream Processing private endpoint interface.

1. In the sidebar, click Private Endpoint.

2. Click the Atlas Stream Processing tab.

   If you have not created an Atlas Stream Processing private endpoint previously, click Create endpoint. If you have, click Add ASP Endpoint.

Select your cloud provider and vendor.

1. Set Cloud Provider to AWS.

2. Set Vendor to S3.

3. Click Next, enter service details

Provide your AWS S3 endpoint region.

Your endpoint must be in the same region in which you intend to host the stream processors that use it.

Set up Unified AWS Access.

Follow the procedure described in Set Up Unified AWS Access.

Ensure that you grant your IAM role ListAllMyBuckets and PutObject permissions. The following example policy grants the AWS principal these permissions for any S3 bucket:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect": "Allow",
         "Principal":{
            "AWS":"<yourAtlasAWSAccountArn>"
         },
         "Action": [
            "s3:ListAllMyBuckets",
            "s3:PutObject"
         ],
         "Resource": [
            "arn:aws:s3:::*"
         ]
      }
   ]
}

Note the ARN value in Statement.Principal.AWS to use later in this procedure.

In Atlas, go to the Users page.

1. If it's not already displayed, select your desired organization from the Organizations menu in the navigation bar.

2. Click All Projects in the sidebar under the Identity & Access section, and select your desired project.

3. Click Project Identity & Access in the sidebar under the Security section.

The Users page displays.

Create an API key.

The Create One Connection API endpoint requires digest authorization when creating an S3 Connection. To support this, you must create an API Key.

1. In the Project Access Manager, select the Applications tab, then click API Keys.

2. Click Create API Key. Provide a short description for the key.

3. In the Project Permissions dropdown menu, select both the Project Stream Processing Owner and Project Owner roles. Click Next.

4. Save both the public and private keys to use later in this procedure.

In Atlas, go to the Stream Processing page for your project.

1. If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.

2. If it's not already displayed, select your project from the Projects menu in the navigation bar.

3. In the sidebar, click Stream Processing under the Streaming Data heading.

The Stream Processing page displays.

Go to the Connection Registry.

1. Locate the overview panel of the stream processing workspace you want to modify and click Configure.

2. Select the Connection Registry tab.

Click + Add connection.

Add a new connection.

1. Select an S3 connection.

2. Provide a Connection Name. Each connection name must be unique within a stream processing workspace. This is the name used to reference the connection in Atlas Stream Processing aggregations.

3. Click the PrivateLink button. From the dropdown menu, select the private endpoint you created earlier.

4. From the AWS IAM Role ARN dropdown, select the ARN of the unified access role you authorized in a prior step.

5. Click Add connection.