Docs Home

/ /

Manage Connections

Docs Home

Development

Atlas Stream Processing

Manage Connections

Docs Home

Development

Atlas Stream Processing

Manage Connections

Add a Kinesis Private Link Connection

Atlas Stream Processing supports creating Private Link connections to AWS Kinesis streams. Select an option to display the procedure to add the connection using your preferred interface:

Add an AWS Kinesis Private Link Connection through the Atlas Administration API

To add an AWS Kinesis Private Link connection to your stream processing workspace through the Atlas Administration API, perform these steps:

Set up Unified AWS Access.

Follow the procedure described in Set Up Unified AWS Access.

Ensure that you grant your IAM role the following permissions:

Connection Type	AWS Permissions
Kinesis as Source	`kinesis:DescribeStreamSummary` `kinesis:ListShards` `kinesis:SubscribeToShard`
Kinesis as Sink	`kinesis:DescribeStreamSummary` `kinesis:PutRecords`
Kinesis as Source and Sink	`kinesis:DescribeStreamSummary` `kinesis:ListShards` `kinesis:SubscribeToShard` `kinesis:PutRecords`

The following example policy grants the AWS principal the required Source and Sink permissions for any Kinesis data stream:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid": "ExamplePolicy",
         "Effect": "Allow",
         "Principal":{
            "AWS":"<yourAtlasAWSAccountArn>"
         },
         "Action": [
            "kinesis:ListShards",
            "kinesis:DescribeStreamSummary",
            "kinesis:SubscribeToShard",
            "kinesis:PutRecords"
         ],
         "Resource": [
            "arn:aws:kinesis:<awsRegion>:<awsAccountId>:stream/<kinesisStreamName>",
            "arn:aws:kinesis:*:<awsAccountId>:*/*/consumer/*:*"
         ]
      }
   ]
}

The consumer resource is required to use a Kinesis data stream as a source.

Note the ARN value in Statement.Principal.AWS to use later in this procedure.

In Atlas, go to the Users page.

If it's not already displayed, select your desired organization from the Organizations menu in the navigation bar.
Click All Projects in the sidebar under the Identity & Access section, and select your desired project.
Click Project Identity & Access in the sidebar under the Security section.

The Users page displays.

Configure a Service Account.

Create a service account.

Grant this service account the following roles within your Atlas Stream Processing project:

Create the Private Link Connection.

The Atlas Administration API provides an endpoint to Create One Private Link.

For an AWS Kinesis Private Link connection, set the following key-value pairs:

Key	Value
`vendor`	`"KINESIS"`
`provider`	`"AWS"`
`region`	The AWS region in which you create the endpoint.

Use Return All Private Link Connections to check the state of your endpoint. When it reaches a DONE state, note the value of the _id field and proceed to the next step.

Create the Kinesis Private Link Connection.

The Atlas Administration API provides an endpoint to Create One Stream Connection.

For an AWS Kinesis Private Link connection, set the following key-value pairs:

Key	Value
`name`	The name you want to give to the connection.
`type`	"AWSKinesisDataStream"
`aws.roleArn`	Value of the ARN noted in an earlier step.
`networking.access.type`	"PRIVATE_LINK"
`networking.access.connectionId`	The `_id` value in the response when you create the Private Link connection.

Add an AWS Kinesis Private Link Connection through the Atlas UI

To add an AWS Kinesis Private Link connection to your stream processing workspace through the Atlas UI, perform these steps:

In Atlas, go to the Database & Network Access page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Database & Network Access under the Security heading.

The Database & Network Access page displays.

Navigate to the Atlas Stream Processing private endpoint interface.

In the sidebar, click Private Endpoint.
Click the Atlas Stream Processing tab.
If you have not created an Atlas Stream Processing private endpoint previously, click Create endpoint. If you have, click Add ASP Endpoint.

Select your cloud provider and vendor.

Set Cloud Provider to AWS.
Set Vendor to Kinesis.
Click Next, enter service details

Provide your AWS Kinesis endpoint region.

Your endpoint must be in the same region in which you intend to host the stream processors that use it.

Set up Unified AWS Access.

Follow the procedure described in Set Up Unified AWS Access.

Ensure that you grant your IAM role the following permissions:

Connection Type	AWS Permissions
Kinesis as Source	`kinesis:DescribeStreamSummary` `kinesis:ListShards` `kinesis:SubscribeToShard`
Kinesis as Sink	`kinesis:DescribeStreamSummary` `kinesis:PutRecords`
Kinesis as Source and Sink	`kinesis:DescribeStreamSummary` `kinesis:ListShards` `kinesis:SubscribeToShard` `kinesis:PutRecords`

The following example policy grants the AWS principal the required Source and Sink permissions for any Kinesis data stream:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid": "ExamplePolicy",
         "Effect": "Allow",
         "Principal":{
            "AWS":"<yourAtlasAWSAccountArn>"
         },
         "Action": [
            "kinesis:ListShards",
            "kinesis:DescribeStreamSummary",
            "kinesis:SubscribeToShard",
            "kinesis:PutRecords"
         ],
         "Resource": [
            "arn:aws:kinesis:<awsRegion>:<awsAccountId>:stream/<kinesisStreamName>",
            "arn:aws:kinesis:*:<awsAccountId>:*/*/consumer/*:*"
         ]
      }
   ]
}

The consumer resource is required to use a Kinesis data stream as a source.

Note the ARN value in Statement.Principal.AWS to use later in this procedure.

In Atlas, go to the Stream Processing page for your project.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Streaming Data heading.

The Stream Processing page displays.

Go to the Connection Registry.

Locate the overview panel of the stream processing workspace you want to modify and click Manage.
In the sidebar, click Connection Registry.

Add a new connection.

Click + Add connection.
Select an AWS Kinesis connection.
Provide a Connection Name. Each connection name must be unique within a stream processing workspace. This is the name used to reference the connection in Atlas Stream Processing aggregations.
Click the PrivateLink button. From the dropdown menu, select the private endpoint you created earlier.
From the AWS IAM Role ARN dropdown, select the ARN of the unified access role you authorized in a prior step.
Click Add connection.

Back

S3 Private Link

AWS Lambda Private Link