Skip to main content

Amazon S3

Amazon S3 (Simple Storage Service) is a cloud-based storage service provided by Amazon Web Services (AWS), allowing you to store and retrieve data from virtually anywhere on the web. S3 is commonly used with Tonkean for general file storage as well as for automated file orchestration.

Prerequisites

Before connecting Amazon S3 with Tonkean, you need credentials for one of the following authentication methods:

  • Access/Secret Keys – Generate an access key and secret key for an IAM user with access to the relevant S3 bucket.

  • IAM Role – Have your AWS admin create an IAM role in your AWS account. Tonkean assumes this role using AWS STS and receives temporary credentials, so you don't need to generate or share long-lived keys. For details, see Set Up the Amazon S3 Data Source with IAM Role Authentication.

Whichever method you use, the IAM user or role requires the following permissions:

  • s3:PutObject - To allow file upload

  • s3:GetObject - To allow file download

  • s3:DeleteObject - To allow file deletion

  • s3:PutObjectAcl - To allow editing file permissions

  • s3:ListBucket - To allow file data collection

  • s3:ListAllMyBuckets (Only required for Global - List All Buckets permission type)

You may adjust permissions granted as needed based on your use case.

Authenticate with Amazon S3

To use S3 in Tonkean, you must first connect and configure it as a data source:

  1. Select the main nav icon, grid.png, in the upper left and select Enterprise Components. The Enterprise Components screen displays.

  2. Select + New Data Source in the upper right.

    connect_data_sources_add_data_source.png

  3. Select Cloud Application. The Add New Data Source window displays.

    add_cloud_application.png

  4. Enter "Amazon S3" in the search field, then select Amazon S3. The New Amazon S3 Connection window displays.

    add_amazon_s3_data_source.png

  5. Select Create a new connection. The Set Up Data Source window displays.

    new_amazon_s3_connection.png

  6. Select the Auth Method. The required configuration fields and available options differ based on which authentication method you select.

    • Access/Secret Keys (default) – Authenticate with an access key and secret key generated for your S3 bucket.

    • IAM Role - Authenticate by allowing Tonkean to assume an IAM role in your AWS account.

    See the relevant section below for your selected authentication method.

Set Up the Amazon S3 Data Source with Access/Secret Keys Authentication

With access/secret keys authentication, Tonkean connects to Amazon S3 using an access key and secret key generated in your AWS account. This is the default authentication method.

For the Access/Secret Keys authentication method, you select a Permission Type: either Global (connecting the entire S3 instance to Tonkean and enabling you to select buckets to monitor) or Restricted (connecting only one specific bucket in S3 to Tonkean).

For the Access/Secret Keys authentication method, the default permission type is Global. See the relevant section below to configure your data source for the Global or Restricted permission type.

Set Up the Amazon S3 Data Source with Global Permission Type

The Global permission type connects the entire S3 instance to Tonkean and lists available buckets for you to connect to:

  1. In the Permission Type field, select Global - List Buckets.

    amazon_s3_set_up_data_source_global.png

  2. Retrieve the required credentials in your AWS instance and populate the following fields:

    • Region

    • Access Key

    • Secret Key

    amazon_s3_set_up_data_source_global_fields_filled.png

  3. Optionally, you may select Enable S3 Event Notifications via SNS Webhooks, turning on the option to use webhooks with your connected S3 bucket.

    To use webhooks with the Amazon S3 integrations, your AWS credentials must have the required permissions to create and manage SNS topics. Ensure your IAM user role has the necessary SMS permissions before enabling the webhooks. See the required permissions below:

    • SNS:CreateTopic

    • SNS:SetTopicAttributes

    • SNS:Subscribe

    • SNS:DeleteTopic

    • s3:PutBucketNotification

    • s3:GetBucketNotification

    The SNS resource prefix is TonkeanS3Topic_

    amazon_s3_set_up_data_source_global_enable_webhooks.png

  4. When finished, select Connect. The bucket and path selection fields display.

  5. In the dropdown provided, select the buckets you want to collect files from, as well as the specific Path if you want to collect files from a bucket subfolder.

    You can only select a subfolder Path if you have one bucket selected.

    amazon_s3_set_up_data_source_global_select_buckets_and_path.png

  6. When finished, select Save. Your Amazon S3 bucket is connected and Tonkean can begin collecting your S3 data.

Set Up the Amazon S3 Data Source with Restricted Permission Type

The Restricted permission type connects a specific bucket to Tonkean.

  1. In the Permission Type field, select Restricted - Specific Buckets.

    amazon_s3_set_up_data_source_restricted.png

  2. Retrieve the required credentials in your AWS instance and populate the following fields:

    • Region

    • Access Key

    • Secret Key

    • Bucket Name

    amazon_s3_set_up_data_source_restricted_fields_filled.png

  3. Optionally, you may select Enable S3 Event Notifications via SNS Webhooks, turning on the option to use webhooks with your connected S3 bucket.

    To use webhooks with the Amazon S3 integrations, your AWS credentials must have the required permissions to create and manage SNS topics. Ensure your IAM user role has the necessary SMS permissions before enabling the webhooks. See the required permissions below:

    • SNS:CreateTopic

    • SNS:SetTopicAttributes

    • SNS:Subscribe

    • SNS:DeleteTopic

    • s3:PutBucketNotification

    • s3:GetBucketNotification

    The SNS resource prefix is TonkeanS3Topic_

    amazon_s3_set_up_data_source_restricted_enable_webhooks.png

  4. When finished, select Connect. The bucket and path selection fields display (however, only the Path field is editable).

  5. If desired, specify a Path if you want to collect files from a bucket subfolder.

    amazon_s3_set_up_data_source_restricted_select_path.png

  6. When finished, select Save. Your Amazon S3 bucket is connected and Tonkean can begin collecting your S3 data.

Set Up the Amazon S3 Data Source with IAM Role Authentication

With IAM role authentication, Tonkean assumes a role in your AWS account instead of using static access keys. Your AWS admin grants Tonkean access by adding two values from Tonkean to the role's trust policy:

  • Tonkean Principal ARN – Identifies the Tonkean AWS account. This value differs for each Tonkean environment.

  • External ID – A unique identifier generated for each connection. Tonkean is a shared environment, so the external ID ensures that only your connection can assume your role.

Because the setup is split between Tonkean and your AWS admin, we recommend completing it in this order:

  1. Get the Role ARN (and the Bucket Name, for the Restricted permission type) from your AWS admin. The role doesn't need to grant Tonkean access yet.

  2. Enter these details in Tonkean, connect, and save the data source.

  3. Provide the Tonkean Principal ARN and External ID to your AWS admin to add to the role's trust policy.

  4. Return to the data source in Tonkean and verify the role.

Tonkean generates a new external ID each time you open the Set Up Data Source window. The external ID isn't stored until you connect and save the data source. If you copy the external ID and then close the window without saving, the value you shared no longer matches your connection, and authentication fails. Always save the data source before providing the external ID to your AWS admin.

Each connection has its own external ID. If you create multiple S3 connections with IAM role authentication, the role's trust policy must include the external ID for each connection. If your organization prefers a single external ID for your entire board, reach out to the Tonkean Support team.

The following is an example trust policy for the customer's IAM role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "<Tonkean Principal ARN>" },
    "Action": "sts:AssumeRole",
    "Condition": { "StringEquals": { "sts:ExternalId": "<External ID>" } }
  }]
}

For the IAM authentication method, you select Permission Type: either Global (connecting the entire S3 instance to Tonkean and enabling you to select buckets to monitor) or Restricted (connecting only one specific bucket in S3 to Tonkean). For the IAM Role authentication method, the default permission type is Restricted. See the relevant section below to configure your data source for the Global or Restricted permission type.

Set Up the Amazon S3 Data Source with IAM Role and Global Permission Type

The Global permission type connects the entire S3 instance to Tonkean and lists available buckets for you to connect to. The IAM role must include the s3:ListAllMyBuckets permission.

  1. In the Auth Method field, select IAM Role. The Permission Type defaults to Restricted - Specific Bucket.

    amazon_s3_set_up_data_source_iam_select_auth_method.png

  2. In the Permission Type field, select Global - List Buckets.

    amazon_s3_set_up_data_source_iam_select_perm_type_global.png

  3. Retrieve the required details from your AWS admin and populate the following fields:

    • Region

    • Role ARN (for example, arn:aws:iam::123456789012:role/example)

    amazon_s3_set_up_data_source_iam_region_and_role_arn.png

  4. Optionally, you may select Enable S3 Event Notifications via SNS Webhooks, turning on the option to use webhooks with your connected S3 bucket.

    You can only enable this option before you select Connect. After you connect, the checkbox is unavailable.

    To use webhooks with the Amazon S3 integration, your IAM role must have the required permissions to create and manage SNS topics. Ensure your IAM role has the necessary SNS permissions before enabling the webhooks. See the required permissions below:

    • SNS:CreateTopic

    • SNS:SetTopicAttributes

    • SNS:Subscribe

    • SNS:DeleteTopic

    • s3:PutBucketNotification

    • s3:GetBucketNotification

    The SNS resource prefix is TonkeanS3Topic_

    amazon_s3_set_up_data_source_iam_enable_s3_event_notifications.png

  5. When finished, select Connect. The Authenticated message displays, along with the bucket and path selection fields.

  6. Select Copy next to the Tonkean Principal ARN and External ID fields, and provide both values to your AWS admin to add to the role's trust policy.

    amazon_s3_set_up_data_source_iam_copy_arn_and_ext_id.png

  7. In the dropdown provided, select the buckets you want to collect files from. If you want to collect files from a bucket subfolder, also select the specific Path.

    You can only select a subfolder Path if you have one bucket selected.

  8. When finished, select Save. Once your AWS admin updates the trust policy, verify the IAM role.

Set Up the Amazon S3 Data Source with IAM Role and Restricted Permission Type

The Restricted permission type connects a specific bucket to Tonkean. The IAM role only needs bucket-scoped permissions for that bucket.

  1. In the Auth Method field, select IAM Role. Restricted - Specific Bucket is selected by default.

    amazon_s3_set_up_data_source_iam_select_auth_method.png

  2. Retrieve the required details from your AWS admin and populate the following fields:

    • Region

    • Role ARN (for example, arn:aws:iam::123456789012:role/example)

    • Bucket Name

    amazon_s3_set_up_data_source_iam_restricted_region_role_arn_and_bucket.png

  3. Optionally, you may select Enable S3 Event Notifications via SNS Webhooks, turning on the option to use webhooks with your connected S3 bucket.

    You can only enable this option before you select Connect. After you connect, the checkbox is unavailable.

    To use webhooks with the Amazon S3 integration, your IAM role must have the required permissions to create and manage SNS topics. Ensure your IAM role has the necessary SNS permissions before enabling the webhooks. See the required permissions below:

    • SNS:CreateTopic

    • SNS:SetTopicAttributes

    • SNS:Subscribe

    • SNS:DeleteTopic

    • s3:PutBucketNotification

    • s3:GetBucketNotification

    The SNS resource prefix is TonkeanS3Topic_

    amazon_s3_set_up_data_source_iam_restricted_enable_s3_event_notifications.png

  4. When finished, select Connect. The Authenticated message displays, along with the Bucket and Path fields.

    A warning also displays indicating that IAM role access hasn't been verified. This is expected if your AWS admin hasn't updated the trust policy yet, and you can still save the data source.

  5. Select Copy next to the Tonkean Principal ARN and External ID fields, and provide both values to your AWS admin to add to the role's trust policy.

    amazon_s3_set_up_data_source_iam_copy_arn_and_ext_id.png

  6. If you want to collect files from a bucket subfolder, specify a Path.

  7. When finished, select Save. Once your AWS admin updates the trust policy, verify the IAM role.

Verify the IAM Role

After your AWS admin adds the Tonkean Principal ARN and External ID to the role's trust policy, confirm that Tonkean can assume the role:

  1. Open your Amazon S3 data source.

  2. On the data source configuration panel, navigate to Connections and then select Edit. The Set Up Data Source window displays.

    amazon_s3_config_connection_edit.png

  3. Confirm that the External ID matches the value in the role's trust policy.

  4. Select Verify next to the Role ARN field.

    • If verification succeeds, the unverified warning no longer displays. No separate success message appears.

    • If verification fails, an error message displays below the Role ARN field. Confirm with your AWS admin that the trust policy includes the correct Tonkean Principal ARN and External ID, and that the role has the required S3 permissions.

  5. Select Save. The IAM role is verified.

    You may select Connect and see "Unable to assume the configured IAM role or access Amazon S3." If so, confirm that the Region, Role ARN, and Bucket Name values are valid.

In this section: