AdministrationData connections

Connecting Amazon S3

Commercial & Enterprise

Connect Amazon S3 buckets to AI Hub.

About this connection

Review the following functionality, limitations, and other considerations when connecting an Amazon S3 bucket.

  • Functionality — Connected Amazon S3 buckets are supported for use as a source of input files, a destination for file output, and as an upstream or downstream integration in deployments. Amazon S3 buckets can also be used as default drives.

  • Authentication — AI Hub supports two authentication methods for Amazon S3 buckets: AWS IAM access keys or AWS IAM roles.

  • Supported content — Any supported file types. You can specify a specific folder path as the mount point, otherwise the bucket’s root directory (/) is mounted and all contents are accessible.

Connecting an Amazon S3 bucket

You can connect your Amazon S3 bucket using the following authentication methods:

Authenticating with an AWS IAM access key

Before you begin

Ensure you’ve set up AWS IAM access key with the required permissions. For a list of permissions, see Access key permissions requirements.

  1. In Workspaces, select a workspace to connect the drive to, then select the Data tab.

  2. Click Add data source, then select Amazon S3.

  3. Select an audience.

    • Workspace members (Recommended) — Connect the drive to the selected workspace. Only members of the selected workspace have access.

    • Organization members — Connect the drive at the organization level, making it available to all workspaces.

  4. Enter a display name for the drive. This name can’t be changed later.

  5. Select Access key as your authentication method.

  6. On the configuration screen, fill in your authentication and bucket details, then click Next.

    SettingRequiredDescription
    Access key IDRequiredYour AWS IAM access key ID.
    Secret access keyRequiredYour AWS IAM secret access key. Review the permissions requirements.
    Bucket nameRequiredThe name of the S3 bucket to use for file storage. Provide the name, not the Amazon Resource Name (ARN).
    RegionRequiredThe region code for your AWS account, such as us-east-1.

    For a full list of region codes, see the AWS Regions and zones documentation.
    Path to driveOptionalA file path to a folder in the S3 bucket where the desired input files are found. Leave empty to accept default (root).
    Server-side encryption typeOptionalSelect the server-side encryption (SSE) type.

    • None — (Default) No server-side encryption.

    • SSE-S3 — Use Amazon-managed server-side encryption of files.

    • SSE-KMS — Use Amazon Key Management Service (KMS) for server-side encryption of files.
    Server-side encryption KMS key IDVisible and required when Server-side encryption type is set to SSE-KMS.The Amazon resource name (ARN) for the KMS key. See the AWS Finding the key ID and key ARN documentation for more information.

  7. Select whether to set the drive as a default drive. Not usually recommended, see Managing default drives for details.

  8. Click Done

Access key permissions requirements

The AWS IAM access key must have the following permissions:

$s3:DeleteObject
>s3:DeleteObjectVersion
>s3:GetObject
>s3:GetObjectAcl
>s3:GetObjectVersion
>s3:PutObject
>s3:PutObjectAcl
>s3:PutObjectVersion
>s3:ListBucket
>s3:ListBucketMultipartUploads
>s3:ListMultipartUploadParts
>s3:AbortMultipartUpload

Authenticating with an AWS IAM role

  1. In Workspaces, select a workspace to connect the drive to, then select the Data tab.

  2. Click Add data source, then select Amazon S3.

  3. Select an audience.

    • Workspace members (Recommended) — Connect the drive to the selected workspace. Only members of the selected workspace have access.

    • Organization members — Connect the drive at the organization level, making it available to all workspaces.

  4. Enter a display name for the drive. This name can’t be changed later.

  5. Select IAM role as your authentication method, then click Next.

  6. Connect a new role or select a previously connected role, then click Next.

    1. If previously added IAM roles are shown, click the + icon next to the role selection dropdown. Otherwise, you’re brought to the Add custom trust policy screen.

    2. Copy the custom trust policy provided. Using the IAM console in the AWS Management Console, configure an IAM role using the custom trust policy. You don’t need to set a permissions boundary.

      See the following AWS documentation for guidance:

    3. Confirm the custom trust policy is added, then click Next.

  7. On the configuration screen, fill in your authentication and bucket details, then click Next.

    SettingRequiredDescription
    IAM role ARNRequiredThe Amazon Resource Name (ARN) for the IAM role being used for authentication. See the AWS IAM identifiers and Find Amazon Resource Names (ARNs) in AMS documentation for details.
    Bucket nameRequiredThe name of the S3 bucket to use for file storage. Provide the name, not the ARN.
    AWS regionRequiredThe region code for your AWS account, such as us-east-1.

    For a full list of region codes, see the AWS Regions and zones documentation.
    Path to driveOptionalA file path to a folder in the S3 bucket where the desired input files are found. Leave empty to accept default (root).
    Server-side encryption typeOptionalSelect the server-side encryption type.

    • None — (Default) No server-side encryption.

    • SSE-S3 — Use Amazon-managed server-side encryption of files.

    • SSE-KMS — Use Amazon Key Management Service (KMS) for server-side encryption of files.
    Server-side encryption KMS key IDVisible and required when Server-side encryption type is set to SSE-KMS.The Amazon resource name (ARN) for the KMS key. See the AWS Finding the key ID and key ARN documentation for more information.

  8. Copy the IAM role policy provided. Using the IAM console in the AWS Management Console, embed the policy as an inline policy for the IAM role used for authentication.

    See the AWS Adding and removing IAM identity permissions documentation for guidance. Follow the instructions for embedding an inline policy for a user or role in the IAM console.

  9. Copy the bucket policy provided. Using the Amazon S3 console in the AWS Management Console, add the bucket policy to the S3 bucket being used for storage.

    See the AWS Adding a bucket policy by using the Amazon S3 console for guidance on editing bucket policies.

  10. Confirm the IAM role policy is embedded and the bucket policy is added, then click Next.

  11. Select whether to set the drive as a default drive. Not usually recommended, see Managing default drives for details.

  12. Click Done

Managing IAM roles

After adding an IAM role, it can be reused when adding other S3 buckets. Roles added when connecting a workspace drive are reusable within the same workspace only. Roles added when connecting an organization drive are reusable across all workspaces. While other organization or workspace members can select a listed IAM role, they must have access to your AWS Management Console to complete all steps in the connection process.

Reusing roles

Previously connected IAM roles display in a role selection dropdown when connecting S3 buckets. When reusing a role, you don’t need to add a new custom trust policy as the trust relationship is already established.

Changing roles

AI Hub doesn’t support changing the IAM role used for authentication. You can remove then reconnect the bucket with a new role.

Deleting roles

When you delete a role, it can no longer be used for authentication. You can’t delete an IAM role that’s in use with a connected drive. If you want to continue using the connection with a different role, remove then reconnect the bucket with a new role.

  1. In Workspaces, select a workspace, then select the Data tab.

  2. Click Add data source, then select Amazon S3.

  3. Select an audience.

  4. Enter a display name for the drive.

  5. Select IAM role as your authentication method, then click Next.

  6. Select the role to delete, then click the delete icon Icon of a trash can..

  7. Click Delete to confirm.

Updating a connection

Select configuration changes are supported.

  • Authenticated with AWS IAM access key — You can update the drive’s security credentials. You must remove and reconnect the drive to change the authentication method.

  • Authenticated with AWS IAM role — No changes supported. You must remove and reconnect the drive to change the authentication method or change the IAM role.

  1. In Workspaces, select All workspaces, then select the Data tab.

  2. Click the overflow icon Icon with three stacked vertical dots. of the drive to update, then select Modify configuration.

  3. Make any changes, then click Update to confirm.

Removing a connection

You can remove a connected drive to disconnect it and revoke AI Hub’s access to its contents.

Before you begin

Review the following limitations:

  • Removing a drive completely disconnects the drive from AI Hub. Any processed AI Hub files stored on the drive aren’t deleted, but AI Hub loses the ability to reference those files in the future. While you can later reconnect the drive, doing so doesn’t restore the ability to reference files previously saved to the drive. To reference such files, you must re-upload them.

  • Default drives can’t be removed. For guidance on changing default drives, see Managing default drives

  1. In Workspaces, select All workspaces, then select the Data tab.

  2. Click the overflow icon Icon with three stacked vertical dots. of the drive to remove, then select Remove.

  3. Type the confirmation text, then click Remove.