AWS Glue Connector

AWS Glue is a serverless data integration service from Amazon Web Services. The AWS Glue component allows listing, starting, and stopping jobs, triggers, and crawlers in an AWS Glue account.

API Documentation

This component was built using the AWS Glue Developer Guide.

Connections

Access Key and Secret

Authenticate requests to AWS Glue using an Access Key and Secret.

An AWS IAM access key pair is required to interact with AWS Glue. Ensure the key pair generated in AWS has proper permissions to the AWS Glue resources to access. Read more about Glue IAM actions in the AWS documentation.

Prerequisites

• An AWS account with IAM access
• Appropriate permissions to create IAM access keys

Setup Steps

To create an IAM access key pair:

1. Sign in to the AWS Console and navigate to Identity and Access Management (IAM)
2. Select the IAM user that will be used for the integration
3. Navigate to the Security credentials tab
4. Under the Access keys section, select Create access key
5. Choose the appropriate use case (e.g., Third-party service or Application running outside AWS)
6. Copy both the Access key ID and Secret access key when displayed

Secret Key Visibility

The Secret access key is only shown once during creation. If it is not copied at this time, a new access key pair must be created.

Configure the Connection

• Enter the Access Key ID into the connection configuration
• Enter the Secret Access Key into the connection configuration

Input	Comments	Default
Access Key ID	The AWS IAM Access Key ID used to authenticate requests.
Secret Access Key	The AWS IAM Secret Access Key used to authenticate requests.

AWS Role ARN

Connect to AWS using an assumed role

AWS Assume Role authentication allows assuming an IAM role using temporary security credentials. This method is useful for cross account access or when implementing principle of least privilege.

Refer to the AWS documentation on assuming roles for detailed information.

Prerequisites

• Completed Access Key and Secret connection setup (access key pair required)
• Appropriate permissions to create IAM roles

Setup Steps

An IAM user with access keys is required to assume a role. If access keys have not been created, follow the Access Key and Secret setup steps first, then return here to create the IAM role.

Create an IAM Role with Trust Policy

1. From the IAM Console, navigate to Roles and select Create Role
2. Select Custom trust policy as the trusted entity type
3. Enter the trust policy below, replacing USER_ARN with the IAM user ARN:

View trust policy template

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "USER_ARN"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

4. Click Next and attach the appropriate Glue permissions policy (e.g., AWSGlueConsoleFullAccess)
5. Complete the remaining steps and select Create Role
6. Copy the Role ARN from the role summary (format: arn:aws:iam::123456789012:role/role-name)

Configure the Connection

• Role ARN: The ARN of the IAM role to assume
• Access Key ID: From the IAM user (see Access Key and Secret)
• Secret Access Key: From the IAM user
• External ID (optional): Shared secret for enhanced security

External ID

The External ID provides additional security for cross-account access. Refer to the AWS documentation for more information.

Verify Connection

The IAM user credentials assume the role, which provides temporary credentials with the role's attached permissions. Ensure the trust policy correctly references the IAM user ARN.

Input	Comments	Default
Role ARN	An AWS IAM Role ARN
Access Key ID	An AWS IAM Access Key ID
Secret Access Key	An AWS IAM Secret Access Key
External ID	Provides enhanced security measures to the connection. Optional, but recommended. Please check AWS docs for more information.

Actions

Get Job Run

Retrieves the metadata for a given job run.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Run ID	The unique identifier for the job run.
Connection	The AWS Glue connection to use.

List Crawlers

Lists crawlers available in AWS Glue.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Max Items	The maximum number of results to return. Provide a value from 1 to 50.
Marker	The pagination token returned by a previous request, used to retrieve the next page of results.
Connection	The AWS Glue connection to use.

List Jobs

Lists jobs available in AWS Glue.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Marker	The pagination token returned by a previous request, used to retrieve the next page of results.
Max Items	The maximum number of results to return. Provide a value from 1 to 50.
Connection	The AWS Glue connection to use.

List Triggers

Lists the names of all triggers in the account.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Max Items	The maximum number of results to return. Provide a value from 1 to 50.
Marker	The pagination token returned by a previous request, used to retrieve the next page of results.
Connection	The AWS Glue connection to use.

Start Crawler

Starts an existing crawler in AWS Glue.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Connection	The AWS Glue connection to use.

Start Job Run

Starts a job run using an AWS Glue job definition.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Allocated Capacity	The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. If this is omitted, Glue will use the default number of DPUs configured for the job.
Security Configuration	The name of the SecurityConfiguration structure to be used with this job. This can be left blank if there is no security configuration.
Arguments	Optional key value parameters to pass into a job.
Connection	The AWS Glue connection to use.

Start Trigger

Starts an existing trigger in AWS Glue.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Connection	The AWS Glue connection to use.

Stop Crawler

Stops the specified crawler if it is currently running.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Connection	The AWS Glue connection to use.

Stop Job Run

Stops one or more job runs for a specified job definition.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Job Run IDs	The job run identifiers to stop for the specified job.
Connection	The AWS Glue connection to use.

Stop Trigger

Stops a specified trigger.

Input	Comments	Default
AWS Region	AWS provides services in multiple regions, like us-west-2 or eu-west-1.
Name	The name of the resource (job, crawler, or trigger) to act on. Provide the name, not the ARN.
Connection	The AWS Glue connection to use.