steampipe plugin install awssteampipe plugin install aws

Table: aws_glue_job

An AWS Glue job encapsulates a script that connects to your source data, processes it, and then writes it out to your data target. Typically, a job runs extract, transform, and load (ETL) scripts. Jobs can also run general-purpose Python scripts (Python shell jobs.) AWS Glue triggers can start jobs based on a schedule or event, or on demand. You can monitor job runs to understand runtime metrics such as completion status, duration, and start time.

Examples

Basic info

select
  name,
  created_on,
  description,
  max_capacity,
  number_of_workers,
  region,
  timeout
from
  aws_glue_job;

List jobs with glue connections attached

select
  title,
  arn,
  created_on,
  connections -> 'Connections' as connections
from
  aws_glue_job
where
  connections is not null;

List job details with bookmark enabled

select
  title,
  arn,
  created_on,
  job_bookmark ->> 'Attempt' as total_attempts,
  job_bookmark ->> 'Run' as total_runs,
  job_bookmark ->> 'RunId' as run_id
from
  aws_glue_job
where
  job_bookmark is not null;

List jobs with cloud watch encryption disabled

select
  j.title,
  j.arn,
  j.created_on,
  j.region,
  j.account_id,
  cloud_watch_encryption
from
  aws_glue_job j
  left join aws_glue_security_configuration s on j.security_configuration = s.name
where
  cloud_watch_encryption is null
  or cloud_watch_encryption ->> 'CloudWatchEncryptionMode' = 'DISABLED';

List jobs with job bookmarks encryption disabled

select
  j.title,
  j.arn,
  j.created_on,
  j.region,
  j.account_id,
  job_bookmarks_encryption
from
  aws_glue_job j
  left join aws_glue_security_configuration s on j.security_configuration = s.name
where
  job_bookmarks_encryption is null
  or job_bookmarks_encryption ->> 'JobBookmarksEncryptionMode' = 'DISABLED';

List jobs with s3 encryption disabled

select
  j.title,
  j.arn,
  j.created_on,
  j.region,
  j.account_id,
  e as s3_encryption
from
  aws_glue_job j
  left join aws_glue_security_configuration s on j.security_configuration = s.name,
  jsonb_array_elements(s.s3_encryption) e
where
  e is null
  or e ->> 'S3EncryptionMode' = 'DISABLED';

List jobs with logging disabled

select
  title,
  arn,
  created_on region,
  account_id
from
  aws_glue_job
where
  default_arguments ->> '--enable-continuous-cloudwatch-log' = 'false';

List jobs with monitoring disabled

select
  title,
  arn,
  created_on region,
  account_id
from
  aws_glue_job
where
  default_arguments ->> '--enable-metrics' = 'false';

List script details associated to the job

select
  title,
  arn,
  created_on,
  command ->> 'Name' as script_name,
  command ->> 'ScriptLocation' as script_location,
  default_arguments ->> '--job-language' as job_language
from
  aws_glue_job;

List jobs with server side encryption disabled

select
  title,
  arn,
  created_on region,
  account_id,
  default_arguments ->> '--encryption-type' as encryption_type
from
  aws_glue_job
where
  default_arguments ->> '--encryption-type' is null;

.inspect aws_glue_job

AWS Glue Job

Name	Type	Description
_ctx	jsonb	Steampipe context in JSON form, e.g. connection_name.
account_id	text	The AWS Account ID in which the resource is located.
akas	jsonb	Array of globally unique identifier strings (also known as) for the resource.
allocated_capacity	double precision	[DEPRECATED] This column has been deprecated and will be removed in a future release, use max_capacity instead. The number of Glue data processing units (DPUs) that can be allocated when this job runs.
arn	text	The Amazon Resource Name (ARN) of the GlueJob.
command	jsonb	The JobCommand that runs this job.
connections	jsonb	The connections used for this job.
created_on	timestamp with time zone	The time and date that this job definition was created.
default_arguments	jsonb	The default arguments for this job, specified as name-value pairs.
description	text	A description of the job.
execution_property	jsonb	An ExecutionProperty specifying the maximum number of concurrent runs allowed for this job.
glue_version	text	Glue version determines the versions of Apache Spark and Python that Glue supports.
job_bookmark	jsonb	Defines a point that a job can resume processing.
last_modified_on	timestamp with time zone	The last point in time when this job definition was modified.
log_uri	text	This field is reserved for future use.
max_capacity	double precision	The number of Glue data processing units (DPUs) that can be allocated when this job runs.
max_retries	bigint	The maximum number of times to retry this job after a JobRun fails.
name	text	The name of the GlueJob.
non_overridable_arguments	jsonb	Non-overridable arguments for this job, specified as name-value pairs.
notification_property	jsonb	Specifies configuration properties of a job notification.
number_of_workers	bigint	The number of workers of a defined workerType that are allocated when a job runs.
partition	text	The AWS partition in which the resource is located (aws, aws-cn, or aws-us-gov).
region	text	The AWS Region in which the resource is located.
role	text	The name or Amazon Resource Name (ARN) of the IAM role associated with this job.
security_configuration	text	The name of the SecurityConfiguration structure to be used with this job.
timeout	bigint	The job timeout in minutes.
title	text	Title of the resource.
worker_type	text	The type of predefined worker that is allocated when a job runs. Accepts a value of Standard, G.1X, or G.2X.

turbot/aws