Table: gcp_composer_environment - Query GCP Composer Environments using SQL
Google Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. The gcp_composer_environment
table in Steampipe allows you to query information about Cloud Composer environments in your GCP environment, including details about Airflow configurations, node settings, security configurations, and more.
Table Usage Guide
The gcp_composer_environment
table is useful for cloud administrators, DevOps engineers, and data engineers who need to gather detailed insights into their Cloud Composer environments. You can query various aspects of the environment, such as its Airflow URI, node configurations, software settings, and network access controls. This table is particularly helpful for monitoring environment states, managing Airflow settings, and ensuring that your workflows are running in a secure and optimized environment.
Examples
Basic info
Retrieve basic information about Composer environments, including their name, location, and state.
select name, location, state, airflow_uri, create_timefrom gcp_composer_environment;
select name, location, state, airflow_uri, create_timefrom gcp_composer_environment;
List environments by size
Identify Composer environments based on their size, such as "ENVIRONMENT_SIZE_SMALL" or "ENVIRONMENT_SIZE_LARGE."
select name, environment_size, location, projectfrom gcp_composer_environmentwhere environment_size = 'ENVIRONMENT_SIZE_SMALL';
select name, environment_size, location, projectfrom gcp_composer_environmentwhere environment_size = 'ENVIRONMENT_SIZE_SMALL';
List environments with specific node configurations
Retrieve environments with specific node configurations, such as node count and machine types.
select name, node_count, node_config ->> 'machineType' as machine_type, location, projectfrom gcp_composer_environmentwhere node_count > 0;
select name, node_count, json_extract(node_config, '$.machineType') as machine_type, location, projectfrom gcp_composer_environmentwhere node_count > 0;
List environments with specific network access controls
Identify environments that have specific network-level access controls for the Airflow web server.
select name, jsonb_path_query_array( web_server_network_access_control, '$.allowedIpRanges[*].value' ) as allowed_ip_ranges, location, projectfrom gcp_composer_environmentwhere web_server_network_access_control is not null;
select name, json_extract( web_server_network_access_control, '$.allowedIpRanges[0].value' ) as allowed_ip_ranges, location, projectfrom gcp_composer_environmentwhere web_server_network_access_control is not null;
List environments with scheduled snapshot configurations
Fetch environments that have scheduled snapshot configurations enabled for recovery.
select name, recovery_config ->> 'scheduledSnapshotsConfig' as scheduled_snapshots_config, location, projectfrom gcp_composer_environmentwhere recovery_config ->> 'scheduledSnapshotsConfig' is not null;
select name, json_extract(recovery_config, '$.scheduledSnapshotsConfig') as scheduled_snapshots_config, location, projectfrom gcp_composer_environmentwhere json_extract(recovery_config, '$.scheduledSnapshotsConfig') is not null;
Schema for gcp_composer_environment
Name | Type | Operators | Description |
---|---|---|---|
_ctx | jsonb | Steampipe context in JSON form. | |
airflow_byoid_uri | text | The 'bring your own identity' variant of the URI of the Apache Airflow Web UI hosted within this environment. | |
airflow_uri | text | The URI of the Apache Airflow Web UI hosted within this environment. | |
akas | jsonb | Array of globally unique identifier strings (also known as) for the resource. | |
create_time | timestamp with time zone | The time at which this environment was created. | |
dag_gcs_prefix | text | The Cloud Storage prefix of the DAGs for this environment. | |
data_retention_config | jsonb | The configuration setting for Airflow database data retention mechanism. | |
database_config | jsonb | The configuration settings for Cloud SQL instance used internally by Apache Airflow software. | |
encryption_config | jsonb | The encryption options for the Cloud Composer environment and its dependencies. Cannot be updated. | |
environment_size | text | The size of the Cloud Composer environment. | |
gke_cluster | text | The Kubernetes Engine cluster used to run this environment. | |
location | text | The GCP multi-region, region, or zone in which the resource is located. | |
maintenance_window | jsonb | The maintenance window is the period when Cloud Composer components may undergo maintenance. | |
master_authorized_networks_config | jsonb | The configuration options for GKE cluster master authorized networks. | |
name | text | = | The resource name of the environment. |
node_config | jsonb | The configuration used for the Kubernetes Engine cluster. | |
node_count | bigint | The number of nodes in the Kubernetes Engine cluster that will be used to run this environment. | |
private_environment_config | jsonb | The configuration used for the Private IP Cloud Composer environment. | |
project | text | =, !=, ~~, ~~*, !~~, !~~* | The GCP Project in which the resource is located. |
recovery_config | jsonb | The Recovery settings configuration of an environment. | |
resilience_mode | text | Resilience mode of the cloud composer environment. | |
software_config | jsonb | The configuration settings for software inside the environment. | |
sp_connection_name | text | =, !=, ~~, ~~*, !~~, !~~* | Steampipe connection name. |
sp_ctx | jsonb | Steampipe context in JSON form. | |
state | text | The current state of the environment. | |
storage_config_bucket | text | Storage configuration for this environment. | |
tags | jsonb | A map of tags for the resource. | |
title | text | Title of the resource. | |
update_time | timestamp with time zone | The time at which this environment was last modified. | |
uuid | text | The UUID (Universally Unique IDentifier) associated with this environment. | |
web_server_config | jsonb | The configuration settings for the Airflow web server App Engine instance. | |
web_server_network_access_control | jsonb | The network-level access control policy for the Airflow web server. | |
workloads_config | jsonb | The workloads configuration settings for the GKE cluster associated with the Cloud Composer environment. |
Export
This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.
You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh
script:
/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- gcp
You can pass the configuration to the command with the --config
argument:
steampipe_export_gcp --config '<your_config>' gcp_composer_environment