steampipe plugin install gcp

Table: gcp_composer_environment - Query GCP Composer Environments using SQL

Google Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. The gcp_composer_environment table in Steampipe allows you to query information about Cloud Composer environments in your GCP environment, including details about Airflow configurations, node settings, security configurations, and more.

Table Usage Guide

The gcp_composer_environment table is useful for cloud administrators, DevOps engineers, and data engineers who need to gather detailed insights into their Cloud Composer environments. You can query various aspects of the environment, such as its Airflow URI, node configurations, software settings, and network access controls. This table is particularly helpful for monitoring environment states, managing Airflow settings, and ensuring that your workflows are running in a secure and optimized environment.

Examples

Basic info

Retrieve basic information about Composer environments, including their name, location, and state.

select
name,
location,
state,
airflow_uri,
create_time
from
gcp_composer_environment;
select
name,
location,
state,
airflow_uri,
create_time
from
gcp_composer_environment;

List environments by size

Identify Composer environments based on their size, such as "ENVIRONMENT_SIZE_SMALL" or "ENVIRONMENT_SIZE_LARGE."

select
name,
environment_size,
location,
project
from
gcp_composer_environment
where
environment_size = 'ENVIRONMENT_SIZE_SMALL';
select
name,
environment_size,
location,
project
from
gcp_composer_environment
where
environment_size = 'ENVIRONMENT_SIZE_SMALL';

List environments with specific node configurations

Retrieve environments with specific node configurations, such as node count and machine types.

select
name,
node_count,
node_config ->> 'machineType' as machine_type,
location,
project
from
gcp_composer_environment
where
node_count > 0;
select
name,
node_count,
json_extract(node_config, '$.machineType') as machine_type,
location,
project
from
gcp_composer_environment
where
node_count > 0;

List environments with specific network access controls

Identify environments that have specific network-level access controls for the Airflow web server.

select
name,
jsonb_path_query_array(
web_server_network_access_control,
'$.allowedIpRanges[*].value'
) as allowed_ip_ranges,
location,
project
from
gcp_composer_environment
where
web_server_network_access_control is not null;
select
name,
json_extract(
web_server_network_access_control,
'$.allowedIpRanges[0].value'
) as allowed_ip_ranges,
location,
project
from
gcp_composer_environment
where
web_server_network_access_control is not null;

List environments with scheduled snapshot configurations

Fetch environments that have scheduled snapshot configurations enabled for recovery.

select
name,
recovery_config ->> 'scheduledSnapshotsConfig' as scheduled_snapshots_config,
location,
project
from
gcp_composer_environment
where
recovery_config ->> 'scheduledSnapshotsConfig' is not null;
select
name,
json_extract(recovery_config, '$.scheduledSnapshotsConfig') as scheduled_snapshots_config,
location,
project
from
gcp_composer_environment
where
json_extract(recovery_config, '$.scheduledSnapshotsConfig') is not null;

Schema for gcp_composer_environment

NameTypeOperatorsDescription
_ctxjsonbSteampipe context in JSON form.
airflow_byoid_uritextThe 'bring your own identity' variant of the URI of the Apache Airflow Web UI hosted within this environment.
airflow_uritextThe URI of the Apache Airflow Web UI hosted within this environment.
akasjsonbArray of globally unique identifier strings (also known as) for the resource.
create_timetimestamp with time zoneThe time at which this environment was created.
dag_gcs_prefixtextThe Cloud Storage prefix of the DAGs for this environment.
data_retention_configjsonbThe configuration setting for Airflow database data retention mechanism.
database_configjsonbThe configuration settings for Cloud SQL instance used internally by Apache Airflow software.
encryption_configjsonbThe encryption options for the Cloud Composer environment and its dependencies. Cannot be updated.
environment_sizetextThe size of the Cloud Composer environment.
gke_clustertextThe Kubernetes Engine cluster used to run this environment.
locationtextThe GCP multi-region, region, or zone in which the resource is located.
maintenance_windowjsonbThe maintenance window is the period when Cloud Composer components may undergo maintenance.
master_authorized_networks_configjsonbThe configuration options for GKE cluster master authorized networks.
nametext=The resource name of the environment.
node_configjsonbThe configuration used for the Kubernetes Engine cluster.
node_countbigintThe number of nodes in the Kubernetes Engine cluster that will be used to run this environment.
private_environment_configjsonbThe configuration used for the Private IP Cloud Composer environment.
projecttext=, !=, ~~, ~~*, !~~, !~~*The GCP Project in which the resource is located.
recovery_configjsonbThe Recovery settings configuration of an environment.
resilience_modetextResilience mode of the cloud composer environment.
software_configjsonbThe configuration settings for software inside the environment.
sp_connection_nametext=, !=, ~~, ~~*, !~~, !~~*Steampipe connection name.
sp_ctxjsonbSteampipe context in JSON form.
statetextThe current state of the environment.
storage_config_buckettextStorage configuration for this environment.
tagsjsonbA map of tags for the resource.
titletextTitle of the resource.
update_timetimestamp with time zoneThe time at which this environment was last modified.
uuidtextThe UUID (Universally Unique IDentifier) associated with this environment.
web_server_configjsonbThe configuration settings for the Airflow web server App Engine instance.
web_server_network_access_controljsonbThe network-level access control policy for the Airflow web server.
workloads_configjsonbThe workloads configuration settings for the GKE cluster associated with the Cloud Composer environment.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- gcp

You can pass the configuration to the command with the --config argument:

steampipe_export_gcp --config '<your_config>' gcp_composer_environment