steampipe plugin install aws

Table: aws_msk_cluster - Query AWS MSK Clusters using SQL

The AWS Managed Streaming for Apache Kafka (MSK) is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data. AWS MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters. It also takes care of the maintenance and operations of the underlying infrastructure, so you can focus on building and running your applications.

Table Usage Guide

The aws_msk_cluster table in Steampipe provides you with information about Managed Streaming for Apache Kafka (MSK) clusters within AWS. This table allows you, as a DevOps engineer, to query cluster-specific details, including the cluster ARN, creation time, and associated metadata. You can utilize this table to gather insights on clusters, such as the number of broker nodes, the version of Apache Kafka, the state of the cluster, and more. The schema outlines the various attributes of the MSK cluster for you, including the broker node group info, encryption info, open monitoring status, and associated tags.

Examples

Basic Info

Explore the status and details of your AWS MSK clusters to understand their configuration and operational state. This can be useful for auditing purposes, or to identify potential issues with cluster setup or version compatibility.

select
arn,
cluster_name,
state,
cluster_type,
creation_time,
current_version,
region,
tags
from
aws_msk_cluster;
select
arn,
cluster_name,
state,
cluster_type,
creation_time,
current_version,
region,
tags
from
aws_msk_cluster;

List inactive clusters

Identify instances where certain clusters within AWS MSK service are not in an active state. This could be useful for system administrators who need to manage resources or troubleshoot issues.

select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
state <> 'ACTIVE';
select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
state != 'ACTIVE';

List clusters that allow public access

Determine the areas in which public access is allowed for clusters. This is useful for identifying potential security risks and ensuring that access to sensitive data is appropriately restricted.

select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
provisioned -> 'BrokerNodeGroupInfo' -> 'ConnectivityInfo' -> 'PublicAccess' ->> 'Type' <> 'DISABLED';
select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
json_extract(
provisioned,
'$.BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type'
) <> 'DISABLED';

List clusters with encryption at rest disabled

Determine the areas in which encryption at rest is disabled for clusters, allowing you to address potential security vulnerabilities by identifying clusters without this added layer of data protection.

select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
provisioned -> 'EncryptionInfo' -> 'EncryptionAtRest' is null;
select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
json_extract(provisioned, '$.EncryptionInfo.EncryptionAtRest') is null;

List clusters with encryption in transit disabled

Determine the areas in which encryption in transit is disabled for clusters. This can be useful for identifying potential security vulnerabilities and ensuring data safety.

select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
provisioned -> 'EncryptionInfo' -> 'EncryptionInTransit' is null;
select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
json_extract(
provisioned,
'$.EncryptionInfo.EncryptionInTransit'
) is null;

List clusters with logging disabled

Discover the segments that consist of clusters with logging disabled, allowing you to identify potential areas for enhancing security measures and ensuring compliance with logging policies.

select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
provisioned -> 'LoggingInfo' is null;
select
arn,
cluster_name,
state,
creation_time
from
aws_msk_cluster
where
json_extract(provisioned, '$.LoggingInfo') is null;

Get total storage used by all the clusters

Determine the total storage utilized by all clusters to manage resources efficiently and optimize cost. This can help in understanding the overall storage usage and planning for future scaling needs.

select
sum(
(
provisioned -> 'BrokerNodeGroupInfo' -> 'StorageInfo' -> 'EbsStorageInfo' ->> 'VolumeSize'
) :: int
) as total_storage
from
aws_msk_cluster;
select
sum(
json_extract(
provisioned,
'$.BrokerNodeGroupInfo.StorageInfo.EbsStorageInfo.VolumeSize'
)
) as total_storage
from
aws_msk_cluster;

Schema for aws_msk_cluster

NameTypeOperatorsDescription
_ctxjsonbSteampipe context in JSON form, e.g. connection_name.
account_idtextThe AWS Account ID in which the resource is located.
active_operation_arntextArn of active cluster operation.
akasjsonbArray of globally unique identifier strings (also known as) for the resource.
arntext=The Amazon Resource Name (ARN) that uniquely identifies the Cluster.
cluster_configurationjsonbDescription of this MSK configuration.
cluster_nametextThe name of the cluster.
cluster_operationjsonbDescription of this MSK operation.
cluster_typetextThe type of the cluster.
creation_timetimestamp with time zoneThe time when the cluster was created.
current_versiontextThe current version of the MSK cluster.
partitiontextThe AWS partition in which the resource is located (aws, aws-cn, or aws-us-gov).
provisionedjsonbInformation about the provisioned cluster.
regiontextThe AWS Region in which the resource is located.
statetextSettings for open monitoring using Prometheus.
state_infojsonbState Info for the Amazon MSK cluster.
tagsjsonbA list of tags attached to the Cluster.
titletextTitle of the resource.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- aws

You can pass the configuration to the command with the --config argument:

steampipe_export_aws --config '<your_config>' aws_msk_cluster