steampipe plugin install azure

Table: azure_data_factory_pipeline - Query Azure Data Factory Pipelines using SQL

Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data. It allows users to create, schedule, and manage data pipelines. These pipelines can ingest data from disparate data stores, transform the data by using compute services such as Azure HDInsight Hadoop, Azure Databricks, and Azure SQL Database.

Table Usage Guide

The azure_data_factory_pipeline table provides insights into the pipelines within Azure Data Factory. As a data engineer or data scientist, explore pipeline-specific details through this table, including pipeline configurations, statuses, and activities. This table can be utilized to manage and monitor data pipelines, ensuring optimal data flow and transformation.

Examples

Basic info

Determine the areas in which Azure Data Factory Pipelines are used in your system. This query is handy when you need to understand the distribution and usage of these pipelines across your infrastructure for better management and optimization.

select
name,
id,
factory_name,
type,
etag
from
azure_data_factory_pipeline;
select
name,
id,
factory_name,
type,
etag
from
azure_data_factory_pipeline;

Schema for azure_data_factory_pipeline

NameTypeOperatorsDescription
_ctxjsonbSteampipe context in JSON form.
activitiesjsonbA list of activities in pipeline.
akasjsonbArray of globally unique identifier strings (also known as) for the resource.
annotationsjsonbA list of tags that can be used for describing the Pipeline.
cloud_environmenttextThe Azure Cloud Environment.
concurrencybigintThe max number of concurrent runs for the pipeline.
descriptiontextThe description of the pipeline.
etagtextAn unique read-only string that changes whenever the resource is updated.
factory_nametext=Name of the factory the pipeline belongs.
idtextThe resource identifier.
nametext=The resource name.
parametersjsonbA list of parameters for pipeline.
pipeline_foldertextThe folder that this Pipeline is in. If not specified, Pipeline will appear at the root level.
pipeline_policyjsonbPipeline ElapsedTime Metric Policy.
resource_grouptext=The resource group which holds this resource.
run_dimensionsjsonbDimensions emitted by Pipeline.
sp_connection_nametext=, !=, ~~, ~~*, !~~, !~~*Steampipe connection name.
sp_ctxjsonbSteampipe context in JSON form.
subscription_idtext=, !=, ~~, ~~*, !~~, !~~*The Azure Subscription ID in which the resource is located.
titletextTitle of the resource.
typetextThe resource type.
variablesjsonbA list of variables for pipeline.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- azure

You can pass the configuration to the command with the --config argument:

steampipe_export_azure --config '<your_config>' azure_data_factory_pipeline