Table: azure_data_factory_pipeline - Query Azure Data Factory Pipelines using SQL

Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data. It allows users to create, schedule, and manage data pipelines. These pipelines can ingest data from disparate data stores, transform the data by using compute services such as Azure HDInsight Hadoop, Azure Databricks, and Azure SQL Database.

Table Usage Guide

The azure_data_factory_pipeline table provides insights into the pipelines within Azure Data Factory. As a data engineer or data scientist, explore pipeline-specific details through this table, including pipeline configurations, statuses, and activities. This table can be utilized to manage and monitor data pipelines, ensuring optimal data flow and transformation.

Examples

Basic info

Determine the areas in which Azure Data Factory Pipelines are used in your system. This query is handy when you need to understand the distribution and usage of these pipelines across your infrastructure for better management and optimization.

select
  name,
  id,
  factory_name,
  type,
  etag
from
  azure_data_factory_pipeline;

select
  name,
  id,
  factory_name,
  type,
  etag
from
  azure_data_factory_pipeline;

Schema for azure_data_factory_pipeline

Name	Type	Operators	Description
_ctx	jsonb		Steampipe context in JSON form.
activities	jsonb		A list of activities in pipeline.
akas	jsonb		Array of globally unique identifier strings (also known as) for the resource.
annotations	jsonb		A list of tags that can be used for describing the Pipeline.
cloud_environment	text		The Azure Cloud Environment.
concurrency	bigint		The max number of concurrent runs for the pipeline.
description	text		The description of the pipeline.
etag	text		An unique read-only string that changes whenever the resource is updated.
factory_name	text	=	Name of the factory the pipeline belongs.
id	text		The resource identifier.
name	text	=	The resource name.
parameters	jsonb		A list of parameters for pipeline.
pipeline_folder	text		The folder that this Pipeline is in. If not specified, Pipeline will appear at the root level.
pipeline_policy	jsonb		Pipeline ElapsedTime Metric Policy.
resource_group	text	=	The resource group which holds this resource.
run_dimensions	jsonb		Dimensions emitted by Pipeline.
sp_connection_name	text	=, !=, ~~, ~~, !~~, !~~	Steampipe connection name.
sp_ctx	jsonb		Steampipe context in JSON form.
subscription_id	text	=, !=, ~~, ~~, !~~, !~~	The Azure Subscription ID in which the resource is located.
title	text		Title of the resource.
type	text		The resource type.
variables	jsonb		A list of variables for pipeline.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- azure

You can pass the configuration to the command with the --config argument:

steampipe_export_azure --config '<your_config>' azure_data_factory_pipeline