steampipe plugin install azuresteampipe plugin install azure

Table: azure_data_factory_dataset - Query Azure Data Factory Datasets using SQL

Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows for orchestrating and automating data movement and data transformation. It enables the creation of various types of inputs and outputs such as files, tables, and SQL query results. Azure Data Factory allows you to integrate the data silos and drive transformational insights.

Table Usage Guide

The azure_data_factory_dataset table provides insights into datasets within Azure Data Factory. As a Data Analyst, explore dataset-specific details through this table, including the structure, schema, and associated metadata. Utilize it to uncover information about datasets, such as data processing and transformation details, the relationships between datasets, and the verification of data schemas.

Examples

Basic info

This query is useful for gaining insights into various datasets in your Azure Data Factory. It allows you to view basic information such as name, ID, and type, which can be helpful for managing your data resources effectively.

select
  name,
  id,
  etag,
  type
from
  azure_data_factory_dataset;

select
  name,
  id,
  etag,
  type
from
  azure_data_factory_dataset;

List relational table type datasets

Explore which datasets in your Azure Data Factory are of the 'RelationalTable' type. This can be beneficial in understanding your data structure and management, especially when working with relational databases.

select
  name,
  id,
  type,
  properties ->> 'type' as dataset_type
from
  azure_data_factory_dataset
where
  properties ->> 'type' = 'RelationalTable';

select
  name,
  id,
  type,
  json_extract(properties, '$.type') as dataset_type
from
  azure_data_factory_dataset
where
  json_extract(properties, '$.type') = 'RelationalTable';

Schema for azure_data_factory_dataset

Name	Type	Operators	Description
_ctx	jsonb		Steampipe context in JSON form, e.g. connection_name.
akas	jsonb		Array of globally unique identifier strings (also known as) for the resource.
cloud_environment	text		The Azure Cloud Environment.
etag	text		An unique read-only string that changes whenever the resource is updated.
factory_name	text	=	Name of the factory the dataset belongs.
id	text		The resource identifier.
name	text	=	The resource name.
properties	jsonb		Dataset properties.
resource_group	text	=	The resource group which holds this resource.
subscription_id	text		The Azure Subscription ID in which the resource is located.
title	text		Title of the resource.
type	text		The resource type.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- azure

You can pass the configuration to the command with the --config argument:

steampipe_export_azure --config '<your_config>' azure_data_factory_dataset