Turbot Pipes + Steampipe

Turbot Pipes is an intelligence, automation & security platform built specifically for DevOps.

Steampipe is an open-source zero-ETL engine to instantly query cloud APIs using SQL.

For example:

select
  user_handle,
  email,
  status
from
  pipes_organization_member
where
  status = 'accepted';

> select user_handle, email, status from pipes_organization_member where status = 'accepted';
+-------------+------------------+----------+
| user_handle | email            | status   |
+-------------+------------------+----------+
| mario       | mario@turbot.com | accepted |
| yoshi       | yoshi@turbot.com | accepted |
+-------------+------------------+----------+

Documentation

Table definitions & examples →

Get started

Install

Download and install the latest Pipes plugin:

steampipe plugin install pipes

Configuration

Installing the latest Pipes plugin will create a config file (~/.steampipe/config/pipes.spc) with a single connection named pipes:

connection "pipes" {
  plugin = "pipes"

  # Turbot Pipes API token. If `token` is not specified, it will be loaded
  # from the `STEAMPIPE_CLOUD_TOKEN` environment variable and if not found
  # there will fallback to the `PIPES_TOKEN` environment variable. If both
  # are set simultaneously, `STEAMPIPE_CLOUD_TOKEN` will take preference.
  # token = "tpt_thisisnotarealtoken_123"

  # Turbot Pipes host URL. This defaults to "https://pipes.turbot.com".
  # You only need to set this if connecting to a remote Turbot Pipes database
  # not hosted in "https://pipes.turbot.com".
  # If `host` is not specified, it will be loaded from the `STEAMPIPE_CLOUD_HOST`
  # environment variable and if not found there will fallback to the
  # `PIPES_HOST` environment variable. If both are set simultaneously,
  # `STEAMPIPE_CLOUD_HOST` will take preference.
  # host = "https://pipes.turbot.com"
}

token (required) - API tokens can be used to access the Turbot Pipes API or to connect to Turbot Pipes workspaces from the Steampipe CLI. May alternatively be set via the STEAMPIPE_CLOUD_TOKEN or PIPES_TOKEN. Note that the value in STEAMPIPE_CLOUD_TOKEN will take preference if both are set.
host (optional) The Turbot Pipes Host URL. This defaults to https://pipes.turbot.com. You only need to set this if you are connecting to a remote Turbot Pipes database that is NOT hosted in https://pipes.turbot.com. This can also be set via the STEAMPIPE_CLOUD_HOST or PIPES_HOST. Note that the value in STEAMPIPE_CLOUD_HOST will take preference if both are set.

Get Involved

Open source: https://github.com/turbot/steampipe-plugin-pipes
Community: Join #steampipe on Slack →

Postgres FDW

This plugin is available as a native Postgres FDW. Unlike Steampipe CLI, which ships with an embedded Postgres server instance, the Postgres FDW can be installed in any supported Postgres database version.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_postgres_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/postgres.sh)" -- pipes

The installer will prompt you for the plugin name and version, download and install the appropriate files for your OS, system architecture, and Postgres version.

To configure the Postgres FDW, you will create an extension, foreign server, and schema and import the foreign schema.

CREATE EXTENSION IF NOT EXISTS steampipe_postgres_pipes;
CREATE SERVER steampipe_pipes FOREIGN DATA WRAPPER steampipe_postgres_pipes OPTIONS (config '<your_config>');
CREATE SCHEMA pipes;
IMPORT FOREIGN SCHEMA pipes FROM SERVER steampipe_pipes INTO pipes;

SQLite Extension

This plugin is available as a SQLite Extension, making the tables available as SQLite virtual tables.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_sqlite_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/sqlite.sh)" -- pipes

The installer will prompt you for the plugin name, version, and destination directory. It will then determine the OS and system architecture, and it will download and install the appropriate package.

To configure the SQLite extension, load the extension module and then run the steampipe_configure_pipes function to configure it with plugin-specific options.

$ sqlite3
sqlite> .load ./steampipe_sqlite_extension_pipes.so
sqlite> select steampipe_configure_pipes('<your_config>');

Export

This plugin is available as a standalone Export CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- pipes

You can pass the configuration to the command with the --config argument:

steampipe_export_pipes --config '<your_config>' <table_name>