Twitter + Steampipe

Steampipe is an open-source zero-ETL engine to instantly query cloud APIs using SQL.

Twitter is an American microblogging and social networking service on which users post and interact with messages known as "tweets".

For example:

select
  id,
  text,
  mentions
from
  twitter_search_recent
where
  query = '#shiftleftjoin'

+---------------------+-----------------------------------------------------+-----------------+
| id                  | text                                                | mentions        |
+---------------------+-----------------------------------------------------+-----------------+
| 1378041446687768578 | New! Steampipe GCP plugin v0.5.0 🚀                 | ["googlecloud"] |
|                     |                                                     |                 |
|                     | Docs — https://t.co/s407cEdLAE                      |                 |
|                     |                                                     |                 |
|                     | Changelog — https://t.co/nFbLeh0KMu                 |                 |
|                     |                                                     |                 |
|                     | #shiftleftjoin @googlecloud https://t.co/0bGH747VRs |                 |
| 1377720348804853761 | New! Steampipe AWS plugin v0.10.0 🚀                | ["awscloud"]    |
|                     |                                                     |                 |
|                     | Docs - https://t.co/Y0vghMB1ub                      |                 |
|                     |                                                     |                 |
|                     | Changelog – https://t.co/8IETtyFBGZ                 |                 |
|                     |                                                     |                 |
|                     | #shiftleftjoin @awscloud https://t.co/o7ifZQlRMe    |                 |
+---------------------+-----------------------------------------------------+-----------------+

Documentation

Table definitions & examples →

Get started

Install

Download and install the latest Twitter plugin:

steampipe plugin install twitter

Credentials

Item	Description
Credentials	Apply for a free developer account. Create a project. Download the bearer token.
Permissions	`Read Only` is required for the app.
Radius	Each connection represents a single set of Twitter credentials.
Resolution	1. `bearer_token` in Steampipe config. 2. `consumer_key`, `consumer_secret`, `access_token`, `access_secret` in Steampipe config. 3. `TWITTER_BEARER_TOKEN` environment variable. 4. `TWITTER_CONSUMER_KEY`, `TWITTER_CONSUMER_SECRET`, `TWITTER_ACCESS_TOKEN` and `TWITTER_ACCESS_SECRET` environment variables.

Configuration

Installing the latest twitter plugin will create a config file (~/.steampipe/config/twitter.spc) with a single connection named twitter:

connection "twitter" {
  plugin       = "twitter"
  bearer_token = "AAAAAAAAAAAAAAAAAAAAAFL8NgEAAAAA2%2FyHFNeRK0CUoZ5ybpsHgnL91n0%3DEJPs4GsJVU8ZlrHYr1x0eyb4Br48WeqLtmM4aAKbIxpInTrrIu"
}

Postgres FDW

This plugin is available as a native Postgres FDW. Unlike Steampipe CLI, which ships with an embedded Postgres server instance, the Postgres FDW can be installed in any supported Postgres database version.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_postgres_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/postgres.sh)" -- twitter

The installer will prompt you for the plugin name and version, download and install the appropriate files for your OS, system architecture, and Postgres version.

To configure the Postgres FDW, you will create an extension, foreign server, and schema and import the foreign schema.

CREATE EXTENSION IF NOT EXISTS steampipe_postgres_twitter;
CREATE SERVER steampipe_twitter FOREIGN DATA WRAPPER steampipe_postgres_twitter OPTIONS (config '<your_config>');
CREATE SCHEMA twitter;
IMPORT FOREIGN SCHEMA twitter FROM SERVER steampipe_twitter INTO twitter;

SQLite Extension

This plugin is available as a SQLite Extension, making the tables available as SQLite virtual tables.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_sqlite_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/sqlite.sh)" -- twitter

The installer will prompt you for the plugin name, version, and destination directory. It will then determine the OS and system architecture, and it will download and install the appropriate package.

To configure the SQLite extension, load the extension module and then run the steampipe_configure_twitter function to configure it with plugin-specific options.

$ sqlite3
sqlite> .load ./steampipe_sqlite_extension_twitter.so
sqlite> select steampipe_configure_twitter('<your_config>');

Export

This plugin is available as a standalone Export CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- twitter

You can pass the configuration to the command with the --config argument:

steampipe_export_twitter --config '<your_config>' <table_name>