GitHub + Steampipe

Steampipe is an open-source zero-ETL engine to instantly query cloud APIs using SQL.

GitHub is a provider of Internet hosting for software development and version control using Git. It offers the distributed version control and source code management (SCM) functionality of Git, plus its own features.

For example:

select
  name,
  primary_language -> 'name' as language,
  fork_count,
  stargazer_count
from
  github_my_repository;

+-------------------------------+------------+-------------+------------------+
| name                          | language   | forks_count | stargazer_count |
+-------------------------------+------------+-------------+------------------+
| steampipe                     | Go         | 11          | 254              |
| steampipe-plugin-aws          | Go         | 8           | 18               |
| steampipe-plugin-shodan       | Go         | 0           | 9                |
| steampipe-plugin-gcp          | Go         | 0           | 8                |
| steampipe-postgres-fdw        | C          | 0           | 8                |
| steampipe-plugin-azure        | Go         | 1           | 8                |
| steampipe-plugin-sdk          | Go         | 0           | 6                |
+-------------------------------+------------+-------------+------------------+

Documentation

Table definitions & examples →

Get started

Install

Download and install the latest GitHub plugin:

steampipe plugin install github

Credentials

Item	Description
Credentials	The GitHub plugin uses a personal access token to authenticate to the GitHub APIs.
Permissions	You must create a personal access token and assign the following scopes: - `repo` (all) - `read:org` - `gist` - `read:user` - `user:email`
Radius	The GitHub plugin query scope is generally the same as the GitHub API. Usually, this means you can list private resources that you have access to, as well as public resources that you own, or that are owned by organizations to which you belong. The same GitHub APIs are used to get information for public resources, but the public items are returned in list calls (because there would be too many). This has an interesting side effect in Steampipe in that you can sometimes query a specific item by a specific key column or columns that does not show up in a list query. For example, `select * from github_my_organization` will list details about all the GitHub Organizations to which you belong. `select * from github_organization where login = 'postgres'` will show you the publicly available details about the `postgres` organization, which didn't show up in your first query! It works this way in Steampipe because that's how it works in the API. While this may seem counter-intuitive at first, it actually can be quite useful.
Resolution	1. Credentials in the Steampipe configuration file (`~/.steampipe/config/github.spc`) 2. Credentials specified in environment variables, e.g., `GITHUB_TOKEN`.

Configuration

Installing the latest github plugin will create a config file (~/.steampipe/config/github.spc) with a single connection named github:

connection "github" {
  plugin = "github"

  # The following tokens are currently supported:
  # - Personal access tokens, e.g., `ghp_3b99b12218f63bcd702ad90d345975ef6c62f7d8`
  #   https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token for more information.
  # - GitHub application installation access token, e.g., `ghs_UdmjfiKzVbFJNBsaiePwFPCmKeFakeToken`
  #   https://docs.github.com/en/rest/apps/apps?apiVersion=2022-11-28#create-an-installation-access-token-for-an-app for more information.
  # Can also be set with the GITHUB_TOKEN environment variable.
  # token = "ghp_3b99b12218f63bcd702ad90d345975ef6c62f7d8"

  # GitHub Enterprise requires a base_url to be configured to your installation location.
  # Can also be set with the GITHUB_BASE_URL environment variable.
  # base_url = "https://github.example.com"

  # GitHub App authentication using a private key to create a JSON Web Token (JWT), which is then exchanged for an installation access token.

  # The unique identifier for the GitHub App.
  # Can also be set with the GITHUB_APP_ID environment variable.
  # app_id = "12345678"

  # The installation ID for a specific installation of the GitHub App.
  # Can also be set with the GITHUB_APP_INSTALLATION_ID environment variable.
  # app_installation_id = "8901234"

  # The path to a private key PEM file of the GitHub App, used for generating JWTs for authentication.
  # Can also be set with the GITHUB_APP_PEM_FILE environment variable.
  # app_private_key = "/Users/myuser/app_private_key.pem"
}

token - Personal access token for your GitHub account or a Github App installation access token. This can also be set via the GITHUB_TOKEN environment variable.
base_url - GitHub Enterprise users have a custom URL location (e.g. https://github.example.com). Not required for GitHub cloud. This can also be via the GITHUB_BASE_URL environment variable.
app_id - Github App ID for your Github organization. This can also be set via the GITHUB_APP_ID environment variable.
app_installation_id - Github App installation ID for your Github App installation. This can also be set via the GITHUB_APP_INSTALLATION_ID environment variable.
app_private_key - Github App private key path for your Github App. This can also be set via the GITHUB_APP_PEM_FILE environment variable.

Postgres FDW

This plugin is available as a native Postgres FDW. Unlike Steampipe CLI, which ships with an embedded Postgres server instance, the Postgres FDW can be installed in any supported Postgres database version.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_postgres_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/postgres.sh)" -- github

The installer will prompt you for the plugin name and version, download and install the appropriate files for your OS, system architecture, and Postgres version.

To configure the Postgres FDW, you will create an extension, foreign server, and schema and import the foreign schema.

CREATE EXTENSION IF NOT EXISTS steampipe_postgres_github;
CREATE SERVER steampipe_github FOREIGN DATA WRAPPER steampipe_postgres_github OPTIONS (config '<your_config>');
CREATE SCHEMA github;
IMPORT FOREIGN SCHEMA github FROM SERVER steampipe_github INTO github;

SQLite Extension

This plugin is available as a SQLite Extension, making the tables available as SQLite virtual tables.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_sqlite_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/sqlite.sh)" -- github

The installer will prompt you for the plugin name, version, and destination directory. It will then determine the OS and system architecture, and it will download and install the appropriate package.

To configure the SQLite extension, load the extension module and then run the steampipe_configure_github function to configure it with plugin-specific options.

$ sqlite3
sqlite> .load ./steampipe_sqlite_extension_github.so
sqlite> select steampipe_configure_github('<your_config>');

Export

This plugin is available as a standalone Export CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- github

You can pass the configuration to the command with the --config argument:

steampipe_export_github --config '<your_config>' <table_name>