Hacker News + Steampipe

Hacker News is a social news website focusing on computer science and entrepreneurship. Steampipe marshalls the HN API data into queryable tables letting you interactivly explore it via our command line interface or your favorite SQL client.

Steampipe is an open-source zero-ETL engine to instantly query cloud APIs using SQL.

Example query:

select
  score,
  descendants as comments,
  title
from
  hackernews_top
where
  type = 'story'
  and lower(title) like '%sql%'
order by
  score desc;

standard output (can use .output to change to csv or json):

+-------+----------+---------------------------------------------------------------------+
| score | comments | title                                                               |
+-------+----------+---------------------------------------------------------------------+
| 242   | 300      | Query Hacker News API with SQL                                      |
| 121   | 127      | Why Uber Engineering Switched from Postgres to MySQL (2016)         |
| 70    | 12       | Show HN: QueryCal – calculate metrics from your calendars using SQL |
| 17    | 10       | Global Associative Arrays in PostgreSQL                             |
+-------+----------+---------------------------------------------------------------------+

Documentation

Table definitions & examples →

Get started

Install

Download and install the latest Hacker News plugin:

steampipe plugin install hackernews

Credentials

The Hacker News API is open to the public and does not require any credentials.

Configuration

Connection configurations are defined using HCL in one or more Steampipe config files. Steampipe will load ALL configuration files from ~/.steampipe/config that have a .spc extension. A config file may contain multiple connections.

Installing the latest hackernews plugin will create a default connection named hackernews in the ~/.steampipe/config/hackernews.spc file. You may edit this connection to set options:

connection "hackernews" {
  plugin    = "hackernews"
  max_items = 5000
}

Postgres FDW

This plugin is available as a native Postgres FDW. Unlike Steampipe CLI, which ships with an embedded Postgres server instance, the Postgres FDW can be installed in any supported Postgres database version.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_postgres_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/postgres.sh)" -- hackernews

The installer will prompt you for the plugin name and version, download and install the appropriate files for your OS, system architecture, and Postgres version.

To configure the Postgres FDW, you will create an extension, foreign server, and schema and import the foreign schema.

CREATE EXTENSION IF NOT EXISTS steampipe_postgres_hackernews;
CREATE SERVER steampipe_hackernews FOREIGN DATA WRAPPER steampipe_postgres_hackernews OPTIONS (config '<your_config>');
CREATE SCHEMA hackernews;
IMPORT FOREIGN SCHEMA hackernews FROM SERVER steampipe_hackernews INTO hackernews;

SQLite Extension

This plugin is available as a SQLite Extension, making the tables available as SQLite virtual tables.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_sqlite_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/sqlite.sh)" -- hackernews

The installer will prompt you for the plugin name, version, and destination directory. It will then determine the OS and system architecture, and it will download and install the appropriate package.

To configure the SQLite extension, load the extension module and then run the steampipe_configure_hackernews function to configure it with plugin-specific options.

$ sqlite3
sqlite> .load ./steampipe_sqlite_extension_hackernews.so
sqlite> select steampipe_configure_hackernews('<your_config>');

Export

This plugin is available as a standalone Export CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- hackernews

You can pass the configuration to the command with the --config argument:

steampipe_export_hackernews --config '<your_config>' <table_name>