urlscan.io + Steampipe
urlscan.io is a website scanner to gather request, cookie, link, headers and other information from a URL.
Steampipe is an open source CLI to instantly query cloud APIs using SQL.
Search website scans:
select id, page_url, task_timefrom urlscan_searchwhere query = 'steampipe.io'
+--------------------------------------+---------------------------+---------------------+| id | page_url | task_time |+--------------------------------------+---------------------------+---------------------+| a20397de-1f23-4821-ae2a-87ca612ebcff | https://steampipe.io/ | 2021-05-06 01:39:14 || 698e6270-cd8f-4090-a3ae-23bb5d3ee0a8 | https://steampipe.io/ | 2020-12-15 19:44:10 || d01fa12c-f30f-48b8-af7e-5539f0d1bc3c | https://hub.steampipe.io/ | 2020-12-04 11:44:02 |+--------------------------------------+---------------------------+---------------------+
Query URL information from a scan:
select subject_name, issuer, valid_tofrom urlscan_certificatewhere -- vercel.com scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
+-------------------------------+--------------------------------------------+---------------------+| subject_name | issuer | valid_to |+-------------------------------+--------------------------------------------+---------------------+| *.vercel.com | R3 | 2021-06-15 03:05:04 || vercel-insights.com | Amazon | 2021-12-22 23:59:59 || *.google.com | GTS CA 1O1 | 2021-07-06 10:11:14 || ... | ... | ... |+-------------------------------+--------------------------------------------+---------------------+
Documentation
Get started
Install
Download and install the latest urlscan.io plugin:
steampipe plugin install urlscan
Credentials
Item | Description |
---|---|
Credentials | urlscan.io requires a free API key for all requests. |
Radius | Each connection represents a single urlscan.io account. |
Configuration
Installing the latest urlscan plugin will create a config file (~/.steampipe/config/urlscan.spc
) with a single connection named urlscan
:
connection "urlscan" { plugin = "urlscan" api_key = "e89d13c7-43a0-4f48-a829-590b4091bac6"}
api_key
- Your urlscan.io API key.
Get involved
- Open source: https://github.com/turbot/steampipe-plugin-urlscan
- Community: Slack Channel