steampipe plugin install urlscan

Table: urlscan_request - Query Urlscan Requests using SQL

Urlscan is a tool used to analyze and scan websites for potential security threats. It checks various aspects of a webpage, including HTTP requests made during the scanning process. These requests are crucial for understanding the nature of the webpage and its potential security implications.

Table Usage Guide

The urlscan_request table provides insights into the HTTP requests made during the scanning process by Urlscan. As a Security Analyst, you can use this table to uncover detailed information about these requests, such as the method, URL, and headers. This can assist in identifying potential security threats and understanding the behavior of the scanned webpage.

Important Notes

  • You must specify the scan in the where clause to query this table.

Examples

List requests

Analyze the sequence of requests made during a specific web scan to understand the progression and interactions within that session. This could be useful in identifying potential security threats or anomalies in the browsing session.

select
*
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
timestamp;
select
*
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
timestamp;

Post requests made while loading the page

Explore which URLs were accessed during a specific page load, particularly focusing on POST requests. This is beneficial to understand the data being transmitted during the loading process, which can aid in troubleshooting or security analysis.

select
url,
type,
response_status,
post_data
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
and has_post_data;
select
url,
type,
response_status,
post_data
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
and has_post_data = 1;

5 largest requests in the page

Determine the areas in which your webpage is making the largest requests to help focus optimization efforts and improve loading times.

select
response_size,
document_type,
url
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
response_size desc
limit
5;
select
response_size,
document_type,
url
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
response_size desc
limit
5;

Types of documents requested in page

Assess the elements within a specific webpage scan to understand the frequency and total size of different document types requested. This can be useful in identifying potential areas of optimization to improve webpage loading speeds.

select
document_type,
count(*) as count,
sum(response_size) as total_size_bytes
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
group by
document_type
order by
total_size_bytes desc;
select
document_type,
count(*) as count,
sum(response_size) as total_size_bytes
from
urlscan_request
where
scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
group by
document_type
order by
total_size_bytes desc;

Schema for urlscan_request

NameTypeOperatorsDescription
_ctxjsonbSteampipe context in JSON form, e.g. connection_name.
document_typetextType of document requested.
document_urltextURL of the document the requests are building.
frame_idtextFrame ID in the request.
has_post_databooleanTrue if the request includes post data.
has_user_gesturebooleanTrue if the request has a user gesture.
headersjsonbRequest headers.
idtextID of the request.
initial_prioritytextInitial priority for the request.
initiatorjsonbInitiator (e.g. page) that made the request.
loader_idtextID of the loader for the request.
methodtextHTTP method used for the request.
mixed_content_typetextMixed content type information.
post_datatextHTTP post data sent with the request.
referrer_policytextReferrer policy for the request.
response_abpjsonbABP information for the response.
response_asnjsonbASN information for the IP address handling the request.
response_data_lengthbigintResponse data length in bytes.
response_document_typetextType of document returned in the response.
response_encoded_data_lengthbigintResponse encoded data length in bytes.
response_geolocationjsonbGeolocation information for the IP address handling the request.
response_hashtextHash of the response.
response_hash_matchesjsonbHash matches for the response.
response_headersjsonbResponse headers.
response_mime_typetextMime type of the response.
response_protocoltextHTTP protocol for the response, e.g. h2.
response_remote_ip_addresstextIP address of the server responding to the request.
response_remote_portbigintPort for the server responding to the request.
response_reverse_dnsjsonbReverse DNS information for the IP address handling the request.
response_security_detailsjsonbSecurity details for the response.
response_security_headersjsonbSecurity headers for the response.
response_security_statetextSecurity state for the response.
response_sizebigintSize of the response.
response_statusbigintHTTP status code of the response, e.g. 200.
response_status_texttextHTTP status text for the response.
response_timingjsonbTiming data for the response.
scantext=ID of the scan result.
timestampdouble precisionRequest timestamp received from urlscan. (Unknown format?)
urltextURL for the request.
wall_timetimestamp with time zoneClock time when the request was made.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- urlscan

You can pass the configuration to the command with the --config argument:

steampipe_export_urlscan --config '<your_config>' urlscan_request