Table: urlscan_request - Query Urlscan Requests using SQL

Urlscan is a tool used to analyze and scan websites for potential security threats. It checks various aspects of a webpage, including HTTP requests made during the scanning process. These requests are crucial for understanding the nature of the webpage and its potential security implications.

Table Usage Guide

The urlscan_request table provides insights into the HTTP requests made during the scanning process by Urlscan. As a Security Analyst, you can use this table to uncover detailed information about these requests, such as the method, URL, and headers. This can assist in identifying potential security threats and understanding the behavior of the scanned webpage.

Important Notes

You must specify the scan in the where clause to query this table.

Examples

List requests

Analyze the sequence of requests made during a specific web scan to understand the progression and interactions within that session. This could be useful in identifying potential security threats or anomalies in the browsing session.

select
  *
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
  timestamp;

select
  *
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
  timestamp;

Post requests made while loading the page

Explore which URLs were accessed during a specific page load, particularly focusing on POST requests. This is beneficial to understand the data being transmitted during the loading process, which can aid in troubleshooting or security analysis.

select
  url,
  type,
  response_status,
  post_data
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
  and has_post_data;

select
  url,
  type,
  response_status,
  post_data
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
  and has_post_data = 1;

5 largest requests in the page

Determine the areas in which your webpage is making the largest requests to help focus optimization efforts and improve loading times.

select
  response_size,
  document_type,
  url
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
  response_size desc
limit
  5;

select
  response_size,
  document_type,
  url
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
order by
  response_size desc
limit
  5;

Types of documents requested in page

Assess the elements within a specific webpage scan to understand the frequency and total size of different document types requested. This can be useful in identifying potential areas of optimization to improve webpage loading speeds.

select
  document_type,
  count(*) as count,
  sum(response_size) as total_size_bytes
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
group by
  document_type
order by
  total_size_bytes desc;

select
  document_type,
  count(*) as count,
  sum(response_size) as total_size_bytes
from
  urlscan_request
where
  scan = '54c78f69-5294-4a17-8ae0-a71943954e09'
group by
  document_type
order by
  total_size_bytes desc;

Schema for urlscan_request

Name	Type	Operators	Description
_ctx	jsonb		Steampipe context in JSON form.
document_type	text		Type of document requested.
document_url	text		URL of the document the requests are building.
frame_id	text		Frame ID in the request.
has_post_data	boolean		True if the request includes post data.
has_user_gesture	boolean		True if the request has a user gesture.
headers	jsonb		Request headers.
id	text		ID of the request.
initial_priority	text		Initial priority for the request.
initiator	jsonb		Initiator (e.g. page) that made the request.
loader_id	text		ID of the loader for the request.
method	text		HTTP method used for the request.
mixed_content_type	text		Mixed content type information.
post_data	text		HTTP post data sent with the request.
referrer_policy	text		Referrer policy for the request.
response_abp	jsonb		ABP information for the response.
response_asn	jsonb		ASN information for the IP address handling the request.
response_data_length	bigint		Response data length in bytes.
response_document_type	text		Type of document returned in the response.
response_encoded_data_length	bigint		Response encoded data length in bytes.
response_geolocation	jsonb		Geolocation information for the IP address handling the request.
response_hash	text		Hash of the response.
response_hash_matches	jsonb		Hash matches for the response.
response_headers	jsonb		Response headers.
response_mime_type	text		Mime type of the response.
response_protocol	text		HTTP protocol for the response, e.g. h2.
response_remote_ip_address	text		IP address of the server responding to the request.
response_remote_port	bigint		Port for the server responding to the request.
response_reverse_dns	jsonb		Reverse DNS information for the IP address handling the request.
response_security_details	jsonb		Security details for the response.
response_security_headers	jsonb		Security headers for the response.
response_security_state	text		Security state for the response.
response_size	bigint		Size of the response.
response_status	bigint		HTTP status code of the response, e.g. 200.
response_status_text	text		HTTP status text for the response.
response_timing	jsonb		Timing data for the response.
scan	text	=	ID of the scan result.
sp_connection_name	text	=, !=, ~~, ~~, !~~, !~~	Steampipe connection name.
sp_ctx	jsonb		Steampipe context in JSON form.
timestamp	double precision		Request timestamp received from urlscan. (Unknown format?)
url	text		URL for the request.
wall_time	timestamp with time zone		Clock time when the request was made.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- urlscan

You can pass the configuration to the command with the --config argument:

steampipe_export_urlscan --config '<your_config>' urlscan_request