Table: splunk_index - Query Splunk Indexes using SQL

Splunk is a software platform widely used for monitoring, searching, analyzing and visualizing the machine-generated data. In Splunk, an Index is a place where Splunk stores the data. Indexes are used in Splunk to maintain the data, with its own set of configuration options that control its behavior.

Table Usage Guide

The splunk_index table provides insights into indexed data within Splunk. As a Splunk administrator or a security analyst, explore index-specific details through this table, including data storage, retention policies, and associated metadata. Utilize it to uncover information about indexes, such as their current size, maximum size, home path, and more.

Examples

Basic info

Discover the segments that have the most events based on their maximum time, useful for identifying trends and high-activity periods within your data. This can be beneficial for optimizing resource allocation and planning future data management strategies.

select
  name,
  max_time,
  total_event_count
from
  splunk_index;

select
  name,
  max_time,
  total_event_count
from
  splunk_index;

List disabled indexes

Assess the elements within your system that have been disabled to better manage your resources and ensure optimal performance. This allows you to identify areas of inefficiency and take corrective measures.

select
  name,
  max_time,
  total_event_count
from
  splunk_index
where
  disabled;

select
  name,
  max_time,
  total_event_count
from
  splunk_index
where
  disabled = 1;

List all internal indexes

Discover the segments that encompass all internal indexes, enabling you to analyze event frequency and duration. This aids in efficient data management and optimal resource allocation.

select
  name,
  max_time,
  total_event_count
from
  splunk_index
where
  is_internal;

select
  name,
  max_time,
  total_event_count
from
  splunk_index
where
  is_internal = 1;

Get index count by type

Determine the distribution of different types of data across indexes to optimize data management and enhance system performance.

select
  data_type,
  count(*)
from
  splunk_index
group by
  data_type;

select
  data_type,
  count(*)
from
  splunk_index
group by
  data_type;

Schema for splunk_index

Name	Type	Operators	Description
_ctx	jsonb		Steampipe context in JSON form.
assure_utf8	boolean		Indicates whether all data retreived from the index is proper UTF8. If enabled (set to True), degrades indexing performance. This is a global setting, not a per index setting.
block_sign_size	bigint		Controls How many events make up a block for block signatures. If this is set to 0, block signing is disabled for this index.
block_signature_database	text		The index that stores block signatures of events. This is a global setting, not a per index setting.
cold_path	text		Filepath to the cold databases for the index.
cold_path_expanded	text		Absoute filepath to the cold databases.
cold_to_frozen_dir	text		Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk software automatically puts frozen buckets in this directory.
cold_to_frozen_script	text		Path to the archiving script.
current_db_size_mb	bigint		Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.
data_type	text	=	The type of index: event, metric.
default_database	text		If no index destination information is available in the input data, the index shown here is the destination of such data.
disabled	boolean		If no index destination information is available in the input data, the index shown here is the destination of such data.
enable_realtime_search	boolean		Indicates if this is a real-time search. This is a global setting, not a per index setting.
frozen_time_period_in_secs	bigint		Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years). Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.
home_path	text		An absolute path that contains the hot and warm buckets for the index.
home_path_expanded	text		An absolute filepath to the hot and warm buckets for the index.
index_threads	text		Number of threads used for indexing. This is a global setting, not a per index setting.
is_internal	boolean		True if this is an internal index (for example, _internal, _audit).
is_ready	boolean		True if the index is properly initialized.
last_init_time	timestamp with time zone		Last time the index processor was successfully initialized. This is a global setting, not a per index setting.
max_concurrent_optimizes	bigint		The number of concurrent optimize processes that can run against a hot bucket. This number should be increased if instructed by Splunk Support. Typically the default value should suffice.
max_data_size	text		The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying 'auto' or 'auto_high_volume' causes Splunk software to autotune this parameter (recommended). Use 'auto_high_volume' for high volume indexes (such as the main index); otherwise, use 'auto'. A 'high volume index' is typically one that gets over 10GB of data per day. 'auto' sets the size to 750MB. 'auto_high_volume' sets the size to 10GB on 64-bit, and 1GB on 32-bit systems. Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.
max_hot_buckets	text		Maximum hot buckets that can exist per index. Defaults to 3. When maxHotBuckets is exceeded, Splunk software rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.
max_hot_idle_secs	bigint		Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time). If a hot bucket exceeds maxHotIdleSecs, Splunk software rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.
max_hot_span_secs	bigint		Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).
max_mem_db	bigint		The amount of memory, in MB, allocated for indexing. This is a global setting, not a per index setting.
max_meta_entries	bigint		Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite). If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you do not use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).
max_running_process_groups	bigint		Maximum number of processes that the indexer fires off at a time. This is a global setting, not a per index setting.
max_time	timestamp with time zone		ISO8601 timestamp of the newest event time in the index.
max_total_data_size_mb	bigint		The maximum size of an index, in MB.
max_warm_db_count	bigint		The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.
mem_pool_mb	text		Determines how much memory is given to the indexer memory pool. This is a global setting, not a per-index setting.
min_raw_file_sync_secs	text		Can be either an integer (or 'disable'). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed. The integer sets how frequently splunkd forces a filesystem sync while compressing journal slices. During this period, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files. If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying 'disable' disables syncing entirely: uncompressed slices are removed as soon as compression is complete.
min_time	timestamp with time zone		ISO8601 timestamp of the oldest event time in the index.
name	text	=	Name of the index.
partial_service_meta_period	bigint		Related to serviceMetaPeriod. By default it is turned off (zero). If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod. partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens. If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.
quarantine_future_secs	bigint		Events with timestamp of quarantineFutureSecs newer than 'now' that are dropped into quarantine bucket. Defaults to 2592000 (30 days). This is a mechanism to prevent main hot buckets from being polluted with fringe events.
quarantine_past_secs	bigint		Events with timestamp of quarantinePastSecs older than 'now' are dropped into quarantine bucket. Defaults to 77760000 (900 days). This is a mechanism to prevent the main hot buckets from being polluted with fringe events.
raw_chunk_size_bytes	bigint		Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.
rotate_period_in_secs	bigint		Rotation period, in seconds, that specifies how frequently to check: If a new hot bucket needs to be created. If there are any cold buckets that should be frozen. If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.
service_meta_period	bigint		Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds). You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.
sp_connection_name	text	=, !=, ~~, ~~, !~~, !~~	Steampipe connection name.
sp_ctx	jsonb		Steampipe context in JSON form.
summarize	boolean		If true, leaves out certain index details, which provides a faster response.
suppress_banner_list	text		List of indexes for which we suppress 'index missing' warning banner messages. This is a global setting, not a per index setting.
sync	bigint		Specifies the number of events that trigger the indexer to sync events. This is a global setting, not a per index setting.
sync_meta	boolean		When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.
thawed_path	text		An absolute path that contains the thawed (resurrected) databases for the index.
thawed_path_expanded	text		Absolute filepath to the thawed (resurrected) databases.
throttle_check_period	bigint		Defines how frequently Splunk software checks for index throttling condition, in seconds. Defaults to 15 (seconds).
total_event_count	bigint		Total number of events in the index.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- splunk

You can pass the configuration to the command with the --config argument:

steampipe_export_splunk --config '<your_config>' splunk_index