SPLK-2002 Performance Tuning — Timed 20 Questions

Question 1

A Splunk administrator needs to identify why a specific search is experiencing high 'Disk Read' wait times. Which tool or log source should be utilized to correlate search IDs with specific disk latency metrics?

20 minute time limit — choose an answer to begin.

Accepted Answer

Audit logs correlated with indexer metrics via the Monitoring Console. The Monitoring Console aggregates indexer metrics, including disk latency, and can correlate those with search activity. Audit logs record search IDs and user actions, so combining audit logs with indexer metrics via the Monitoring Console allows an administrator to link a specific search ID to disk read wait times. The other tools either lack the necessary correlation or do not provide disk latency metrics.

Answer

Search Inspector

Answer

Splunk Web UI Performance Monitor

Answer

splunkd.log

Question 2

You are tuning a Splunk heavy forwarder. The 'parsingQueue' is consistently full. Which configuration change in props.conf or inputs.conf is recommended?

Accepted Answer

Increase the 'maxQueueSize' for the parsing queue in limits.conf. The parsing queue is typically blocked by complex regex transformations or heavy parsing loads. Adjusting pipeline settings or splitting the workload across more CPUs is necessary.

Answer

Reduce the number of indexers in the output group

Answer

Disable 'sourcetype' auto-discovery

Question 3

Which setting in indexes.conf should be tuned to balance memory usage and indexing speed for a high-volume indexer?

Accepted Answer

maxDataSize. maxDataSize controls the target size of hot buckets before Splunk rolls them to warm; tuning it directly balances how much memory the indexer dedicates to active buckets versus how frequently it performs bucket rolling and disk I/O. Larger values keep more data in memory (faster search/indexing) but consume more RAM, while smaller values reduce memory pressure at the cost of more frequent rolling. This is the primary knob in indexes.conf for that trade-off.

Answer

hot_bucket_count

Answer

io_priority

Answer

block_size

Question 4

An administrator notices that searches are slow due to high CPU utilization on indexers. Which configuration change in limits.conf directly controls the maximum number of concurrent searches allowed on an indexer?

Accepted Answer

max_searches_per_cpu. max_searches_per_cpu in limits.conf sets the multiplier applied to the number of CPU cores to determine the maximum concurrent searches an indexer will run. Splunk computes the cap as max_searches_per_cpu × number of cores, so lowering this value directly throttles concurrent search load and reduces CPU contention. This is the documented knob for controlling search concurrency per indexer.

Answer

concurrent_search_limit

Answer

base_max_searches

Answer

max_searches

Question 5

When using Data Model acceleration, which setting should be monitored to ensure the background summary search does not starve other searches of resources?

Accepted Answer

accelerated_search_max_threads. The 'tstats_max_search_time' or similar limits in limits.conf manage how long these accelerated searches run, but the 'accelerated_search_max_threads' setting controls the concurrency.

Answer

summary_index_concurrency

Answer

max_summary_size

Answer

data_model_memory_limit

Question 6

When performance tuning the 'Splunk Web' interface for users, which configuration helps manage the 'Search Results' cache?

Accepted Answer

cache_results in limits.conf. In Splunk, the search results cache is controlled by the 'cache_results' setting in limits.conf, which determines whether search results are cached and how the cache behaves. Tuning this setting helps manage how often identical searches reuse cached results versus re-running against the indexers, directly impacting Splunk Web responsiveness and indexer load. This is the documented configuration for managing the search results cache.

Answer

results_ttl in server.conf

Answer

max_results_cache in authorize.conf

Answer

web_cache_size in web.conf

Question 7

Which Splunk component is responsible for receiving data from forwarders and distributing it to the correct indexers?

Accepted Answer

Indexer. In Splunk, the indexer is the component that receives data from forwarders (universal or heavy) and indexes it. Indexers also distribute data to other indexers in a cluster based on replication and search factors. Thus, the indexer is responsible for both receiving and distributing data to the correct indexers in a clustered environment.

Answer

Search Head

Answer

License Master

Answer

Deployment Server

Question 8

You are configuring a heavy forwarder to filter data before sending it to indexers. Which stanza in props.conf is used to define the filtering criteria?

Accepted Answer

[sourcetype::*]. Filtering is performed using the 'TRANSFORMS' attribute in props.conf, which points to a stanza in transforms.conf.

Answer

[FILTER]

Answer

[OUTPUT]

Answer

[FORWARDER]

Question 9

An indexer is running out of disk space. Which setting in indexes.conf prevents the indexer from crashing by stopping ingestion?

Accepted Answer

maxTotalDataSizeMB. maxTotalDataSizeMB sets a hard size ceiling for an index. When the index reaches this limit, Splunk rolls the oldest buckets to frozen and deletes them (or archives them), freeing disk space so ingestion can continue without the indexer running out of disk and crashing. This is the correct setting to cap index growth and prevent disk exhaustion.

Answer

frozenTimePeriodInSecs

Answer

minFreeSpaceMB

Answer

maxHotBuckets

Question 10

Which setting in limits.conf limits the amount of memory a single search can consume on the search head?

Accepted Answer

max_mem_usage_mb. In Splunk's limits.conf, the [search] stanza contains the setting max_mem_usage_mb, which caps the memory (in megabytes) that a single search process can consume on the search head. When a search exceeds this limit, Splunk terminates it to prevent resource exhaustion. This is the correct setting for controlling per-search memory usage on the search head.

Answer

process_memory_limit

Answer

max_ram_per_search

Answer

search_memory_limit

Question 11

Which TWO settings in limits.conf are used to control search concurrency for roles?

Accepted Answer

max_searches_per_user (authorize.conf). The 'max_searches_per_user' and 'max_searches_per_role' define concurrency limits per entity in authorize.conf, but limits.conf has general concurrency settings. Actually, these are primarily in authorize.conf. Wait, which TWO?

Answer

max_search_concurrency (limits.conf)

Answer

search_concurrency (limits.conf)

Answer

max_concurrent_searches (server.conf)

Question 12

Which THREE settings in indexes.conf help manage bucket size and count?

Accepted Answer

frozenTimePeriodInSecs. In indexes.conf, maxDataSize (E) controls the target size of hot buckets in MB (auto, auto_high_volume, or a numeric value), directly governing how large a bucket grows before rolling to warm, which is central to managing bucket size. maxWarmDBCount (C) sets the maximum number of warm buckets retained per index before the oldest are rolled to cold or frozen, directly managing bucket count. frozenTimePeriodInSecs (A) defines the retention period in seconds after which buckets are frozen (and typically deleted), which bounds the total number of buckets kept over time and thus manages bucket count. max_searches_per_cpu (B) is a limits.conf setting controlling search concurrency per CPU, not bucket sizing or count, and index_concurrency (D) is not a valid indexes.conf setting for managing bucket size or count.

Answer

max_searches_per_cpu

Answer

index_concurrency

Question 13

A Splunk architect is tuning a search head that runs many concurrent scheduled reports. Users report that ad-hoc searches are frequently queued behind scheduled searches, even though CPU and memory usage are moderate. Which configuration change in limits.conf would best address this by allowing ad-hoc searches to bypass the scheduled search queue?

Accepted Answer

Enable the prioritize_ad_hoc_searches setting in the [search] stanza.. The prioritize_ad_hoc_searches setting in limits.conf is specifically designed to give ad-hoc searches precedence over scheduled searches in the search queue. When enabled, ad-hoc searches are moved ahead of scheduled searches, reducing wait times for interactive users. This directly solves the problem of ad-hoc searches being delayed by scheduled reports, without requiring changes to concurrency limits or queue sizes.

Answer

Increase the queue size for the [queue] stanza in server.conf.

Answer

Set the max_searches_per_cpu setting to a higher value.

Answer

Set the max_searches_per_cpu setting to a lower value in limits.conf.

Question 14

When managing indexer clustering, what is the impact of a high 'replication_factor' on indexing performance?

Accepted Answer

Increases indexing latency due to synchronous replication requirements. A high replication_factor means each bucket must be replicated to more indexer peers before the data is considered safely indexed. Because replication is synchronous for hot buckets in an indexer cluster, the indexing pipeline waits for acknowledgments from all required peers, increasing indexing latency. This is the direct performance impact.

Answer

Allows for faster data ingestion without resource constraints

Answer

Improves search speed by reducing data density

Answer

Decreases indexer storage requirements

Question 15

Which dashboard in the Monitoring Console provides the best overview of resource consumption per search head?

Accepted Answer

Search Head Activity. The Search Head Activity dashboard in the Monitoring Console provides the best overview of resource consumption per search head, showing metrics such as CPU usage, memory, disk I/O, and search load broken down by search head. It is designed specifically to help admins identify which search heads are under heavy load and to balance search distribution. Other dashboards focus on different tiers (indexers, forwarders) or different concerns (ingestion performance).

Answer

Data Ingestion Performance

Answer

Resource Usage: Forwarders

Answer

Indexer Activity

Question 16

You are analyzing search performance using the Search Activity dashboard. You observe that 'Result Count' is extremely high for a specific saved search. Which optimization technique is most effective to reduce the load on the indexer?

Accepted Answer

Use specific time ranges and selective filters at the beginning of the search. To reduce the load on the indexer when a saved search returns an extremely high result count, the most effective optimization is to use specific time ranges and selective filters at the beginning of the search. This reduces the amount of data that the indexer must scan and process, directly lowering the load. Other options either do not address indexer load or are incorrect.

Answer

Increase the memory allocation for the search head

Answer

Disable data model acceleration for the index

Answer

Use the | table command at the start

Question 17

A user complains that a search is slow despite having a small time range. The Search Inspector reveals 'event_count' is high, but 'scanned_count' is also high. What is the likely cause?

Accepted Answer

The search is not using an indexed field to filter data. A high 'scanned_count' alongside a high 'event_count' means Splunk is reading many events from disk even though the time range is small — the search is not narrowing the dataset with an indexed field (like index, sourcetype, host, or source) before the base search runs. Without an indexed filter, Splunk must scan raw buckets and apply filtering at search time, which is far slower. Adding an indexed field to the base search dramatically reduces scanned_count.

Answer

The indexer has too much memory

Answer

Data model acceleration is failing

Answer

The search head is underpowered

Question 18

In a multi-site indexer cluster, which setting controls the number of copies of data kept per site?

Accepted Answer

site_replication_factor. In a multi-site indexer cluster, site_replication_factor is the setting in server.conf (under the [clustering] stanza) that defines how many copies of each bucket must exist within each site. It works alongside replication_factor, which sets the total number of copies across all sites. This ensures site-level redundancy so that a site failure does not cause data loss.

Answer

max_site_copies

Answer

site_data_count

Answer

cross_site_replication

Question 19

Which command helps you understand if your search is retrieving too much data from the disk by analyzing the 'index_time' and 'search_time'?

Accepted Answer

Search Inspector. The Search Inspector in Splunk is a built-in tool that provides detailed performance metrics for a specific search job, including the time spent on index_time (disk I/O) and search_time (processing). By examining these timings, you can determine if the search is reading excessive data from disk, which often indicates inefficiencies like lack of filtering or improper use of indexes. It is specifically designed for this diagnostic purpose, making it the correct choice.

Answer

Monitoring Console Search Activity Dashboard

Answer

btool

Answer

splunkd log analysis

Question 20

When tuning search performance, what is the 'join' command's primary drawback in terms of resource usage?

Accepted Answer

It consumes significant memory on the search head. The join command in Splunk loads the entire subsearch result set into memory on the search head and holds it while matching against the main search results. This in-memory hash/merge operation is memory-intensive and can trigger search head memory pressure or the subsearch result limit. That memory cost on the search head is the command's primary resource drawback.

Answer

It requires data to be fully ingested before searching

Answer

It causes high disk I/O on the indexer

Answer

It disables parallel processing