Performance Tuning practice questions

Search Inspector

Why it fails: Search Inspector shows search performance but not OS-level disk latency wait times.

Splunk Web UI Performance Monitor

Why it fails: This is not a native tool within Splunk Enterprise.

splunkd.log

Why it fails: splunkd.log contains errors but lacks the direct correlation for disk I/O metrics per search ID.

Reduce the number of indexers in the output group

Why it fails: This would likely increase the queue backup.

Disable 'sourcetype' auto-discovery

Why it fails: This has negligible impact on parsing queue capacity.

hot_bucket_count

Why it fails: This is not a parameter in indexes.conf.

io_priority

Why it fails: This is an OS-level setting, not an indexes.conf parameter.

block_size

Why it fails: This is a low-level setting not typically tuned for index performance.

concurrent_search_limit

Why it fails: This is not a configuration parameter in Splunk.

base_max_searches

Why it fails: This setting does not exist in the indexer performance section.

max_searches

Why it fails: This is not a valid setting name in limits.conf.

summary_index_concurrency

Why it fails: This is not a standard Splunk setting.

max_summary_size

Why it fails: There is no such setting.

data_model_memory_limit

Why it fails: Not a valid configuration setting.

results_ttl in server.conf

Why it fails: This is not the correct path or setting.

max_results_cache in authorize.conf

Why it fails: This does not exist.

web_cache_size in web.conf

Why it fails: This is not a valid setting for search results.

Search Head

Why it fails: Search heads coordinate searches, not data ingestion.

License Master

Why it fails: The License Master manages usage tokens.

Deployment Server

Why it fails: The Deployment Server manages configurations.

[FILTER]

Why it fails: This is not a valid stanza name.

[OUTPUT]

Why it fails: This is not a valid configuration stanza.

[FORWARDER]

Why it fails: Not a valid stanza.

frozenTimePeriodInSecs

Why it fails: This controls data retention, not disk space hard limits.

minFreeSpaceMB

Why it fails: This is for emergency stop, but maxTotalDataSizeMB is the primary management tool.

maxHotBuckets

Why it fails: This limits the number of hot buckets, not total size.

process_memory_limit

Why it fails: Not the specific setting for searches.

max_ram_per_search

Why it fails: Incorrect syntax.

search_memory_limit

Why it fails: Incorrect syntax.

max_search_concurrency (limits.conf)

Why it fails: Not a valid setting.

search_concurrency (limits.conf)

Why it fails: Not a valid setting.

max_concurrent_searches (server.conf)

Why it fails: Not a valid setting.

max_searches_per_cpu

Why it fails: This is in limits.conf.

index_concurrency

Why it fails: Not a valid setting.

Increase the queue size for the [queue] stanza in server.conf.

Why it fails: The [queue] stanza in server.conf controls the size of internal queues such as the search queue, but increasing queue size only allows more searches to be queued; it does not change the order in which they are processed. Ad-hoc searches would still wait behind scheduled searches. This setting affects throughput, not priority.

Set the max_searches_per_cpu setting to a higher value.

Why it fails: max_searches_per_cpu controls the maximum number of concurrent searches per CPU core, but it does not prioritize ad-hoc searches over scheduled ones. Increasing it would allow more searches overall, potentially worsening contention, and it does not prevent scheduled searches from occupying queue slots ahead of ad-hoc searches. This setting is not designed to bypass the scheduled search queue.

Set the max_searches_per_cpu setting to a lower value in limits.conf.

Why it fails: Lowering max_searches_per_cpu reduces the number of concurrent searches allowed per CPU, which would likely cause more queuing and longer wait times for all searches. It does not provide any prioritization for ad-hoc searches. This change would degrade performance rather than improve ad-hoc search responsiveness.

Allows for faster data ingestion without resource constraints

Why it fails: Replication adds write and network overhead, so ingestion slows rather than proceeding without resource constraints. Replication_factor sets how many bucket copies exist; it would be correct only if the question asked how searchable copies are maintained when an indexer goes offline.

Improves search speed by reducing data density

Why it fails: Replication copies buckets to peer indexers; it does not reduce data density or accelerate searches. Higher replication_factor increases network and disk write overhead, slowing ingestion. It would be correct only if the question asked how search availability survives an indexer failure.

Decreases indexer storage requirements

Why it fails: Each replica consumes additional storage on peer indexers, so storage requirements rise rather than fall. Replication_factor governs how many copies of each bucket exist; it would be the right answer only if the question asked how clustering improves data durability across peer failures.

Data Ingestion Performance

Why it fails: Data Ingestion Performance reports indexing throughput, parsing queues and pipeline latency per indexer, so it carries no per-search-head CPU, memory or disk figures. It is tempting because ingestion load does consume search head resources when indexing is distributed, but it answers "how fast is data arriving", not "which search head is overloaded".

Resource Usage: Forwarders

Why it fails: Resource Usage: Forwarders reports per-forwarder CPU, memory and queue metrics, not search head consumption, so it cannot answer the stem. It is tempting because it does aggregate resource data across a distributed deployment, and it would be the right dashboard when diagnosing forwarder performance or ingestion bottlenecks rather than search head load.

Indexer Activity

Why it fails: Indexer Activity reports indexing-pipeline throughput and queue behaviour on indexers, not per-search-head resource consumption, so it cannot answer the stem. It is tempting because it is the natural dashboard for diagnosing indexer performance, licence usage, and indexing bottlenecks — the correct choice when the question asks about indexing tier health rather than search head resource usage.

Increase the memory allocation for the search head

Why it fails: Search head memory governs search-time processing such as merging, lookups and dashboard rendering; it does not reduce the indexer-side work of scanning a high result count. It is tempting because memory tuning is a standard Splunk performance lever, and it would be correct when the search head itself is the constrained component.

Disable data model acceleration for the index

Why it fails: Data model acceleration builds separate TSIDX summary files that speed up analytics; disabling it removes that benefit and does not reduce the raw events the search scans. It is tempting because acceleration consumes indexer disk and build resources, so it would be the right lever when acceleration build load, not result count, is the bottleneck.

Use the | table command at the start

Why it fails: Placing | table at the start does not reduce indexer load; it merely reshapes fields, and early in the pipeline it can strip fields later commands need. The indexer still scans and returns every matching event. | table suits final output formatting, not load reduction.

The indexer has too much memory

Why it fails: Indexer memory does not affect scanned_count; it governs search-time processing, not how many events a search reads from buckets. High scanned_count with a small time range points to inefficient search terms or missing index-time filtering. Memory tuning addresses indexer throughput and search head concurrency, not scan volume.

Data model acceleration is failing

Why it fails: Data model acceleration speeds searches against accelerated models, but it does not raise scanned_count; a failing acceleration would leave the search running against raw events. It is tempting because acceleration failures slow searches, yet the high scanned_count points to raw-event scanning instead.

The search head is underpowered

Why it fails: An underpowered search head slows every search uniformly, whereas the inspector shows a specific high scanned_count against a small time range. It is tempting because hardware limits do cause slowness, but the scan volume indicates the search itself is reading excessive raw data.

max_site_copies

Why it fails: max_site_copies is not a valid Splunk indexer cluster attribute; site replication is set by site_replication_factor. It is tempting as a plausible-sounding name, and would be correct only if such a setting existed for capping per-site copies, which it does not.

site_data_count

Why it fails: site_data_count is not a valid indexer cluster setting; replication within a site is governed by site_replication_factor, with total copies set by replication_factor. It is tempting because its name suggests per-site copy control, but no such attribute exists in the cluster configuration.

cross_site_replication

Why it fails: cross_site_replication sets how many copies travel between sites, not how many reside within each site. It is tempting because it governs replication topology in multi-site clusters, and would be correct when you need to guarantee a copy survives the loss of an entire site.

Monitoring Console Search Activity Dashboard

Why it fails: The Search Activity Dashboard reports concurrent users, search counts and durations, but not the index_time and search_time split per search that exposes disk-heavy retrieval. It is tempting because it is the natural place to review overall search load, and would be correct when identifying which users or apps are consuming the most search capacity.

btool

Why it fails: btool inspects and modifies Splunk configuration files and endpoints; it reports nothing about index_time or search_time. It is genuinely useful for troubleshooting configuration and deployment issues, which makes it tempting, but search performance analysis uses the job inspector or search job properties.

splunkd log analysis

Why it fails: splunkd.log records internal daemon events, not per-search index_time versus search_time breakdowns, so it cannot reveal excessive disk retrieval. It is tempting because splunkd.log is the standard place to diagnose licence violations, crashed searches and configuration errors, and would be the right choice when troubleshooting why Splunk itself is misbehaving.

It requires data to be fully ingested before searching

Why it fails: Join operates on already-indexed events at search time and imposes no ingestion prerequisite; data can be searched as it arrives. Its real resource cost is memory and CPU consumed by the subsearch and matching. Join suits correlating two searches on a common field.

It causes high disk I/O on the indexer

Why it fails: The join command's resource cost is memory and CPU, not indexer disk I/O; it builds the subsearch results in memory on the search head. Disk I/O describes index-time and search-time field extraction from raw buckets. Join would be chosen when correlating two result sets on a common field.

It disables parallel processing

Why it fails: Join does not disable parallel processing; Splunk still distributes the base search across indexers, and only the subsearch is constrained. Its actual cost is memory and CPU from holding the subsearch results and matching rows. Join remains valid for combining two datasets on a shared field.