SPLK-2002 Hard Questions — 25 Difficult Questions

Question 1

You want to perform a rolling restart of an Indexer Cluster without stopping ingestion. What configuration ensures that indexers remain available?

Accepted Answer

Set 'restart_mode=rolling'.. A rolling restart in an Indexer Cluster is initiated by setting the restart_mode to rolling (e.g., via the CLI 'splunk rolling-restart cluster-peers' or by setting restart_mode=rolling in server.conf on the cluster master). This ensures that indexers are restarted one at a time (or in batches) while maintaining replication and searchability, so ingestion continues without interruption. The cluster master orchestrates the restart, waiting for buckets to be replicated before proceeding.

Answer

Disable the indexer peers.

Answer

Stop the Cluster Master first.

Answer

Restart all indexers at once.

Answer

Set 'replication_factor=1'.

Question 2

If you need to move a bucket from one peer to another in an indexer cluster manually, how should you proceed?

Accepted Answer

Use the Cluster Master to trigger a rebalance. To move a bucket between peers in an indexer cluster, the correct approach is to use the Cluster Master to trigger a rebalance (or a fixup). The CM orchestrates bucket placement according to replication and search factors, and a rebalance redistributes buckets across peers to balance load. Manually moving files bypasses the CM's metadata and corrupts cluster state.

Answer

Edit the bucket.json file on the peer

Answer

Delete the bucket directory from the peer

Answer

Use the 'mv' command on the filesystem

Question 3

A customer is experiencing 'Search peer [peer-name] is not responding' errors during indexer maintenance. Which configuration parameter in server.conf should be tuned to prevent search failures during rolling restarts?

Accepted Answer

restart_timeout. The 'restart_timeout' parameter in the [clustering] stanza controls how long a search head waits for a peer to return before marking it as down during a rolling restart.

Answer

peer_down_timeout

Answer

max_search_time

Answer

heartbeat_timeout

Answer

search_retry_timeout

Question 4

In a multisite cluster, how does the 'site_replication_factor' override the global 'replication_factor'?

Accepted Answer

It defines the total number of copies per site.. In a Splunk multisite cluster, the global replication_factor sets the total number of copies across all sites, but site_replication_factor overrides this by specifying exactly how many copies must exist per site. For example, site_replication_factor = origin:2, total:3 means each site must have at least 2 copies (origin site) and the cluster must maintain 3 total copies. This ensures data redundancy and availability even if an entire site fails, as each site independently satisfies its replication requirement.

Answer

It only applies to the primary site.

Answer

It ignores the search factor.

Answer

It is used for load balancing.

Answer

It is only used when the CM is down.

Question 5

You are investigating a search issue where results are inconsistent across SHC members. What is a possible cause?

Accepted Answer

Different indexes.conf versions on SHC members.. In a Search Head Cluster (SHC), all members must have identical configuration, including indexes.conf, to ensure consistent search results. If indexes.conf differs across members, some search heads may reference indexes or retention settings that others do not, producing inconsistent results. This is a classic SHC configuration drift issue.

Answer

Indexer peers are in maintenance mode.

Answer

The forwarder is sending bad data.

Answer

The license master is unreachable.

Answer

The Cluster Master is too busy.

Question 6

If a search head is overloaded with concurrent searches, what is the best strategy to offload the processing?

Accepted Answer

Implement a Search Head Cluster. A Search Head Cluster distributes search workload across multiple search head members, allowing concurrent searches to be load-balanced and providing horizontal scaling for search processing. This directly addresses an overloaded single search head by adding capacity and redundancy. Increasing indexers or indexer memory addresses indexing/search execution at the indexer layer, not the search head's concurrency bottleneck.

Answer

Increase the number of indexers

Answer

Reduce the number of users

Answer

Increase indexer memory

Question 7

You are troubleshooting a KV store issue in an Indexer Cluster. Which command allows you to verify the status of the KV store and its replication on a node?

Accepted Answer

splunk show kvstore-status. The command 'splunk show kvstore-status' is the supported CLI command to display the KV Store status on a Splunk instance, including replication status, port, and storage engine details. It works on indexers, search heads, and cluster members, making it the correct tool for verifying KV Store health and replication in an Indexer Cluster.

Answer

mongostat

Answer

check_kvstore.py

Answer

splunk status kvstore

Question 8

In an indexer cluster, why might you use the 'splunk list cluster-peers' command?

Accepted Answer

To verify the synchronization status of all indexer peers. The 'splunk list cluster-peers' command queries the cluster master and returns the status of all peer nodes, including whether each peer is Up, Down, or in a degraded state, along with its replication and searchability status. It is the standard CLI tool for verifying peer synchronization and health in an indexer cluster. It does not modify cluster state; it is read-only diagnostic output.

Answer

To remove a node from the cluster

Answer

To update the license configuration

Answer

To force a cluster rebalance

Question 9

In a SmartStore architecture, what happens when a bucket is evicted from local cache?

Accepted Answer

The local disk space is freed. In SmartStore, buckets are cached locally on indexers for search performance, but when the cache fills, the cache manager evicts buckets back to remote storage (S3, Azure Blob, GCS). Eviction frees the local disk space occupied by that bucket's data, allowing new buckets to be cached. The bucket remains fully searchable because it can be re-fetched from remote storage on demand.

Answer

The bucket is moved to the cold path

Answer

The bucket is no longer searchable

Answer

The bucket is deleted from remote storage

Question 10

What is the primary function of the 'search_factor' in an indexer cluster?

Accepted Answer

The number of searchable copies of each bucket. The search_factor in an indexer cluster determines how many searchable copies of each bucket exist across the cluster, i.e., the number of peers that hold a searchable (indexed) copy of the data. It is set on the cluster master and must be less than or equal to the replication factor. This directly affects search performance and availability because searches can be distributed across more peers.

Answer

The maximum number of concurrent searches

Answer

The number of indexer peers in the cluster

Answer

The number of search heads allowed in the cluster

Question 11

When designing a large-scale multisite cluster, which factor most significantly impacts the search performance when running searches across all sites?

Accepted Answer

Network latency between sites. In a multisite indexer cluster, search heads must dispatch search requests to indexers in all sites and then merge results back. When a search spans all sites, the search head sends the search to peer nodes in every site, and results must traverse the WAN between sites. Network latency between sites directly adds to the round-trip time for each search phase (map and reduce), making it the dominant factor affecting cross-site search performance.

Answer

The number of buckets per indexer

Answer

The number of users

Answer

The number of search heads

Question 12

You are managing a large Indexer Cluster and need to reduce the time it takes for a new node to catch up. Which setting allows you to limit the amount of bandwidth used during bucket replication?

Accepted Answer

replication_throttle. The 'replication_throttling' setting in server.conf allows for control over the network bandwidth used for bucket replication, preventing saturation.

Answer

cluster_bandwidth_control

Answer

bandwidth_limit

Answer

max_replication_speed

Question 13

A Search Head Cluster (SHC) is experiencing frequent 'captain election' issues during high-load periods. Upon reviewing the logs, you notice network latency spikes between nodes. Which parameter in the SHC configuration should be adjusted to allow for more tolerance in node communication?

Accepted Answer

heartbeat_timeout. In a Search Head Cluster, the captain is elected and maintained via heartbeat messages between members. The 'heartbeat_timeout' setting defines how long a member waits before considering another member unreachable. Raising it tolerates longer network latency spikes without triggering unnecessary captain elections.

Answer

election_timeout

Answer

replication_factor

Answer

search_timeout

Question 14

You are troubleshooting a performance issue where cold buckets are being moved to frozen storage too slowly. You find that the indexer is saturated with I/O operations. Which setting can be used to throttle the migration of cold buckets to frozen to improve system responsiveness?

Accepted Answer

coldToFrozenScript. There is no native 'throttle' setting for bucket freezing; however, the correct architectural approach for bucket lifecycle management often involves offloading the freezing process or managing throughput through indexer settings.

Answer

maxHotBuckets

Answer

rotatePeriodInSecs

Answer

maxTotalDataSizeMB

Question 15

Which setting in server.conf on the Cluster Master prevents buckets from being replicated to peers that have been offline for a specific amount of time?

Accepted Answer

rep_max_offline_time. The 'stale_bucket_threshold' or similar replication settings help ensure that the cluster does not attempt to replicate data to long-dead nodes.

Answer

max_peer_offline_time

Answer

bucket_expiry_time

Answer

cluster_peer_timeout

Question 16

In an Indexer Cluster, you want to perform a rolling restart to update the configuration. Which CLI command ensures that the rolling restart happens only after the indexer cluster has reached a state of 'Complete' synchronization?

Accepted Answer

splunk rolling-restart cluster-peers. The 'splunk rolling-restart cluster-peers' command allows for wait-related flags, but specifically, one must ensure the cluster is ready before initiating the rolling process to prevent data loss or search degradation.

Answer

splunk apply cluster-bundle --force

Answer

splunk rolling-restart cluster-peers --check-status

Answer

splunk restart indexer

Question 17

When migrating to a Search Head Cluster, what happens to existing saved searches that were defined in local/savedsearches.conf on a standalone search head?

Accepted Answer

They are ignored by the cluster.. These need to be manually migrated into the 'shcluster' application directory to be recognized by the new cluster.

Answer

They are automatically migrated by the cluster.

Answer

They are deleted by the captain.

Answer

They are replicated to all members automatically.

Answer

They cause a cluster startup error.

Question 18

You are configuring a heavy forwarder to filter data before sending it to indexers. Which stanza in props.conf is used to define the filtering criteria?

Accepted Answer

[sourcetype::*]. Filtering is performed using the 'TRANSFORMS' attribute in props.conf, which points to a stanza in transforms.conf.

Answer

[FILTER]

Answer

[OUTPUT]

Answer

[FORWARDER]

Question 19

You are tuning a Splunk heavy forwarder. The 'parsingQueue' is consistently full. Which configuration change in props.conf or inputs.conf is recommended?

Accepted Answer

Increase the 'maxQueueSize' for the parsing queue in limits.conf. The parsing queue is typically blocked by complex regex transformations or heavy parsing loads. Adjusting pipeline settings or splitting the workload across more CPUs is necessary.

Answer

Reduce the number of indexers in the output group

Answer

Disable 'sourcetype' auto-discovery

Question 20

An indexer is running out of disk space. Which setting in indexes.conf prevents the indexer from crashing by stopping ingestion?

Accepted Answer

maxTotalDataSizeMB. maxTotalDataSizeMB sets a hard size ceiling for an index. When the index reaches this limit, Splunk rolls the oldest buckets to frozen and deletes them (or archives them), freeing disk space so ingestion can continue without the indexer running out of disk and crashing. This is the correct setting to cap index growth and prevent disk exhaustion.

Answer

frozenTimePeriodInSecs

Answer

minFreeSpaceMB

Answer

maxHotBuckets