Hard Difficulty Questions

Disable the indexer peers.

Why it fails: Disabling peers removes them from the cluster.

Stop the Cluster Master first.

Why it fails: Stopping the CM stops all cluster management.

Restart all indexers at once.

Why it fails: This causes a complete outage.

Set 'replication_factor=1'.

Why it fails: This reduces redundancy and increases risk.

Edit the bucket.json file on the peer

Why it fails: This is not a supported method.

Delete the bucket directory from the peer

Why it fails: This causes data loss.

Use the 'mv' command on the filesystem

Why it fails: This corrupts the cluster's bucket map.

peer_down_timeout

Why it fails: peer_down_timeout governs how long a search head waits before declaring a peer down; raising it delays the failure rather than preventing it during rolling restarts. It tempts because it directly references peer availability, and would be right when transient network latency, not maintenance restarts, causes peers to drop.

max_search_time

Why it fails: This limits individual search job duration.

heartbeat_timeout

Why it fails: This controls communication between indexers and the master node.

search_retry_timeout

Why it fails: This is not a valid parameter.

It only applies to the primary site.

Why it fails: It applies to all sites defined.

It ignores the search factor.

Why it fails: Search factor still applies across the whole cluster.

It is used for load balancing.

Why it fails: It is for redundancy, not load balancing.

It is only used when the CM is down.

Why it fails: It is used at all times.

Indexer peers are in maintenance mode.

Why it fails: This would affect availability, not consistency.

The forwarder is sending bad data.

Why it fails: This would affect all search results equally.

The license master is unreachable.

Why it fails: This stops indexing or searching globally.

The Cluster Master is too busy.

Why it fails: CM has no role in search result consistency.

Increase the number of indexers

Why it fails: Adding indexers expands indexer-side search capacity but the search head still runs the search-time processing, so it cannot offload that work. It is tempting because scaling indexers is the standard remedy for slow searches, and it would be correct if the bottleneck were indexer CPU or search parallelism rather than the search head.

Reduce the number of users

Why it fails: Reducing users lowers demand but does not offload processing; the search head still executes every search locally. It is tempting because cutting concurrent load relieves an overloaded search head, and it would be the right response if the overload came from too many users rather than insufficient distributed search capacity.

Increase indexer memory

Why it fails: Indexer memory affects indexer-side performance, not the search head's concurrent search load, so the overload persists. It is tempting because memory tuning is a common Splunk scaling lever, and it would be correct if indexers were the constrained tier rather than the search head running the searches.

mongostat

Why it fails: mongostat reports MongoDB operation counters and throughput on a single instance, but it does not show Splunk KV store replication topology or the cluster-level status needed here. It is tempting because the KV store is backed by MongoDB, so MongoDB tooling would be the right choice when diagnosing raw database performance rather than replication health.

check_kvstore.py

Why it fails: check_kvstore.py is not a supported Splunk diagnostic script, so it cannot report KV store or replication status on a node. It is tempting because Splunk ships numerous Python health-check scripts, and a similarly named script would be the correct choice for verifying KV store integrity when one is provided by support.

splunk status kvstore

Why it fails: The CLI command for checking KV store status and replication is splunk show kvstore-status, not splunk status kvstore, which is not a valid command and returns an error. It is tempting because it follows the familiar splunk status pattern used for other component health checks, and that pattern would be correct for verifying general Splunk service status.

To remove a node from the cluster

Why it fails: Listing peers only reports peer nodes and their status; removing a node requires the 'splunk remove cluster-peers' command or decommissioning via the master. Peer listing is used to audit cluster membership and replication health, not to alter it.

To update the license configuration

Why it fails: This command queries peer status and replication information; it cannot modify licence settings, which are configured through licence manager or cluster master configuration. Listing peers is used to verify peer connectivity, replication factor and searchability status during health checks, not to change licensing.

To force a cluster rebalance

Why it fails: Rebalancing is triggered by the master through bucket reallocation settings or 'splunk rebalance cluster-data', not by listing peers. The list command reports peer status, replication counts and searchability, which administrators use to confirm cluster health before deciding whether a rebalance is needed.

The bucket is moved to the cold path

Why it fails: SmartStore eviction removes the local cache copy while the bucket remains in remote storage, still searchable via cache miss retrieval. Moving buckets to the cold path is a separate index-time or lifecycle routing decision based on retention policy, not a consequence of local cache eviction.

The bucket is no longer searchable

Why it fails: Eviction removes only the local cache copy; the bucket's data remains in remote storage and stays searchable, with Splunk re-fetching it on demand. Confusing eviction with deletion is tempting because cache removal sounds destructive, yet eviction is purely a local space-reclamation action, not a searchability change.

The bucket is deleted from remote storage

Why it fails: Eviction frees local disk space only; the authoritative bucket copy persists in the remote object store and is re-fetched when needed. Deletion from remote storage is a separate lifecycle or retention action, so this option is tempting when eviction is misread as permanent removal rather than cache management.

The maximum number of concurrent searches

Why it fails: search_factor sets the replication factor for searchable copies across peers, not a concurrency ceiling; concurrent search limits come from search head or peer capacity settings. It tempts because search concurrency is a genuine cluster tuning concern, and search_factor sounds search-related, but it governs index replication for searchability, not simultaneous query counts.

The number of indexer peers in the cluster

Why it fails: search_factor defines how many copies of each bucket are searchable, expressed as a multiple of the replication factor, not the peer count itself. It tempts because the value interacts with cluster size, but peer membership is set by the replication_factor and the number of joined indexers, not by search_factor.

The number of search heads allowed in the cluster

Why it fails: search_factor controls searchable copy replication, not how many search heads may join the cluster. It tempts because search heads query the cluster and the name implies search capacity, but search head count is limited by licensing and configuration, while search_factor only affects bucket searchability across peers.

The number of buckets per indexer

Why it fails: Bucket count per indexer influences local disk and memory overhead, but a cross-site search's dominant cost is shipping partial results between sites, which bucket counts do not govern. Bucket sizing is the lever when tuning single-site indexer storage and retention, not multisite search fan-out.

The number of users

Why it fails: User count does not govern cross-site search cost; concurrency affects resource contention, not the volume of data each site must scan. It is tempting because more users mean more simultaneous searches, and it would matter for sizing search-head capacity, but the number of indexers and replicated data per site drives performance.

The number of search heads

Why it fails: Search heads only dispatch and merge results; adding them does not reduce the inter-site data transfer that dominates a search spanning all sites. Search-head count is the scaling lever for concurrent user sessions and dashboard load, not for the latency of a single cross-site search.

cluster_bandwidth_control

Why it fails: This is not a valid Splunk cluster parameter.

bandwidth_limit

Why it fails: This is not a valid Splunk cluster parameter.

max_replication_speed

Why it fails: This is not a valid Splunk cluster parameter.

election_timeout

Why it fails: This affects the election duration, not the network tolerance.

replication_factor

Why it fails: This relates to data replication, not node connectivity.

search_timeout

Why it fails: This affects individual search execution, not cluster health.

maxHotBuckets

Why it fails: This controls hot buckets, not the cold-to-frozen lifecycle.

rotatePeriodInSecs

Why it fails: This controls hot bucket rotation, not cold-to-frozen migration.

maxTotalDataSizeMB

Why it fails: This controls when a bucket is frozen, not the speed of the migration.

max_peer_offline_time

Why it fails: This is not the exact parameter name.

bucket_expiry_time

Why it fails: This is for bucket lifecycle management.

cluster_peer_timeout

Why it fails: This is not a valid parameter.

splunk apply cluster-bundle --force

Why it fails: This forces an update but does not manage the rolling restart process.

splunk rolling-restart cluster-peers --check-status

Why it fails: This is not a valid flag for the rolling-restart command.

splunk restart indexer

Why it fails: This restarts a single node, not the cluster.

They are automatically migrated by the cluster.

Why it fails: They must be moved manually to the app bundle.

They are deleted by the captain.

Why it fails: The captain doesn't delete files outside its control.

They are replicated to all members automatically.

Why it fails: The captain only replicates bundles it has pushed.

They cause a cluster startup error.

Why it fails: They are simply ignored.

[FILTER]

Why it fails: This is not a valid stanza name.

[OUTPUT]

Why it fails: This is not a valid configuration stanza.

[FORWARDER]

Why it fails: Not a valid stanza.

Reduce the number of indexers in the output group

Why it fails: This would likely increase the queue backup.

Disable 'sourcetype' auto-discovery

Why it fails: This has negligible impact on parsing queue capacity.

frozenTimePeriodInSecs

Why it fails: This controls data retention, not disk space hard limits.

minFreeSpaceMB

Why it fails: This is for emergency stop, but maxTotalDataSizeMB is the primary management tool.

maxHotBuckets

Why it fails: This limits the number of hot buckets, not total size.