SPLK-2002 Troubleshooting And Operations Practice Test 2 — 15 Questions

Question 1

You have a cluster where buckets are not replicating as expected. Which THREE of the following factors should you verify to troubleshoot the replication failure?

Accepted Answer

The Cluster Master status shows the peer as 'Ready'. Option B is correct because the Cluster Master (master node) must show each peer as 'Ready' in its peer status; a peer in 'Down', 'Pending', or 'Detention' state cannot participate in bucket replication, so verifying Ready status confirms the peer is eligible to receive replicated buckets. Option D is correct because replication traffic flows over the replication port (default 8080, configured via replication_port in server.conf) between peers, so if that port is blocked by a firewall or unreachable, buckets will fail to replicate even when peers are otherwise healthy. Option E is correct because a bucket in 'streaming' state is still actively being written to and is not yet a finalized, replicable bucket; only non-streaming (hot/warm) buckets are candidates for replication, so confirming the bucket is not streaming ensures it is in a state that should replicate. Option A is incorrect because maxHotBuckets in indexes.conf governs how many hot buckets an indexer keeps per index and does not control inter-peer bucket replication. Option C is incorrect because the Deployment Server distributes configuration/apps to forwarders and has no role in indexer cluster bucket replication, which is managed by the Cluster Master and peer replication ports.

Answer

The 'maxHotBuckets' setting in indexes.conf

Answer

The Deployment Server connection status

Question 2

You are setting up the Monitoring Console to monitor a distributed environment. Which THREE of the following steps are required to ensure the Monitoring Console is correctly configured?

Accepted Answer

Set up the distributed search group. Option B is correct because the Monitoring Console relies on distributed search to gather data from all Splunk instances, so you must define a distributed search group containing the peers to be monitored. Option C is correct because the deployment server must be added as a monitoring node (typically via the DMC's server roles/settings) so the Monitoring Console can collect its internal logs and metrics. Option E is correct because every instance that should be monitored must be assigned to the appropriate instance group (e.g., indexer, search head, license manager) in the Monitoring Console's configuration for the correct dashboards to populate. Option A is not required because restarting the KV store on every instance is unrelated to Monitoring Console setup and is not a standard configuration step. Option D is not required because the Monitoring Console app is enabled on the search head running the DMC, not on all indexers, and indexers only need to forward internal data via distributed search.

Answer

Restart the KV store on every instance

Answer

Enable the 'Monitoring Console' app on all indexers

Question 3

You are troubleshooting a scenario where the Indexer Cluster is not meeting its 'Search Factor' requirement. Which TWO of the following actions can help you resolve this?

Accepted Answer

Increase the 'search_factor' in the Master's server.conf. Option B is correct because the Search Factor is defined by the 'search_factor' setting in the Master's server.conf ([clustering] stanza); raising it (e.g., from 1 to 2) instructs the cluster to create additional searchable copies of each bucket, directly satisfying a higher Search Factor requirement. Option C is correct because the Cluster Master can only fulfill the Search Factor when enough peers are available and healthy; if peers are not in the 'Ready' state (e.g., they are 'Down', 'Pending', or 'Detention'), the Master cannot allocate or replicate the required searchable copies, so verifying peer readiness is a valid remediation step. Option A is wrong because deleting cold buckets destroys indexed data and does not create searchable copies—it would only worsen data availability. Option D is wrong because restarting the Search Head Cluster affects search heads, not the indexer peers that hold bucket replicas, so it does not address Search Factor compliance. Option E is wrong because 'replication_port' is the port used for peer-to-peer bucket replication, and there is no timeout value on it that governs Search Factor; adjusting it would not increase the number of searchable copies.

Answer

Delete all cold buckets

Answer

Restart the Search Head Cluster

Answer

Increase the 'replication_port' timeout

Question 4

You notice that the Monitoring Console reports that the 'Search Head Captain' has changed. Which TWO of the following logs would contain the most information about this event?

Accepted Answer

metrics.log. Option A (metrics.log) is correct because it records internal Splunk metrics such as the captaincy status of the search head cluster, including which member is currently captain and when that role changes, making it a key source for tracking captain transitions. Option C (splunkd.log) is correct because the Splunk daemon logs detailed messages about search head clustering operations, including captain election, re-election, and membership changes, which directly describe the event reported by the Monitoring Console. Option B (audit.log) is not correct because it primarily records user authentication, authorization, and configuration-change activity rather than internal clustering captaincy events. Option D (web_access.log) is not correct because it logs HTTP/HTTPS access to Splunk Web, not search head cluster captain changes. Option E (scheduler.log) is not correct because it records scheduled search activity and related scheduler events, not captain election or role changes.

Answer

audit.log

Answer

web_access.log

Answer

scheduler.log

Question 5

Which TWO of the following locations in the Monitoring Console can be used to monitor the health and performance of the KV Store?

Accepted Answer

Monitoring Console: Resource Usage: Indexer Instance. Option A (Monitoring Console: Resource Usage: Indexer Instance) is correct because the KV Store runs as a service on indexer peers, and the Resource Usage: Indexer Instance view shows per-instance resource metrics such as CPU, memory, and disk usage that reflect KV Store health and performance. Option C (Monitoring Console: KV Store: Instance) is correct because it is the dedicated Monitoring Console view that reports KV Store-specific status and performance data, including replication status, backup/restore status, and per-instance KV Store metrics. Option B (Licensing) only shows license usage and violations, not KV Store health or performance. Option D (Search: Activity) reports search-related activity such as search counts and durations, which is unrelated to the KV Store. Option E (Indexer Clustering: Status) shows indexer cluster bucket replication and peer status, not KV Store health or performance.

Answer

Monitoring Console: Licensing

Answer

Monitoring Console: Search: Activity

Answer

Monitoring Console: Indexer Clustering: Status

Question 6

A Splunk Enterprise deployment uses an indexer cluster with a search head cluster. Users report that dashboard panels take much longer than usual, and the Monitoring Console shows that the search head cluster captain has changed three times in the last hour. You need to review the most relevant artifacts to determine why the captain is flapping. (Choose two.)

Accepted Answer

Check the search head cluster members' `server.conf` for consistent `[shclustering]` settings such as `pass4SymmKey`, `replication_factor`, and `heartbeat_timeout`.. Captain flapping in a search head cluster usually points to unstable communication or inconsistent clustering configuration among members. The members' own `splunkd.log` files capture election and heartbeat messages, while the `[shclustering]` stanza in `server.conf` governs the shared secret and heartbeat timing. Together, these reveal whether the problem is network/resource-related or configuration-related.

Answer

Run `splunk show shcluster-status` on the deployer to determine which search head is currently the captain and why the previous captain stepped down.

Answer

Review `$SPLUNK_HOME/var/log/splunk/splunkd_access.log` on the cluster manager to identify which search heads are making REST calls to the manager.

Answer

Examine `$SPLUNK_HOME/var/log/splunk/metrics.log` on the license manager to see whether license violations are triggering captain elections.

Question 7

A Splunk Enterprise deployment has a search head cluster and an indexer cluster. Users report that a saved search scheduled to run every 15 minutes has not produced alerts for the last hour. You need to determine which search head in the search head cluster is currently running the scheduled search. Which command should you run on a search head cluster member?

Accepted Answer

`splunk show shcluster-status`. In a search head cluster, the captain coordinates scheduled search execution by delegating searches to members. `splunk show shcluster-status` on a member reports the captain and member status, allowing you to identify the node responsible for scheduling and then investigate why the search is not producing alerts.

Answer

`splunk show cluster-status`

Answer

`splunk list shcluster-members`

Answer

`splunk list cluster-fixup`

Question 8

You notice that the Monitoring Console (MC) reports a 'Disk usage' warning for an indexer. Which dashboard should you navigate to in order to identify which specific index is consuming the most disk space?

Accepted Answer

Indexes and Volumes: Instance. The Monitoring Console's 'Indexes and Volumes: Instance' dashboard shows per-index disk usage on a specific indexer instance, including the size of each index and its volumes. This is the correct dashboard to identify which index is consuming the most disk space when a Disk usage warning fires. It breaks down usage by index, making the culprit immediately visible.

Answer

Indexing Performance: Instance

Answer

Resource Usage: Indexer Instance

Answer

Cluster Status

Question 9

You are tasked with verifying the health of the indexer cluster. Which dashboard in the Monitoring Console provides a high-level overview of the health of all indexer nodes, including their replication status?

Accepted Answer

Indexer Cluster: Status. The Monitoring Console's 'Indexer Cluster: Status' dashboard is purpose-built to show a high-level health view of all indexer cluster nodes, including peer status, replication and search factor compliance, and bucket replication status. It aggregates cluster health so you can quickly see whether the cluster is fully replicated and whether any peers are down or lagging. The other dashboards focus on performance, resource usage, or distributed search rather than cluster replication health.

Answer

Indexing Performance: Cluster

Answer

Resource Usage: Indexer Cluster

Answer

Distributed Search: Status

Question 10

An Indexer Cluster is experiencing 'Search Factor' violations. Which component is responsible for orchestrating the fix-up process to restore the search factor?

Accepted Answer

The Cluster Master. In an Indexer Cluster, the Cluster Master (now called Cluster Manager in newer versions) is the component responsible for orchestrating fix-up. When a Search Factor violation occurs (fewer searchable copies than the configured search factor), the Cluster Master identifies which buckets need additional searchable copies and directs peers to replicate them, restoring the search factor.

Answer

The Indexer Peer containing the missing copies

Answer

The Search Head

Answer

The Deployment Server

Question 11

You have added a new indexer to a cluster. The indexer is up, but the Cluster Master does not show it as 'Ready'. What is the most likely cause?

Accepted Answer

The security key on the indexer does not match the Cluster Master. For an indexer to join a cluster and reach 'Ready' status, it must present the same cluster security key (pass4SymmKey under the clustering stanza) as the cluster master. If the keys differ, the indexer can start and appear 'Up' but will be rejected during cluster handshake, so the master never marks it Ready.

Answer

The indexer is missing the license file

Answer

The cluster master is in maintenance mode

Answer

The indexer has not indexed any data yet

Question 12

A user reports that they cannot see data from an indexer peer in the Monitoring Console. Which service should you check first to ensure the MC is able to communicate with the remote instance?

Accepted Answer

splunkd. The Monitoring Console (MC) communicates with distributed instances via the splunkd management port (default 8089) to collect health and performance data. If the MC cannot see data from an indexer peer, the first service to verify is splunkd on the remote instance, because splunkd must be running and reachable for the MC to poll it. This makes splunkd the correct first check.

Answer

splunkweb

Answer

mongod

Answer

Deployment Client

Question 13

A Search Head Cluster is experiencing frequent 'Search Head Captain Election' messages in the logs. What is the most likely cause?

Accepted Answer

Network latency or instability between SHC members. Frequent Search Head Captain Election messages indicate that the captain is changing often, which is typically caused by network latency or instability between SHC members. The captain election process relies on consensus and heartbeats; if members cannot communicate reliably, elections are triggered. The other options would cause different symptoms, such as search failures or licensing warnings, but not frequent elections.

Answer

The indexer cluster is offline

Answer

The KV store is running

Answer

The license is expired

Question 14

You are investigating a scenario where search performance has degraded. You observe that many searches are hitting the 'warm' buckets but not the 'cold' buckets. Where should you look to verify bucket rotation policies?

Accepted Answer

indexes.conf. Bucket rotation policies — including the thresholds that move buckets from hot to warm to cold to frozen, and the retention periods for each — are defined per index in indexes.conf. Parameters like maxHotBuckets, maxWarmDBCount, frozenTimePeriodInSecs, and homePath/coldPath control when and where buckets roll. To verify or change rotation behavior, you inspect indexes.conf on the indexers (or via the cluster manager for clustered environments).

Answer

outputs.conf

Answer

inputs.conf

Answer

server.conf

Question 15

You are troubleshooting a KV store issue in an Indexer Cluster. Which command allows you to verify the status of the KV store and its replication on a node?

Accepted Answer

splunk show kvstore-status. The command 'splunk show kvstore-status' is the supported CLI command to display the KV Store status on a Splunk instance, including replication status, port, and storage engine details. It works on indexers, search heads, and cluster members, making it the correct tool for verifying KV Store health and replication in an Indexer Cluster.

Answer

mongostat

Answer

check_kvstore.py

Answer

splunk status kvstore