SPLK-2002 Troubleshooting And Operations — Timed 20 Questions

Question 1

You need to perform a clean upgrade of an indexer in a cluster. You place the indexer into maintenance mode. What happens to the replication and search factor during this time?

Accepted Answer

Bucket fix-up activities are suspended for this peer. When an indexer is placed into maintenance mode, the cluster master suspends bucket fix-up activities for that peer. This means the peer will not participate in re-replication or searchable copies until it is taken out of maintenance mode. This prevents unnecessary replication attempts while the peer is unavailable.

Answer

The Master initiates bucket re-replication immediately

Answer

The search factor is lowered to zero

Answer

Search queries are routed to other peers

Question 2

You are troubleshooting a clustered indexer environment where buckets are stuck in a 'streaming' state. Which log file on the indexer peer should you examine to determine why the replication is failing?

Accepted Answer

replication_data.log. The `replication_data.log` on an indexer peer contains detailed information about bucket replication activities, including errors and reasons for failures. When buckets are stuck in a 'streaming' state, this log will show why replication is not completing, such as network issues or disk problems.

Answer

metrics.log

Answer

cluster_master.log

Answer

splunkd.log

Question 3

You are performing a maintenance task on a Search Head Cluster (SHC). You need to ensure the Captain is not currently running any scheduled reports before you decommission it. Where do you check this?

Accepted Answer

Search Head Clustering: Status dashboard. The Search Head Clustering: Status dashboard in the Monitoring Console shows the current captain, cluster members, replication status, and whether the captain is running scheduled searches. Before decommissioning a captain, you must verify it is not actively running scheduled reports, and this dashboard provides that visibility. The other options do not show captain-specific scheduled search activity.

Answer

Monitoring Console: Search: Activity dashboard

Answer

The 'distsearch.conf' file

Answer

The 'savedsearches.conf' file

Question 4

During a rolling upgrade of an Indexer Cluster, one indexer fails to rejoin the cluster. The Cluster Master indicates the node is 'Down'. Which command should you run on the Indexer Peer to verify if the indexer has successfully joined the cluster manager's configuration secret?

Accepted Answer

splunk show cluster-config. The 'splunk show cluster-config' command on an indexer peer displays the peer's cluster configuration, including the cluster master URI and the encrypted pass4SymmKey (the cluster secret) it is using. If the peer cannot join the master, this command verifies whether the secret matches the master's configuration. It is the correct CLI tool for diagnosing secret mismatch during a failed rejoin.

Answer

splunk validate cluster-config

Answer

splunk list cluster-members

Answer

splunk status

Question 5

You want to troubleshoot why a specific search head is failing to offload scheduled searches. Which log on the Search Head should you review for delegation errors?

Accepted Answer

scheduler.log. The scheduler.log on a search head records the scheduling, dispatching, and execution status of scheduled searches, including errors related to search head delegation to indexers or search peers. Delegation errors — where the search head cannot offload or distribute a scheduled search — are logged here. This is the primary log for troubleshooting scheduled search failures.

Answer

searchservice.log

Answer

dispatch.log

Answer

audit.log

Question 6

During an upgrade, you need to verify that all indexers are running the same version of the Splunk software. What is the most efficient way to check this across the entire cluster using the CLI?

Accepted Answer

splunk show cluster-status. 'splunk show cluster-status' on the Cluster Master displays the status of all peers, including their Splunk version, replication status, and bucket counts. This single command gives a cluster-wide view of versions, making it the most efficient CLI method to verify version uniformity across all indexers.

Answer

splunk btool server list

Answer

splunk list forwarders

Answer

Check /etc/splunk.version on each node

Question 7

You have a deployment server managing multiple forwarder classes. You need to update a configuration file for only one specific server class. Which directory on the Deployment Server should you update?

Accepted Answer

$SPLUNK_HOME/etc/deployment-apps//local. On a Deployment Server, deployment apps are stored under $SPLUNK_HOME/etc/deployment-apps/. To target a specific server class, you place the app in a subdirectory named after the server class (e.g., $SPLUNK_HOME/etc/deployment-apps//local). The Deployment Server then pushes that app only to forwarders mapped to that server class via serverclass.conf.

Answer

$SPLUNK_HOME/etc/slave-apps

Answer

$SPLUNK_HOME/etc/system/local

Answer

$SPLUNK_HOME/etc/apps

Question 8

When troubleshooting a failed App deployment, which log on the Deployment Client should you check to confirm that the app was downloaded and extracted?

Accepted Answer

splunkd.log. On a Deployment Client, the splunkd.log records the deployment client's communication with the deployment server, including the download and extraction of apps from the deployment-apps directory. Confirming that an app was downloaded and extracted requires checking splunkd.log for messages from the DeploymentClient and ApplicationDeployment components. This is the authoritative log for deployment client activity.

Answer

app_install.log

Answer

deploymentclient.log

Answer

web_service.log

Question 9

A user cannot log in to the Splunk instance. You suspect authentication is failing. Which log should you check to see the authentication attempt details?

Accepted Answer

audit.log. The audit.log file in Splunk records authentication and authorization events, including successful and failed login attempts, user actions, and session activity. When a user cannot log in, audit.log is the authoritative source for seeing the exact authentication attempt details such as the username, source IP, and failure reason. This makes it the correct log to inspect for login troubleshooting.

Answer

splunkd.log

Answer

web_access.log

Answer

introspection.log

Question 10

A cluster master is reporting that an indexer is out of disk space. After you add more disk space, what is the best way to force the indexer to re-check its storage and clear the error?

Accepted Answer

Restart the splunkd service. Restarting the splunkd service on the indexer forces it to re-read its disk configuration and re-evaluate available storage, which clears the disk-space error reported to the cluster master. The cluster master receives updated status via the indexer's periodic heartbeats, but a restart triggers an immediate re-check rather than waiting for the next cycle. This is the documented remediation after expanding storage on an indexer.

Answer

Run 'splunk cluster-master-refresh'

Answer

Wait for the hourly heartbeat

Answer

Run 'splunk reload deploy-server'

Question 11

You have a search head that is part of a cluster. You want to see the current list of peers it is communicating with. Which command is most appropriate?

Accepted Answer

splunk show shcluster-status. The command `splunk show shcluster-status` displays the status of a search head cluster, including the list of members (peers) and their current state, captain, and replication status. This is the correct command to see which peers a search head cluster member is communicating with. It is part of the search head clustering CLI and is run on any cluster member.

Answer

splunk status

Answer

splunk show cluster-status

Answer

splunk list search-peers

Question 12

Where do you check the current status of the license usage in the Monitoring Console?

Accepted Answer

Monitoring Console: Licensing. The Splunk Monitoring Console has a dedicated Licensing menu under the Distributed Management Console that shows license usage, quota, warnings, and per-indexer consumption in real time. This is the authoritative place to check current license status across a distributed deployment. Server Settings and the other Monitoring Console panels do not surface license pool consumption.

Answer

Settings: Server Settings

Answer

Monitoring Console: Indexer Status

Answer

Monitoring Console: Resource Usage

Question 13

You are managing a large indexer cluster and notice that search performance is inconsistent. Which TWO of the following items should you investigate to troubleshoot search distribution issues?

Accepted Answer

The contents of distsearch.conf on the search head. Option A, the contents of distsearch.conf on the search head, is correct because this file defines the search head's distributed search configuration, including the list of search peers (servers and their ports) and the shared key used to authenticate to them; if peers are missing, misaddressed, or keys mismatch, searches will be distributed unevenly or fail to reach some peers, causing inconsistent performance. Option B, the 'max_search_peers' setting in limits.conf, is correct because this setting caps how many peers a search head can dispatch a search to, so if it is lower than the number of available peers, some peers are excluded and search load is not evenly distributed across the cluster. Option C, indexing throughput on each peer, relates to ingest and indexing load rather than how searches are distributed across peers, so it does not directly explain search distribution issues. Option D, the license master logs, concerns license usage and violations, not search peer selection or dispatch. Option E, the deployment server logs, concerns configuration deployment to clients, not the search head's distribution of searches to indexers.

Answer

The indexing throughput on each peer

Answer

The license master logs

Answer

The deployment server logs

Question 14

You are tasked with reviewing the health of a Search Head Cluster. Which TWO of the following dashboard locations in the Monitoring Console would be most helpful for this?

Accepted Answer

Search Head Clustering: Status. Option D, Search Head Clustering: Status, is correct because the Monitoring Console's Search Head Clustering dashboards are purpose-built to show the health of an SHC, including captain election, member status, replication of the configuration and artifacts, and quorum, which is exactly what a health review of a Search Head Cluster requires. Option E, Search: Activity, is correct because it surfaces search-related health and load across the SHC, such as search concurrency, skipped searches, and search performance per member, which are key indicators of whether the cluster is functioning well. Option A, Cluster Status, refers to indexer clustering health rather than search head clustering, so it does not address the SHC scenario. Option B, Deployment: Server Classes, is about deployment server server-class configuration and is unrelated to SHC health. Option C, Indexing Performance: Instance, reports indexer-side indexing throughput and latency, not search head cluster health.

Answer

Cluster Status

Answer

Deployment: Server Classes

Answer

Indexing Performance: Instance

Question 15

When troubleshooting a failed rolling upgrade of an Indexer Cluster, which TWO of the following log files on the Cluster Master will provide the most relevant information regarding the peer's failure to return to the cluster?

Accepted Answer

splunkd.log. Option D, splunkd.log, is correct because it is the primary Splunk daemon log on the Cluster Master and records the cluster-related activity of the master, including peer registration, heartbeats, bucket replication, and searchability status, so a peer failing to rejoin after a rolling upgrade will surface errors such as heartbeat timeouts or replication failures here. Option E, cluster_master.log, is correct because it is the dedicated log for Cluster Master clustering activity, capturing peer join/leave events, generation changes, fixup tasks, and peer status transitions, which directly show why a peer did not return to the cluster. Option A, web_access.log, is not relevant because it only records Splunk Web HTTP access requests and has nothing to do with indexer cluster peer membership. Option B, audit.log, is not relevant because it tracks user authentication, authorization, and configuration-change auditing rather than cluster peer health. Option C, metrics.log, is not relevant because it contains periodic performance and throughput metrics, not the peer join/failure diagnostics needed for this troubleshooting scenario.

Answer

web_access.log

Answer

audit.log

Answer

metrics.log

Question 16

You have a cluster where buckets are not replicating as expected. Which THREE of the following factors should you verify to troubleshoot the replication failure?

Accepted Answer

The Cluster Master status shows the peer as 'Ready'. Option B is correct because the Cluster Master (master node) must show each peer as 'Ready' in its peer status; a peer in 'Down', 'Pending', or 'Detention' state cannot participate in bucket replication, so verifying Ready status confirms the peer is eligible to receive replicated buckets. Option D is correct because replication traffic flows over the replication port (default 8080, configured via replication_port in server.conf) between peers, so if that port is blocked by a firewall or unreachable, buckets will fail to replicate even when peers are otherwise healthy. Option E is correct because a bucket in 'streaming' state is still actively being written to and is not yet a finalized, replicable bucket; only non-streaming (hot/warm) buckets are candidates for replication, so confirming the bucket is not streaming ensures it is in a state that should replicate. Option A is incorrect because maxHotBuckets in indexes.conf governs how many hot buckets an indexer keeps per index and does not control inter-peer bucket replication. Option C is incorrect because the Deployment Server distributes configuration/apps to forwarders and has no role in indexer cluster bucket replication, which is managed by the Cluster Master and peer replication ports.

Answer

The 'maxHotBuckets' setting in indexes.conf

Answer

The Deployment Server connection status

Question 17

You are setting up the Monitoring Console to monitor a distributed environment. Which THREE of the following steps are required to ensure the Monitoring Console is correctly configured?

Accepted Answer

Set up the distributed search group. Option B is correct because the Monitoring Console relies on distributed search to gather data from all Splunk instances, so you must define a distributed search group containing the peers to be monitored. Option C is correct because the deployment server must be added as a monitoring node (typically via the DMC's server roles/settings) so the Monitoring Console can collect its internal logs and metrics. Option E is correct because every instance that should be monitored must be assigned to the appropriate instance group (e.g., indexer, search head, license manager) in the Monitoring Console's configuration for the correct dashboards to populate. Option A is not required because restarting the KV store on every instance is unrelated to Monitoring Console setup and is not a standard configuration step. Option D is not required because the Monitoring Console app is enabled on the search head running the DMC, not on all indexers, and indexers only need to forward internal data via distributed search.

Answer

Restart the KV store on every instance

Answer

Enable the 'Monitoring Console' app on all indexers

Question 18

You are troubleshooting a scenario where the Indexer Cluster is not meeting its 'Search Factor' requirement. Which TWO of the following actions can help you resolve this?

Accepted Answer

Increase the 'search_factor' in the Master's server.conf. Option B is correct because the Search Factor is defined by the 'search_factor' setting in the Master's server.conf ([clustering] stanza); raising it (e.g., from 1 to 2) instructs the cluster to create additional searchable copies of each bucket, directly satisfying a higher Search Factor requirement. Option C is correct because the Cluster Master can only fulfill the Search Factor when enough peers are available and healthy; if peers are not in the 'Ready' state (e.g., they are 'Down', 'Pending', or 'Detention'), the Master cannot allocate or replicate the required searchable copies, so verifying peer readiness is a valid remediation step. Option A is wrong because deleting cold buckets destroys indexed data and does not create searchable copies—it would only worsen data availability. Option D is wrong because restarting the Search Head Cluster affects search heads, not the indexer peers that hold bucket replicas, so it does not address Search Factor compliance. Option E is wrong because 'replication_port' is the port used for peer-to-peer bucket replication, and there is no timeout value on it that governs Search Factor; adjusting it would not increase the number of searchable copies.

Answer

Delete all cold buckets

Answer

Restart the Search Head Cluster

Answer

Increase the 'replication_port' timeout

Question 19

You notice that the Monitoring Console reports that the 'Search Head Captain' has changed. Which TWO of the following logs would contain the most information about this event?

Accepted Answer

metrics.log. Option A (metrics.log) is correct because it records internal Splunk metrics such as the captaincy status of the search head cluster, including which member is currently captain and when that role changes, making it a key source for tracking captain transitions. Option C (splunkd.log) is correct because the Splunk daemon logs detailed messages about search head clustering operations, including captain election, re-election, and membership changes, which directly describe the event reported by the Monitoring Console. Option B (audit.log) is not correct because it primarily records user authentication, authorization, and configuration-change activity rather than internal clustering captaincy events. Option D (web_access.log) is not correct because it logs HTTP/HTTPS access to Splunk Web, not search head cluster captain changes. Option E (scheduler.log) is not correct because it records scheduled search activity and related scheduler events, not captain election or role changes.

Answer

audit.log

Answer

web_access.log

Answer

scheduler.log

Question 20

Which TWO of the following locations in the Monitoring Console can be used to monitor the health and performance of the KV Store?

Accepted Answer

Monitoring Console: Resource Usage: Indexer Instance. Option A (Monitoring Console: Resource Usage: Indexer Instance) is correct because the KV Store runs as a service on indexer peers, and the Resource Usage: Indexer Instance view shows per-instance resource metrics such as CPU, memory, and disk usage that reflect KV Store health and performance. Option C (Monitoring Console: KV Store: Instance) is correct because it is the dedicated Monitoring Console view that reports KV Store-specific status and performance data, including replication status, backup/restore status, and per-instance KV Store metrics. Option B (Licensing) only shows license usage and violations, not KV Store health or performance. Option D (Search: Activity) reports search-related activity such as search counts and durations, which is unrelated to the KV Store. Option E (Indexer Clustering: Status) shows indexer cluster bucket replication and peer status, not KV Store health or performance.

Answer

Monitoring Console: Licensing

Answer

Monitoring Console: Search: Activity

Answer

Monitoring Console: Indexer Clustering: Status