Splunk Enterprise Certified Architect (SPLK-2002) (SPLK-2002) (SPLK-2002) — Questions 1–75

In server.conf, the `[general]` stanza's `parallelIngestionPipelines` setting controls how many ingestion pipelines run per indexer, directly governing CPU parallelism during indexing. Raising it scales throughput across cores, satisfying the stem's indexer performance tuning requirement, while `[clustering]` options manage replication load.

The Monitoring Console communicates with indexer peers through splunkd, which handles inter-instance data and management traffic on the relevant ports. If splunkd is not running or reachable on the remote instance, the MC cannot retrieve that peer's data, so it is checked first.

When the search factor is at risk, the cluster master immediately promotes an existing non-searchable copy to searchable status to restore the search factor. This ensures that the required number of searchable copies is maintained without waiting for full replication. The master selects a peer that has a non-searchable copy of the bucket and instructs it to make that copy searchable. This is a rapid operation compared to replicating a new copy.

The peer must complete the handshake with the CM to be active.

Search head clustering uses an elected captain to coordinate replication and job scheduling. If the captain fails, the remaining members trigger a captain election automatically, restoring the cluster to a healthy state without manual intervention, so the degraded condition is temporary.

Concurrent searches compete with indexing for the same indexer disk bandwidth, so higher search concurrency increases read I/O and can slow bucket operations. This satisfies the stem's factor requirement because search load directly affects the disk I/O demands placed on indexer peers.

The Cluster Master distributes configuration bundles to peer nodes, satisfying the requirement for centralised configuration management across the indexer cluster. It coordinates bundle replication, pushing updated configurations to indexers and triggering restarts when needed. This role is unique to the Cluster Master, distinguishing it from search heads and licence managers.

This ensures configuration redundancy among search heads.

Search head clusters use a quorum-based election process. A majority of members must be reachable for a captain to be elected. With five members, a majority is three. If fewer than three members are reachable, no captain can be elected, and the cluster cannot replicate configurations. This is the most likely cause of the prolonged lack of a captain.

Data model acceleration precomputes and stores summarised results in tsidx files, so searches against the model read compact prebuilt summaries instead of raw events. This avoids repeated full-dataset scans, directly addressing the large-dataset performance constraint.

site_replication_factor defines the per-site copy count in a multi-site indexer cluster, expressed as a comma-separated list matching each site's label. It satisfies the stem's constraint of controlling copies kept per site, unlike replication_factor, which governs total cluster-wide copies regardless of site boundaries.

Reviewing the Search Activity dashboard regularly reveals slow searches, concurrency peaks and resource contention. It satisfies the performance task requirement by surfacing the search-side constraint, enabling tuning of limits and knowledge objects before user experience degrades.

Editing the `site` field in `server.conf` and restarting the indexer causes it to re-register with the cluster manager, which updates the site mapping. This satisfies the stem's requirement that the cluster recognise the new site assignment, since site membership is read from `server.conf` at startup.

Doubling raw volume while keeping the same storage footprint requires halving the retention period, since compressed storage scales linearly with daily ingest. Reducing 30 days to 15 days exactly offsets the doubled daily volume at the unchanged 50% compression ratio, preserving the original total index size.

splunkd.log records the Search Head Cluster member's captain election, replication and heartbeat activity, so connection refusals, port mismatches or secret errors preventing cluster joining appear there. Other logs cover search-time or indexing concerns, not the cluster membership communication the stem describes.

Running splunk validate cluster-bundle on the Cluster Master checks the current bundle for configuration errors before it is pushed to peers. Since a rejected or invalid bundle halts replication, this satisfies the stem's requirement to identify why peers are not receiving the latest configuration.

The Monitoring Console aggregates indexer, search head and forwarder health metrics in one dashboard, letting you correlate resource saturation across the deployment. This directly satisfies the stem's requirement for a UI tool that assists performance troubleshooting.

Disabling the server class stops the Deployment Server from pushing the faulty app to indexers, halting further errors without deleting the app definition. This directly satisfies the stem's requirement to rectify ongoing indexer errors caused by that specific app.

This configuration ensures that each bucket has one copy at its origin site and a total of three copies across all sites, meaning one copy in each of the three sites. The search factor of origin:1, total:2 ensures that at least one searchable copy exists at the origin site and a total of two searchable copies across sites, providing high availability for searches.

The Monitoring Console gathers data by running searches against distributed search peers. If indexers are missing from the Monitoring Console's search peer list, its dashboards cannot retrieve their data. Confirming and adding the missing indexers as search peers directly restores visibility into those indexers.

The cluster master distributes configuration bundles to peer indexers from $SPLUNK_HOME/etc/master-apps, making it the authoritative source. Any lookup managed centrally is replicated from here, so version discrepancies across indexers are resolved by comparing against this directory.

100GB daily compressed data over 30 days gives 3TB, then the 20% overhead for bucket metadata raises this to 3.6TB. This satisfies the stem's requirement to include overhead alongside the 1:10 compression factor, yielding the correct total storage capacity for the indexer.

The splunk apply shcluster-bundle command pushes a configuration bundle from the deployer to all cluster members, restarting search heads as needed. This is the supported mechanism for distributing configuration changes across a Search Head Cluster, satisfying the stem's requirement.

The Cluster Master distributes configuration bundles to peer nodes and coordinates the cluster's replication and search factors. It is the central management and communication hub, distinct from indexers that store data and search heads that handle queries.

The Dispatch directory holds the artefacts a search head generates while running a search, including temporary results and metadata such as the search job's configuration and status. It is not the indexer's rawdata store, which persists indexed events.

Indexer peers hold replicated buckets and continue indexing incoming data independently; the master only coordinates replication and search head functions. A brief master outage therefore does not halt ingestion, satisfying the scenario's short-duration unreachability constraint.

The Cluster Master coordinates bucket replication across peer indexers, tracking which buckets exist, their searchable and replicated states, and triggering fixes when copies fall short. This registry function is what maintains the replication factor the cluster promises.

When frozenTimePeriodInSecs elapses, the bucket rolls past the frozen state; Splunk then deletes it, or moves it to the cold storage path if coldToFrozenDir is configured. This satisfies the stem's condition that the bucket's frozen time period has been reached.

The Indexer: Instance dashboard in the Monitoring Console reports per-indexer throughput, latency, queue size and skipped or failed events, directly verifying indexer health and performance. Other dashboards cover search heads, licence usage or cluster-wide resource consumption rather than individual indexer instances.

The cluster master coordinates replication and peer management, so upgrading it first ensures it can manage peers running the newer version. Upgrading peers before the master risks version incompatibility and search degradation during the rolling upgrade.

Very short 'maxHotSpanSecs' values force frequent bucket rollover, so the indexer must create and track many more hot buckets. Each bucket carries its own metadata, and this proliferation increases metadata overhead on the indexer, directly satisfying the architectural trade-off the stem asks about.

Search Inspector breaks down a search's execution into phases, exposing index_time and search_time per operator. High search_time relative to index_time reveals excessive disk retrieval, directly satisfying the stem's requirement to diagnose whether too much data is being read from disk.

SmartStore uses object storage as the remote tier for warm and cold buckets, while indexers cache hot buckets locally. It satisfies the constraint of cost-effective, scalable retention for indexer data without relying solely on attached disks.

Eviction removes the bucket's cached copy from the indexer's local disk, reclaiming that capacity for other buckets. The bucket's data remains available in remote storage, so only local disk space is freed, satisfying the cache-eviction behaviour described.

This setting caps the combined on-disk footprint of all hot and warm buckets stored under homePath, forcing older data to roll to cold or frozen storage once the threshold is reached. It directly satisfies the stem's constraint by bounding homePath growth rather than per-index or per-bucket size.

A low replication_factor means fewer searchable copies exist across indexers, so a single indexer failure can leave some buckets with no surviving replica, causing permanent loss of that indexed data. This directly satisfies the stem's data-loss risk, since replication is the mechanism protecting against indexer failure.

The Cluster Master (manager node) coordinates peer replication and orchestrates fix-up when a bucket's searchable copies fall below the Search Factor. It identifies which peer should receive the missing searchable copy and triggers the copy, restoring the Search Factor. Indexers hold the data but do not orchestrate this remediation themselves.

The Cluster Master (now manager) coordinates the indexer cluster: it controls bucket replication and search factor, and tells peers which buckets to hold. This satisfies the stem's requirement for the component that assigns buckets, not the peers or search heads themselves.

For a three-member search head cluster, a quorum is a majority, which is two members. If the captain fails, the remaining two members can still form a quorum and elect a new captain. Therefore, at least two members must be operational and able to communicate for captain election to succeed. This ensures high availability and continued operation.

A high scanned_count alongside high event_count means Splunk read far more raw data than the time range alone implies, because the filter is not an indexed field. Without an indexed term narrowing the search at index time, Splunk must scan and filter events at search time, slowing results.

The captain is elected from the cluster members and uniquely handles cluster membership changes, configuration replication and coordination of search artefact replication. Other members run searches and hold replicated knowledge objects but do not perform these central management duties.

Bucket replication is a master-level function: the master coordinates replication of buckets across peer indexers, instructing peers to copy and search replicated data. This satisfies the stem's requirement for a genuine Indexer Cluster Master feature, distinct from search head or peer responsibilities.

Captain elections trigger when members cannot exchange heartbeats reliably. Network latency or instability between search head cluster members causes missed heartbeats, forcing repeated elections. Stable, low-latency connectivity is required for the captain to maintain leadership.

The join command builds the entire subsearch result set in memory on the search head before matching, so large datasets exhaust RAM and can trigger search failures. This memory-bound behaviour is its primary resource drawback compared with streaming alternatives such as stats or lookup.

Both search and indexing read and write to the same underlying storage, so heavy indexing can saturate disk throughput and degrade search latency. Disk I/O contention is therefore a factor that must be considered when balancing resource allocation between the two.

Frozen buckets are the archived, read-only index data moved to cheaper storage. Keeping copies off-site protects that indexed data from site-wide loss such as fire or flood, directly satisfying the disaster recovery plan's need for recoverable data at a separate location.

Corrupted bucket files halt the indexer's replication and searchability recovery, so the cluster repeatedly attempts to repair the same buckets, producing repair loops. The lifecycle constraint is that buckets must reach a searchable, replicated state; persistent corruption prevents that transition, driving endless repair cycles.

Bucket rolling closes the current hot bucket and opens a new one, so all incoming writes target a single active bucket. This isolates write I/O from the many older, read-only buckets, reducing disk contention and improving indexing throughput.

The cluster secret is the shared credential that authenticates inter-node traffic, so every peer and indexer must present it before joining replication or search traffic. It therefore secures node-to-node communication across the cluster, satisfying the stem's requirement for the secret's primary function.

Running 'splunk show cluster-status' on a cluster member returns replication and search-factor compliance, peer counts and any ongoing fixup. It satisfies the stem by giving a supported CLI method to monitor indexer cluster health directly from a node.

maxDataSize sets the size threshold for a hot bucket; once the bucket reaches that volume, Splunk rolls it to warm. This satisfies the stem's question by naming the size-based condition that triggers the hot-to-warm transition.

On the cluster manager, `splunk list cluster-fixup` displays the current fixup tasks, including which buckets are being replicated or searched to meet the replication and search factors. This directly answers whether the manager is actively fixing the buckets that show a search factor of 1.

The captain is elected by a majority of SHC members using a consensus protocol. If the captain becomes unavailable, the remaining members detect the failure and hold a new election. During this brief period, searches can still be dispatched because search heads operate independently for search execution; only configuration replication is paused until a new captain is elected.

The replication port is configured in server.conf, under the `[replication_port://<port>]` stanza, which defines the listening port used for indexer-to-indexer replication traffic. Setting this stanza correctly on each peer node satisfies the stem's requirement to locate where the replication port is defined.

The search_factor of 2 directly determines how many searchable copies of each bucket the cluster maintains, satisfying the stem's stated search_factor constraint. Replication_factor governs total copies (3), not searchable ones, so it is irrelevant here. Microsoft Entra ID plays no role in Splunk indexer replication.

Each index on an indexer consumes file handles and search resources. A very high index count therefore increases file handle usage and can degrade search performance, as the indexer must manage and search across many separate index buckets.

Frozen buckets are not directly searchable, so the archived files must first be copied into the thawedPath directory, then the rebuild command regenerates the index entries. This restores searchability, satisfying the stem's requirement to make frozen data searchable again.

When a peer goes offline, the cluster master marks the buckets that were on that peer as under-replicated. The cluster enters a degraded state if the number of available copies falls below the replication factor. While peer2 was offline, other peers may have been unable to maintain three copies. Upon rejoin, the master initiates replication to bring the cluster back to a complete state, so it remains degraded until replication finishes.

Daily ingestion volume directly determines how much raw data lands in the index each day, and Splunk's compression and retention policies scale from that figure. It is the primary input for sizing calculations, whereas licence usage, user counts or search concurrency affect performance rather than storage capacity.

With SF=2 and RF=3, the cluster should maintain two searchable copies of each bucket. The Monitoring Console indicates missing searchable copies, meaning some buckets have fewer than two searchable copies. This can cause incomplete search results if the search head cannot access all bucket copies. The cluster master automatically initiates fix-up, but manual intervention may be needed if fix-up is stalled.

The tstats command reads indexed fields directly from TSIDX files instead of raw events, bypassing the slower event-retrieval and field-extraction stages. This satisfies the speed requirement by querying pre-computed index structures, making aggregations over large datasets substantially faster.

A mock upgrade in staging exercises the full upgrade path, including app compatibility and configuration migrations, without risking production. It satisfies the stem's system-health constraint by exposing failures, rollback gaps and performance regressions before the change window opens on live indexers.

The Indexes and Volumes: Instance dashboard breaks down disk consumption per index on a specific indexer, letting you pinpoint the largest consumer. This directly satisfies the need to identify which index is driving the disk usage warning.

Adding nodes distributes search workload across more peers, so concurrent searches and heavy jobs run in parallel rather than queuing on one search head. This directly addresses the stem's performance constraint by increasing aggregate search capacity.

The search_factor sets how many searchable copies of each bucket the cluster maintains, separate from the replication_factor that governs total copies. Raising it increases search capacity and resilience, satisfying the stem's requirement to define searchable replica count across peer nodes.

Excessive search load starves the captain heartbeat, so the member's periodic captain communication times out and it drops out of the cluster. This satisfies the stem's timeout constraint directly: resource contention, not network failure, breaks captain contact.

Running 'splunk offline' triggers graceful shutdown, letting the peer flush buckets and transfer primary copies to other peers before leaving the cluster. This satisfies the stem's requirement for handling a permanently failing peer, since the cluster rebalances replication and search factor without data loss.

Indexer peers store the bucket index files, including rawdata journals, tsidx files and bloom filters, on their local disk. This satisfies the stem's constraint that the component holding bucket data must be the peer itself, not the cluster manager, which only coordinates replication and searchability across peers.

Cluster members authenticate replication traffic using the shared pass4SymmKey secret. A mismatch on any single node prevents that member from joining the replication channel, so configuration bundles never propagate, directly producing the search head cluster replication failures the stem describes.

The local directory is specifically ignored by the SHC replication mechanism.

A 'Missing' node in Search Head Clustering means the captain has stopped receiving that peer's heartbeat, so the node is presumed down or unreachable. This directly satisfies the dashboard's status check, which relies on periodic heartbeats rather than replication or search artefact checks.

coldPath is the directory storing buckets that have rolled from warm to cold, holding older searchable data on cheaper storage. Together with homePath and thawedPath it forms the three configurable index path components, directly answering which parts constitute the Splunk index directory structure.

Adding an indexer triggers bucket redistribution solely when replication or search factors are unmet, not automatically. Existing buckets stay put unless the cluster needs to satisfy those factor requirements across peers. This preserves historical data placement while maintaining availability, matching the stem's scenario of a newly added indexer.

A filling parsing queue shows that incoming raw data exceeds the parsing pipeline's throughput, typically from heavy props, transforms or regex. Monitoring this percentage pinpoints the ingestion stage causing the slowdown, satisfying the troubleshooting requirement to locate the constrained queue.

Upgrading indexer peers sequentially, one at a time, allows each peer to rejoin and the cluster to regain full searchability and replication health before the next is taken offline. This preserves data availability and avoids multiple simultaneous outages.