Hard Difficulty Questions

Only index1.

Why it fails: Splunk unions srchIndexesAllowed across all inherited roles, so the user reaches both index1 and index2. It tempts because a single role's list would restrict access to index1 alone, which applies only when RoleA is the sole role assigned.

Only index2.

Why it fails: Splunk unions srchIndexesAllowed across all inherited roles, so the user reaches both index1 and index2. It tempts because a single role's list would restrict access to index2 alone, which applies only when RoleB is the sole role assigned.

No indexes.

Why it fails: Splunk unions srchIndexesAllowed across inherited roles, granting index1 and index2, so access is not empty. It tempts because conflicting or absent index lists can deny search, but two roles each naming an index produce a combined allow set.

LDAP Bind Password

Why it fails: Used for LDAP, not SAML.

The Splunk license key.

Why it fails: Unrelated to SAML.

The file with the largest size

Why it fails: File size has no impact on precedence.

It randomly chooses one

Why it fails: Splunk is deterministic in its configuration processing.

The file created most recently

Why it fails: Timestamp is not used for precedence tie-breaking.

Delete the bucket folder from the OS.

Why it fails: This corrupts the index and causes errors.

Rename the bucket folder to start with 'frozen'.

Why it fails: Splunk ignores buckets that do not follow the naming convention.

Use the 'splunk edit index' command.

Why it fails: There is no native 'freeze' command.

Query the license manager's REST endpoint /services/licenser/usage and aggregate results by pool name.

Why it fails: The licenser usage endpoint reports pool-level and stack-level usage rather than a per-indexer breakdown, so aggregating by pool name would not identify individual indexers. This approach answers which pool is over quota, not which indexers sent the data, leaving the administrator unable to isolate the specific hosts causing the overage.

Inspect the indexer's metrics.log for per-sourcetype throughput and rank sourcetypes by volume.

Why it fails: metrics.log contains pipeline and throughput metrics but is not organized to attribute license volume to individual indexers across the deployment. Ranking sourcetypes shows what data is heavy, not which indexer reported it, so this method cannot satisfy the requirement to identify the indexers contributing the most volume.

Open the Monitoring Console License Usage view and read the Indexing Volume by Indexer panel for the last 24 hours.

Why it fails: The Monitoring Console License Usage view does present indexer-level volume, but as a visual dashboard it is intended for monitoring rather than extracting a ranked list for investigation. The scenario asks for a searchable breakdown to determine the source, and the underlying license_usage.log search provides that detail directly and reproducibly.

The Deployment Server will ignore forwarders with duplicate serverName values, causing them not to receive configurations.

Why it fails: This is incorrect because duplicate serverName values can cause issues, but the scenario states that the forwarders have different serverName values than their hostnames, not that they have duplicates. The Deployment Server does not ignore forwarders with duplicate names; it may have issues with tracking them, but that is not the cause here. The primary issue is the mismatch with serverclass filters.

The forwarders will not be able to download apps because the serverName is used to determine the app directory path.

Why it fails: This is incorrect because the serverName is not used to determine the app directory path. The app directory path is based on the app name and the Deployment Server's configuration. The serverName is used for identification and matching. The forwarder downloads apps to its own etc/apps directory regardless of its serverName. Therefore, this is not the cause.

The forwarders cannot connect to the Deployment Server because the serverName is used for authentication.

Why it fails: This is incorrect because the serverName is not used for authentication. Forwarders connect to the Deployment Server using the URI specified in deploymentclient.conf. Authentication is not based on serverName. The serverName is used for identification and matching in serverclasses. Even if the serverName is different, the forwarder can still connect and download apps if it matches a serverclass.

The indexer's license status.

Why it fails: Licence status governs whether an indexer will search data at all, not whether a search head cluster member can reach a specific bucket. It tempts because licensing failures do stop indexing. Verify replication and search affinity settings, plus network connectivity, since bucket access depends on those.

The OS version of the search head.

Why it fails: The search head's OS version is irrelevant to reaching an indexer's bucket; cluster members run supported platforms already. It tempts because version mismatches cause odd behaviour. Check the indexer's replication and search affinity configuration and network reachability instead, as those govern bucket accessibility.

The search head's RAM.

Why it fails: RAM exhaustion would stall the search process itself, not block access to a specific indexer's bucket; bucket access depends on replication and searchability settings. Checking RAM is tempting because resource pressure causes many Splunk search failures, and it would be the right first check for slow or crashing searches rather than a bucket-access error.

bindDN

Why it fails: The bindDN is the distinguished name of the user that Splunk uses to bind to the LDAP server for searching. While often used, it is not strictly required if anonymous bind is allowed, and it is not directly involved in mapping groups to roles. The question asks for settings to authenticate users and map groups, so bindDN is not one of the two essential settings.

srvBaseDN

Why it fails: srvBaseDN is not a valid authentication.conf setting for LDAP. It might be confused with server-related settings, but it does not exist. The correct settings for LDAP are userBaseDN and groupBaseDN, among others like bindDN and userBaseFilter.

roleBaseDN

Why it fails: roleBaseDN is not a valid setting in authentication.conf. Role mapping is configured via groupBaseDN and roleMap, not by a roleBaseDN. This distractor might confuse those thinking of a separate base for roles, but roles are Splunk-side and not stored in LDAP.

dataDir

Why it fails: dataDir is a deprecated, read-only attribute reflecting the volume's default path, not the configurable location. It is tempting because the name suggests the data directory, but coldPath, thawedPath and homePath define where index buckets actually reside.

maxTotalDataSizeMB

Why it fails: maxTotalDataSizeMB caps index size and triggers deletion of oldest data; it does not relocate storage. The path is controlled by the homePath (or coldPath) key, which must point at the new partition. Size limits are tempting because they govern disk usage, but they would be the right setting only when constraining retention volume, not moving directories.

indexPool

Why it fails: indexPool is not an indexes.conf setting; it does not control storage paths. The name is tempting because it sounds storage-related, but relocating index data requires updating homePath, coldPath and thawedPath to point at the new partition.

Use the CLI command 'splunk apply cluster-bundle' to redistribute buckets.

Why it fails: The 'apply cluster-bundle' command is used to push configuration bundles to peers, not to manage bucket replication or search factor. It does not affect the distribution of bucket copies. Therefore, it would not help in maintaining the search factor when a peer is down.

Increase the replication factor to 4 to compensate for the lost peer.

Why it fails: Increasing the replication factor changes the cluster's data redundancy requirements and may not be possible if there are insufficient peers. It does not directly address the search factor, which is about searchable copies. The immediate need is to maintain searchability, not to increase replication, which could strain resources and is not the recommended first step.

Manually copy bucket copies from the failed peer to other peers.

Why it fails: Manually copying buckets is not a supported or practical method in a clustered environment. The cluster manager automatically handles bucket replication and searchability. Manual intervention could lead to inconsistencies and is unnecessary because the cluster manager will detect the offline peer and take corrective actions.

outputs.conf

Why it fails: outputs.conf configures data forwarding destinations and output processor settings, not event line breaking. BREAK_ONLY_BEFORE is a props.conf line-merging setting applied by the parsing pipeline, so placing it in outputs.conf has no effect on how multiline events are grouped.

inputs.conf

Why it fails: inputs.conf configures data inputs such as file monitoring, ports and source types, but line breaking is applied later by the parsing pipeline. BREAK_ONLY_BEFORE belongs in props.conf, so setting it in inputs.conf leaves multiline events ungrouped. inputs.conf would be correct for defining a new file monitor stanza.

indexes.conf

Why it fails: indexes.conf defines index storage paths, retention and sizing policies, not event parsing rules. Line breaking is controlled in props.conf, so BREAK_ONLY_BEFORE placed here is ignored. indexes.conf would be correct for configuring index homePath, coldPath or frozenTimePeriodInSecs.

Only the differences from defaults

Why it fails: btool prints the merged effective configuration, including default and system settings, rather than a diff against defaults. Showing only differences from defaults is correct when comparing a modified configuration against a baseline, which btool does not do by default.

The raw file contents

Why it fails: btool outputs the merged, effective configuration after layering, not the literal contents of individual .conf files. Raw file contents are correct when inspecting a single configuration file directly, for example reading inputs.conf on disk, rather than resolving effective settings.

Only the changes made in /local

Why it fails: btool merges configuration across all apps and system directories, printing the effective stanzas, not just /local overrides. /local-only output requires filtering or the --debug flag. Showing only /local changes is correct when troubleshooting which custom settings a specific app has added.

The search head is not configured to search all indexers.

Why it fails: In an indexer cluster, the search head automatically searches all peers that have searchable copies. If the search head is properly connected to the cluster, it will search all available peers. Misconfiguration of the search head could cause issues, but the scenario does not indicate that; the focus is on the downed indexer and search factor.

The indexer that is down was the master node.

Why it fails: The master node manages the cluster but does not hold buckets for search. If the master is down, it affects cluster management but not the searchability of existing buckets, as long as search heads can connect to peers. The scenario states one indexer is down, not the master. Even if it were the master, incomplete search results would not be directly caused by that.

The replication factor is set too low.

Why it fails: Replication factor 2 means each bucket has two copies. If one indexer is down, the other copy should still exist. However, replication factor only ensures data copies, not searchability. If replication factor were 1, data loss could occur, but here it is 2, so this is not the primary cause of incomplete search results.

The browser.

Why it fails: The browser is the user agent relaying SAML requests and responses between the two parties; it holds no SAML role and is never configured as Service Provider. It is tempting because the browser visibly carries the SAML exchange. It would be the correct focus only when troubleshooting browser-side redirect or cookie behaviour.

The Identity Provider (IdP).

Why it fails: The IdP is the asserting party that issues SAML assertions, not the relying party requesting them. Configuring the IdP as Service Provider inverts the trust relationship. It is tempting because the IdP is the other mandatory SAML participant, and it is correctly configured as IdP when the application itself acts as Service Provider.

The LDAP Server.

Why it fails: LDAP is a directory-access protocol, not a SAML role; SAML defines only Identity Provider and Service Provider. An LDAP server supplies user attributes for authentication but cannot issue or consume SAML assertions. It is tempting because LDAP commonly backs authentication, and it would be correct where directory-based bind authentication, not SAML federation, is required.

Configure the forwarder to use a proxy account by setting 'proxyUser' and 'proxyPassword' in inputs.conf.

Why it fails: There is no 'proxyUser' or 'proxyPassword' setting in inputs.conf for file monitoring. Splunk does not support specifying credentials for network share access via inputs.conf. The only way to access a network share is to run the forwarder service under an account that has the necessary permissions.

Enable 'useSSL = true' in outputs.conf to allow secure authentication to the share.

Why it fails: useSSL in outputs.conf enables SSL encryption for data forwarding to indexers, not authentication to network shares. It has no effect on file share access. The forwarder's ability to read a network share depends on the service account's permissions, not on SSL settings.

Add the Local System account to the share's permissions with read access.

Why it fails: The Local System account is a local machine account and cannot be granted permissions on a remote network share. It has no domain identity, so it cannot be added to a remote share's ACL. This approach will not work for accessing a network share that requires domain authentication.

Data model encryption.

Why it fails: Indexer clustering replicates buckets and manages searchable copies, replication factors and master node election; data model encryption is configured separately through the data models and settings, not cluster peer behaviour. It tempts because clustered indexers do store replicated data, so encryption sounds related, but it is not a clustering feature.

Universal forwarder load balancing.

Why it fails: Universal forwarder load balancing is configured on the forwarder itself, selecting among multiple receiving indexers; clustering concerns indexer peer replication and search head failover. It tempts because clustered indexers commonly receive forwarded data, so the two are deployed together, but load balancing is not a clustering feature.

$SPLUNK_HOME/var/run/

Why it fails: $SPLUNK_HOME/var/run holds runtime state such as process IDs and dispatch artefacts, not executable scripts, so the forwarder will not find the script there. Scripts belong in $SPLUNK_HOME/bin/scripts or an app's bin directory; var/run would be correct for transient runtime files.

/opt/scripts/

Why it fails: Splunk expects scripted input scripts under $SPLUNK_HOME/bin/scripts (or etc/apps/<app>/bin), where the forwarder's scripted input handler locates them; /opt/scripts/ is not searched. That path would be acceptable only if inputs.conf referenced the script by its absolute path.

/tmp/

Why it fails: /tmp/ is world-writable and cleared on reboot, so scripts there risk tampering and vanish after restart, breaking the input. It is tempting as a scratch space for temporary files, and would suit throwaway test scripts, but persistent forwarder inputs belong in $SPLUNK_HOME/etc/apps/<app>/bin/.

To define the authentication port.

Why it fails: The User Base DN is the directory subtree searched for user entries, not a network port; port configuration sits in the LDAP server connection settings. It is tempting because authentication does involve a port, but that field is the server port, not the base distinguished name.

To define the administrator's password.

Why it fails: The User Base DN identifies the directory subtree containing user entries; credentials for the bind account are configured separately. It is tempting because LDAP authentication requires a password, but that password belongs to the bind DN, not the user search base.

To specify the LDAP server IP address.

Why it fails: The User Base DN scopes the directory search for user accounts; the server IP address belongs in the LDAP host setting. It is tempting because both are connection-related, yet the base DN is a distinguished name path, not an address.

audit.log

Why it fails: audit.log captures user search activity and configuration changes made through Splunk Web or the CLI, not network-level forwarder connections. It would be the right file when investigating who ran a particular search or altered a knowledge object.

web_service.log

Why it fails: web_service.log records Splunk Web UI and REST endpoint activity, so it never sees the forwarder-to-indexer data channel on port 9997. It is the correct file when troubleshooting dashboard, login or REST API problems rather than forwarder connectivity.

metrics.log

Why it fails: metrics.log records internal Splunk performance counters such as queue sizes and throughput, not inbound TCP connection attempts, so it cannot show forwarder handshakes. It is genuinely useful for diagnosing indexing pipeline slowdowns and resource saturation, which is why it is tempting here.

indexes.conf

Why it fails: Usually defined on the indexer.

splunk-launch.conf

Why it fails: Rarely included in app bundles.