Scaling Vault For Performance practice questions

You must disable request forwarding in the listener configuration.

Why it fails: Request forwarding is standard behavior for non-performance standby nodes.

The node requires a separate license file for each standby.

Why it fails: Performance standby is a feature of Vault Enterprise and managed via the license already applied.

The node must be joined to the cluster using vault operator raft join.

Why it fails: This is for raft clustering, not specifically for enabling standby read capability.

The Performance Secondary is in a sealed state due to a replication lag.

Why it fails: Replication lag does not cause a seal state.

The Performance Secondary node has not been initialized with a separate encryption key.

Why it fails: Performance secondary clusters inherit the master key from the primary.

The client is using a Batch token, which is not supported on performance secondaries.

Why it fails: Batch tokens are supported on performance secondaries if the cluster is configured for it.

The secondary node is not configured as a performance standby.

Why it fails: Performance secondaries are inherently performance-capable nodes.

The primary node is unavailable.

Why it fails: If the primary is unavailable, the secondary would not be receiving updates, increasing lag.

The secondary node is missing the required policies.

Why it fails: This would result in 403 errors, not necessarily high latency.

The token has already expired.

Why it fails: This would cause rejection on the primary as well.

The secondary node has a different root CA.

Why it fails: Replication links would be broken if root CAs differed.

The token was issued with a policy that does not exist on the secondary.

Why it fails: Batch tokens don't rely on policy lookups in the same way service tokens do.

replication_max_reads

Why it fails: This parameter does not exist.

rate_limit

Why it fails: Rate limits are applied per-namespace or per-token, not globally per node in the config.

max_request_limit

Why it fails: This parameter does not exist in Vault.

They become Disaster Recovery secondaries.

Why it fails: DR roles are distinct from Performance roles.

They automatically promote themselves to Primary.

Why it fails: There is no automatic promotion of secondary clusters.

They are automatically removed from the cluster.

Why it fails: They remain in a disconnected state.

It automatically promotes itself to primary.

Why it fails: Promotion is not automatic.

It enters a sealed state.

Why it fails: It remains in a running state, serving cached reads.

It shuts down immediately.

Why it fails: It stays active.

The Active node dictates the traffic flow.

Why it fails: The Active node does not control load balancer traffic.

Vault uses a proprietary load balancing protocol.

Why it fails: Vault relies on external load balancers.

The client performs the load balancing based on a local config file.

Why it fails: Clients typically hit a load balancer VIP.

The token is silently ignored.

Why it fails: Vault will return an error.

The token is automatically updated to the new path.

Why it fails: Paths are static configuration.

The secondary redirects the request to the primary.

Why it fails: Batch tokens are validated locally; no redirect occurs.

Batch tokens support periodic renewal.

Why it fails: They do not support renewal.

Batch tokens can be revoked by their ID.

Why it fails: They cannot be tracked for revocation.

Batch tokens must have an explicit TTL.

Why it fails: They can inherit a default TTL.

Dynamic secrets.

Why it fails: Dynamic secrets are available in all versions.

Integrated storage (Raft).

Why it fails: Raft is part of Vault open source.

Querying the consul health check API.

Why it fails: Consul checks are for cluster membership, not replication health.

Using the vault sys/health endpoint.

Why it fails: This only checks node health.

Watching the system logs for 'replication successful' messages.

Why it fails: Logs are not a primary monitoring tool for status.

Generation of audit logs on the primary.

Why it fails: Audit logs are not path-filtered.

Authentication methods on the primary.

Why it fails: Auth methods are not path-filtered.

Restart the secondary cluster.

Why it fails: Promotion does not require a restart.

Upgrade the secondary's license to Enterprise.

Why it fails: It is already enterprise if it was a secondary.

The need to replicate all tokens across regions.

Why it fails: Batch tokens are not replicated.

Increased memory footprint for token storage.

Why it fails: They are not stored, so they don't consume memory for persistence.

Increased storage IOPS on the backend.

Why it fails: Batch tokens reduce storage load.

Performance Standby nodes can service read requests for all secrets engines, including those that are not replication-enabled.

Why it fails: Performance Standby nodes only serve reads for replication-enabled secrets engines. Not all secrets engines support replication; for example, some external plugins may not be replicated. Therefore, reads for non-replicated engines must be forwarded to the active node. This limitation is crucial for capacity planning.

Performance Standby nodes can process write requests if the active node is unavailable.

Why it fails: Performance Standby nodes are strictly read-only. They cannot process write requests under any circumstances. If the active node fails, a standby must be promoted to active, but until then, writes will fail. This design ensures consistency and prevents split-brain scenarios. Writes are always forwarded to the active node.

Performance Standby nodes must be configured with the same storage backend as the active node.

Why it fails: Performance Standby nodes do not use their own storage backend; they receive data via replication from the active node. The storage backend configuration is only relevant for the active node. Standbys rely on the replication stream for data. Therefore, they do not require identical storage configuration, though they may have local storage for other purposes.

The secondary cluster is serving read requests for replication-enabled secrets engines.

Why it fails: Serving read requests on the secondary does not cause replication lag; it is a designed use case. The secondary can handle reads independently, which actually reduces load on the primary. Lag is typically due to network or primary performance issues, not because the secondary is serving reads. In fact, offloading reads is beneficial.

The secondary cluster has a different seal configuration than the primary.

Why it fails: Seal configuration does not affect replication performance. The seal is used for encrypting and decrypting the storage, but replication operates on the encrypted data. A difference in seal type does not impact the speed of replication. Lag is caused by factors like network, disk I/O, or CPU on the primary or secondary.

The load balancer is not configured to route traffic to the secondary cluster.

Why it fails: Load balancer configuration affects client traffic distribution, not replication lag. Replication is a background process between clusters. If the load balancer is misconfigured, clients may not reach the secondary, but that does not increase lag. Lag is about data synchronization delay, which is independent of client traffic routing.

Increase the `max_request_duration` parameter in the Vault configuration to allow more time for read requests to complete.

Why it fails: `max_request_duration` controls the maximum time a request can take before being canceled. Increasing it may allow longer-running requests to complete, but it does not distribute load or improve performance. It could actually worsen the situation by allowing more requests to pile up on the leader.

Add more nodes to the Raft cluster and configure them as non-voters to handle read requests.

Why it fails: Non-voter nodes in Raft do not serve client read requests; they only participate in replication to improve write durability and failover. They cannot handle reads. Adding non-voters will not offload read traffic from the leader. To serve reads, nodes must be Performance Standbys.

Enable the `raft_autopilot` feature and set the `cleanup_dead_servers` parameter to true.

Why it fails: Autopilot is used for automatic dead server cleanup and stable server introduction, not for load balancing reads. Enabling Autopilot does not allow non-leader nodes to serve reads; it only manages cluster membership and health. It will not alleviate the leader's read load.

Enable audit log file rotation with a short rotate_duration and set a maximum number of retained audit files.

Why it fails: Audit log rotation manages the size and retention of audit files on disk, but it does not reduce the volume of writes to the Raft log. The Raft log is separate from audit files and will still grow with each write operation, so this does not mitigate WAL growth or disk exhaustion in the Raft storage backend.

Set performance_multiplier to a lower value to reduce the number of nodes required to acknowledge writes.

Why it fails: The performance_multiplier parameter adjusts the number of additional voters in a Raft cluster for performance standby nodes, but it does not control write acknowledgment or WAL growth. Lowering it may reduce cluster size but does not address the core issue of high write volume causing Raft log growth and potential disk exhaustion.

Increase the raft_snapshot_threshold and raft_snapshot_interval to reduce the frequency of snapshots.

Why it fails: Increasing snapshot thresholds and intervals reduces snapshot frequency, which actually allows the Raft log to grow larger between snapshots, increasing disk usage and WAL growth risk. This setting does not directly address the high write throughput from audit logs and can worsen disk exhaustion by delaying log truncation.

The secondary cluster is forwarding token creation requests to the primary for signing, causing cross-region latency.

Why it fails: In Performance Replication, token creation can be handled locally on the secondary because the token signing keys are replicated. Forwarding to the primary would only occur for operations that require the primary, such as certain writes to replicated mounts. Token creation is a local operation, so this is not the cause of increased latency.

The secondary cluster is performing a local token creation but must wait for the primary to acknowledge the write to the token store before returning.

Why it fails: Performance Replication allows secondaries to handle token creation locally without waiting for primary acknowledgment. The token store is replicated, and secondaries can issue tokens independently. Waiting for primary acknowledgment would defeat the purpose of performance replication, which is to offload read and some write operations to secondaries.

The secondary cluster's integrated storage is experiencing high disk I/O due to replication of audit logs from the primary.

Why it fails: Audit logs are not replicated between primary and secondary clusters in Performance Replication. Audit logging is configured per cluster and writes to local storage. Therefore, replication of audit logs is not a factor, and disk I/O from audit logs would be local to each cluster's own audit configuration, not a result of replication.