Troubleshooting Scenario Questions

The S3 bucket uses server-side encryption with customer-provided keys (SSE-C).

Why it fails: SSE-C is a per-object encryption mode, not a bucket-level default, and it is not supported by CloudTrail for log delivery. CloudTrail automatically uses SSE-S3 for new buckets and can optionally use SSE-KMS, but it cannot supply the customer-provided key headers required for SSE-C. Even if an encryption mismatch occurred, it would surface as an S3 encryption error rather than an Access Denied caused by a bucket policy deny.

The S3 bucket does not have versioning enabled.

Why it fails: Versioning is an optional bucket property for CloudTrail log storage, not a prerequisite for writing new objects. When versioning is disabled, CloudTrail still successfully delivers log files; the only impact is that overwritten or deleted versions are not retained. Since a delivery failure would appear as an authorization or access error rather than a versioning-related error, the absence of versioning cannot explain the issue.

The S3 bucket is in a different AWS account.

Why it fails: CloudTrail explicitly supports delivering logs to an S3 bucket in a different AWS account, provided the bucket policy includes the appropriate cross-account Allow statements for the CloudTrail service principal and the expected AWSLogs prefix. A remote bucket alone does not prevent delivery; only a missing or misconfigured resource policy would. Therefore, the bucket's account location is not inherently the cause, and the contents of the bucket policy are what should be examined.

The bucket policy changes were made through the AWS Management Console, which is not logged.

Why it fails: This is incorrect because AWS CloudTrail records management events regardless of whether they originate from the AWS Management Console, CLI, or SDKs. PutBucketPolicy is a management event for S3 and appears in CloudTrail when performed through the console. The actual reason no events seem to appear is that the command uses --max-results 1, which truncates the output to a single event.

The event is not logged because PutBucketPolicy is not supported by CloudTrail.

Why it fails: This is incorrect. PutBucketPolicy is a supported S3 control-plane (management) API and is logged by CloudTrail as a management event. S3 API calls such as PutBucketPolicy, DeleteBucketPolicy, and GetBucketPolicy are all tracked, so the absence of output is not due to lack of CloudTrail support. The command's --max-results parameter is what hides additional matching events.

The command is filtering by the wrong attribute.

Why it fails: This is incorrect because EventName is a valid LookupAttribute Key for the lookup-events operation, and PutBucketPolicy is an accurate event name for that API call. Other valid attribute keys include EventSource, Username, ResourceType, and ResourceName, but EventName is appropriate here. The filtering attribute is correct; the output is constrained by the maximum-results setting.

The S3 bucket has an S3 Object Lock configuration that prevents writes.

Why it fails: S3 Object Lock does not block initial PutObject requests; it only restricts deletion or overwriting of existing versions according to a retention mode and period. CloudTrail writes new log objects on a regular basis, and unless the bucket policy denies s3:PutObject, Object Lock will not prevent those deliveries. Even with a default retention policy, writes succeed and the logs are merely locked against modification.

The S3 bucket is in a different AWS Region from the trail.

Why it fails: CloudTrail can deliver log files to an S3 bucket in any AWS Region, not only the bucket's or trail's home region. The trail must have a bucket policy that permits the CloudTrail service principal to write objects, and the bucket may be in a different account or region. A regional mismatch alone is not a cause of delivery failure, so this does not explain the troubleshooting issue.

CloudTrail is not enabled in the AWS Region where the S3 bucket resides.

Why it fails: CloudTrail is a regional service, but a trail is created in a home region and can record and deliver events for all regions when configured as a multi-region trail. The S3 bucket resides in some region, but CloudTrail does not need to be enabled specifically in the bucket's region to deliver logs to it. What matters is that the trail is active and the bucket policy allows CloudTrail to write; the bucket region is irrelevant.

Re-enable the GuardDuty finding in Amazon Detective.

Why it fails: Re-enabling a GuardDuty finding in Amazon Detective is not the issue here: the finding is already present in Detective, and the missing piece is the supporting network activity data. Detective automatically ingests GuardDuty findings as a source of contextual clues, but it cannot retrofit network paths without the underlying VPC Flow Logs. Simply toggling the finding would not cause Detective to backfill or begin receiving flow log data, so it would leave the network information blank.

Enable GuardDuty EKS Audit Logs monitoring.

Why it fails: Enabling GuardDuty EKS Audit Logs monitoring addresses Kubernetes control-plane activity for Amazon EKS clusters, not VPC network traffic for EC2 instances. EKS audit logs capture API calls to the Kubernetes API server, which is a different telemetry source than VPC Flow Logs. Since the missing data is about network connections between resources, this option would not populate the network activity section in Detective and is therefore not the correct remediation.

Install the Amazon Detective agent on the EC2 instance.

Why it fails: Amazon Detective is an agentless service; it does not require or support installing any agent on an EC2 instance to gather telemetry. Instead, Detective ingests data from existing sources such as AWS CloudTrail, VPC Flow Logs, GuardDuty findings, and EKS audit logs. Installing an agent would not help because there is no Detective agent to install, and even a generic network agent would not feed into Detective’s graph. Therefore, this option cannot enable the missing network activity data.

Delete the root user account.

Why it fails: The AWS account root user cannot be deleted; it persists for billing and account recovery, so this action is technically impossible. It is tempting because removing a risky credential seems decisive, but the correct hardening is disabling root access keys and enforcing MFA on the root user.

Disable the root user password and require all logins via IAM users.

Why it fails: The root user password cannot be disabled; root credentials remain recoverable for account-level tasks, so this control cannot be applied. It is tempting because forcing IAM users appears to eliminate root exposure, but the achievable hardening is MFA on root plus removing root access keys.

Enable IAM Access Analyzer to detect and alert on root user activity.

Why it fails: IAM Access Analyzer evaluates resource policies for external access, not CloudTrail authentication events, so it cannot detect or alert on root console logins. It is tempting because it is an IAM security service, but root activity monitoring requires CloudWatch alarms or EventBridge rules on CloudTrail.

The NAT gateway does not have an Elastic IP address assigned.

Why it fails: Incorrect. A NAT gateway requires an Elastic IP to be assigned at creation; if it were missing, the NAT gateway would not have been successfully created. The engineer would have seen an error during creation.

The security group for the NAT gateway does not allow inbound traffic from the private subnets.

Why it fails: NAT gateways are a managed AWS service and do not have security groups associated with them; security groups only apply to Elastic Network Interfaces (ENIs) of EC2 instances or other resources like RDS. Therefore, checking a NAT gateway's security group is not a valid troubleshooting step, and the absence of such a security group is not the root cause. The actual issue is that the private subnet's route table must contain a 0.0.0.0/0 route pointing to the NAT gateway ID.

The network ACL for the private subnet does not allow inbound HTTP/HTTPS traffic.

Why it fails: Incorrect. While network ACLs could potentially block traffic, the most likely cause given the scenario (route tables updated, security group check) is a missing default route. Network ACLs are stateless and would need to allow both inbound and outbound traffic, but this is less common than a routing issue.

The security group egress rule is blocking return traffic.

Why it fails: The egress rule cannot be blocking return traffic because it is configured to allow all outbound traffic. Security groups are stateful, so after the inbound SSH connection is accepted, response traffic from the instance to the client's ephemeral port is automatically permitted irrespective of egress rules. Since the egress rule is permissive and stateful tracking covers the reply packets, this option is not the reason for the SSH failure.

The network ACL's default deny rule (32767) is blocking all inbound traffic.

Why it fails: The network ACL's default deny rule numbered 32767 is only a catch-all for traffic that has not already matched an earlier explicit rule. Inbound rule 100 in the exhibit explicitly allows SSH from all source IP addresses, and because network ACL rules are evaluated in ascending order, that allow match is applied before flow ever reaches the default deny. Therefore the catch-all rule is not blocking the inbound SSH connection.

The network ACL inbound rule for SSH is misconfigured, denying all traffic.

Why it fails: This is incorrect because the network ACL inbound rule for SSH is actually configured with an Allow action for all source IPs, not a Deny action. A network ACL is stateless, so the rule simply permits TCP port 22 from any source, which would allow rather than block the engineer's SSH attempt. The SSH failure therefore must be caused by a different layer, namely the security group's source restriction.

The network ACL for the subnet is blocking outbound traffic to the on-premises CIDR.

Why it fails: If the subnet's network ACL were blocking outbound traffic to the on-premises CIDR, the VPC instances would be unable to send any packets toward Direct Connect, and the on-premises side would receive nothing at all. The scenario demonstrates that outbound traffic is reaching the on-premises network, which proves the NACL is permitting the egress flow. NACLs are stateless and require explicit allow rules for both outbound and inbound traffic, but here only the return path is broken.

The virtual private gateway is not attached to the VPC.

Why it fails: A virtual private gateway that is not attached to the VPC would leave the VPC route table without any viable target for Direct Connect traffic, preventing all outbound packets from ever leaving the VPC. Since the observed symptoms show outbound traffic successfully arriving at on-premises, the gateway must be attached and operational. An unattached gateway would be a total connectivity failure, not a one-way connectivity issue.

The security group does not allow inbound traffic from the on-premises server.

Why it fails: Security groups are stateful: if the instance's outbound rule permits traffic to the on-premises server, the security group automatically allows the corresponding inbound response traffic, even without an explicit inbound rule. Therefore, a missing inbound rule for the on-premises CIDR cannot cause the return packets to be dropped. The issue lies in an external router lacking a return route, not in the security group configuration.

The bucket is in a different AWS account.

Why it fails: A bucket being in a different AWS account is not inherently a cause of denial. S3 bucket policies support cross-account access by specifying the IAM role ARN as the Principal; if the role has the required s3: actions allowed in its identity policy and the bucket policy grants access to that principal (with no explicit deny), the request succeeds. Therefore, an account boundary alone does not block access, and the troubleshooting focus should be on the bucket policy's Allow/Deny statements.

The bucket policy does not explicitly allow the IAM role.

Why it fails: For a bucket and IAM role in the same AWS account, a bucket policy does not need to separately list the role if the role already has an identity-based allow for the S3 action. S3 combines identity-based and resource-based permissions with a logical OR in the same account, so an absence of an explicit Allow in the bucket policy is not a denial. Unless the bucket policy includes an explicit Deny for the role, the role's own grant is still effective, so 'not explicitly allowing' is not the root cause.

The IAM role has an explicit deny statement.

Why it fails: An explicit deny attached to the IAM role's identity policy would indeed override any allow from the bucket policy and would cause the access failure. However, the scenario states that the role has the necessary permissions, which rules out a role-level explicit deny or any effective deny elsewhere in the identity policy. The cause must therefore lie in the resource-based bucket policy, not in the role's permissions, since explicit deny on the role would be independent of the bucket policy and would contradict the premise.

Terminate the instance immediately and launch a replacement.

Why it fails: Terminating the instance immediately is wrong because it is a destructive action that by default deletes the root volume, destroying all disk-based evidence such as malware binaries, logs, and attacker scripts, along with any memory-resident data. It also causes an immediate production outage without first containing the compromise, which is premature because the incident response team has not yet preserved the system state or determined the blast radius. Launching a replacement from a clean AMI may restore service, but it forfeits the forensic artifacts needed to identify the root cause and prevent recurrence.

Take an EBS snapshot of the instance's root volume for analysis.

Why it fails: Taking an EBS snapshot is a reasonable forensic step, but it is wrong as the first action because it does not contain the live compromise—an attacker can continue to exfiltrate data and move laterally while the snapshot is being taken. A snapshot only captures the disk at a point in time and misses memory, open network sockets, and running processes, so it provides an incomplete picture of the incident. It should be performed after isolating the instance, as part of a documented evidence collection process, not as a substitute for immediate containment.

Use AWS Systems Manager Run Command to install a forensic agent on the instance.

Why it fails: Using SSM Run Command to install a forensic agent on a potentially compromised instance is risky because the instance's integrity is already suspect; an attacker with root or elevated privileges could have tampered with the SSM agent, its certificates, or the instance's IAM role, meaning the command may be modified or the agent may exfiltrate the forensic tooling. Installing software also writes new files and changes system state, potentially overwriting the very evidence you are trying to collect. In incident response, you must never trust a compromised host to execute your tools; instead, isolate it and analyze a snapshot or attached volume from a separate, trusted forensic instance.

The IAM policy explicitly denies console access.

Why it fails: An IAM policy that explicitly denies console access would produce a role-based denial after the user has authenticated, not a failure during MFA verification. AWS enforces MFA as a pre-authentication step, so the sign-in process never reaches IAM policy evaluation if the MFA code is rejected. Therefore, a policy deny could explain an “AccessDenied” page but not an error that the MFA code is invalid or a credential challenge failing.

The user is using the root account instead of an IAM user.

Why it fails: Signing in as the root account instead of an IAM user does not bypass MFA; the root user can have its own MFA device, and AWS strongly recommends applying MFA to root. If root has no MFA, it would simply not prompt for MFA, rather than prompting and then rejecting the entered token. The issue described is specifically a rejected MFA challenge, which points to the device or time synchronization, not to which account type is being used.

The MFA token has expired.

Why it fails: MFA tokens generated by TOTP (RFC 6238) are short-lived, usually 30 seconds, and are valid for only that single time step; they do not expire in the same way a password or access key does. If a user enters a previously used or stale code, AWS will reject it because it is outside the allowed time window, but this is a freshness or reuse problem, not a permanent expiration of the token. The fix is to use the current code shown in the authenticator app, not to wait out or renew an MFA token, so 'token has expired' is not a correct diagnostic.

The instance is participating in a DDoS attack against the external IP.

Why it fails: The flow log entry captures a single accepted connection, not the high-volume, multi-source packet flood characteristic of a DDoS attack. DDoS participation would manifest as thousands of simultaneous flows or massive byte counts from the instance to the target, whereas this log shows one discrete outbound session. Additionally, the source is a private EC2 IP, which would require spoofing to reflect traffic in a DDoS, and a single flow cannot sustain an amplification attack.

An external host is scanning the internal network on port 443.

Why it fails: This option misreads the flow direction: the source is the internal IP 10.0.1.5 and the destination is the external 203.0.113.50, meaning the traffic is outbound from the VPC, not an inbound scan. A port scan of the internal network would originate from an external source to multiple internal IPs or ports, generating many short-lived flows to a range of destinations. The log shows only a single flow to the external host on port 443, which is not indicative of a scanning pattern.

The security group allows inbound RDP from 0.0.0.0/0.

Why it fails: VPC Flow Logs record metadata about actual traffic (source, destination, port, protocol, action) and do not include security group rules or their configuration. To determine whether a security group allows inbound RDP from 0.0.0.0/0, you would need to inspect the security group's inbound rules via the console, CLI, or API. Furthermore, the referenced traffic is outbound (source internal, destination external), so an inbound rule on the instance's security group would not even apply to this connection.

AWS/KMS: KeyUsage

Why it fails: KeyUsage in the AWS/KMS namespace counts cryptographic operations performed with a customer master key, such as Encrypt, Decrypt, or GenerateDataKey. Secrets Manager uses a KMS key to encrypt the secret, but KMS does not expose a metric describing the rotation state or the scheduled rotation result. An increase or absence of KeyUsage therefore tells you nothing about whether a rotation succeeded or failed, only that the key may have been used. Monitoring this metric would not satisfy the requirement to detect rotation failures.

AWS/Lambda: Invocations

Why it fails: Rotation in Secrets Manager is implemented through a Lambda function, but CloudWatch Invocations only reports the number of times that function is invoked. A rotation attempt might fail after the function has started—for example, due to a network timeout or an invalid master secret—while still counting as an invocation and causing retries. Additionally, Lambda retries failed invocations, so Invocations can exceed the number of rotation attempts and cannot distinguish successful rotations from failed ones. Thus, while it relates to rotation plumbing, it is not a reliable indicator of rotation health.

AWS/RDS: DatabaseConnections

Why it fails: DatabaseConnections measures the current number of sessions to an Amazon RDS instance and is published by the RDS service. This metric is influenced by application workload and connection pool settings, not by the Secrets Manager rotation lifecycle. A rotation can succeed or fail without ever changing the database connection count, and connections can remain established with old credentials, so this metric provides no visibility into whether the secret was rotated. It is therefore not an appropriate signal for monitoring Secrets Manager rotation errors.

CloudTrail and SNS

Why it fails: CloudTrail records management and data-plane API calls made by or on behalf of the Lambda function, such as Invoke, CreateFunction, or policy changes, but it does not capture the function's stdout/stderr, runtime exceptions, or custom application logs. Therefore SNS alerts triggered by CloudTrail events would not indicate processing errors or anomalous execution of the function. It is useful for audit trail analysis, not real-time application-level log monitoring.

AWS Config and SQS

Why it fails: AWS Config evaluates resource configurations, such as whether the Lambda function's environment variables are encrypted or its IAM role is compliant, and can deliver configuration snapshots or compliance changes via SNS, but it cannot inspect log output or runtime errors. SQS is a message queue, not a notification destination, and Config would only trigger on resource configuration changes, not on sensitive data processing failures. Hence this combination is unsuitable for real-time alerting on function execution.

Amazon Detective and SES

Why it fails: Amazon Detective ingests findings from GuardDuty, CloudTrail, VPC Flow Logs, and EKS and builds behavior graphs for post-incident security investigations; it does not continuously monitor CloudWatch Logs from a Lambda function or generate real-time operational alerts. SES is an email-sending service and could theoretically deliver an alert, but Detective does not emit event notifications when application logs contain an error. This combination is designed for forensic analysis after a security event, not for immediate processing-failure alerts.

The S3 bucket has a lifecycle policy that deletes objects too quickly

Why it fails: A lifecycle policy that transitions or expires objects operates asynchronously after objects are written; it does not block CloudTrail from delivering log files in real time. Even an aggressive expiration rule would only remove objects minutes or days later, after the delivery attempt has already succeeded. Therefore, lifecycle configuration cannot prevent CloudTrail from writing the initial log files.

CloudTrail is not enabled in the region

Why it fails: CloudTrail trails are regional resources; if CloudTrail is not enabled in the region where the trail is configured, no trail would exist to deliver logs. Because the CloudTrail console and APIs only show trails that were successfully created in a region, a missing trail indicates a creation failure or no configuration, not a delivery failure of an existing trail. This option cannot explain why an already created trail stops delivering logs.

The S3 bucket is in a different region than the trail

Why it fails: CloudTrail explicitly supports delivering logs to an S3 bucket in a different region than the trail, as long as the bucket policy allows cross-account or cross-region writes. When you create a trail, CloudTrail adds the necessary bucket policy statements to deliver to any valid S3 bucket regardless of region. Therefore, the bucket being in a different region does not inherently prevent log delivery and is not a valid cause of the issue.