Monitoring, Logging and Runtime Security practice questions

crictl logs <container-id>

Why wrong: crictl logs retrieves only the standard output and standard error streams produced by the container's main process, which reveals application-level messages but provides no access to the container's filesystem. Logs are stored by the runtime or a logging driver and do not reflect deleted files, configuration artifacts, or binaries present on disk. For filesystem forensic analysis, capturing the root filesystem via crictl export is required.

crictl inspect <container-id>

Why wrong: crictl inspect returns a JSON object containing the container's configuration, status, mounts, network settings, and resource details, but it does not expose the actual content of files on the container's filesystem. It can be useful to identify the image and mount points, yet it cannot answer questions about file contents or hidden artifacts. To inspect raw files, you must export the filesystem or use a copy-on-write tool.

crictl exec <container-id> tar cvf /tmp/fs.tar /

Why wrong: crictl exec runs a command inside the container, which creates a new process and thus changes the runtime state; using tar inside the container writes an archive to /tmp/fs.tar, leaving forensic artifacts and potentially triggering anti-forensic mechanisms in a compromised pod. Additionally, the archive remains inside the container, requiring a second exec to copy it out, and the tar binary itself may be subverted by an adversary. A proper acquisition should be performed externally using crictl export to guarantee the container's filesystem is not modified.

rules: [- level: RequestResponse, resources: [group: '', resources: [*]], - level: Metadata]

Why wrong: This applies RequestResponse to all resources, not just secrets in kube-system.

rules: [- level: Metadata, resources: [group: '', resources: [secrets]], namespaces: [kube-system], - level: RequestResponse]

Why wrong: This applies Metadata to secrets in kube-system and RequestResponse to others, opposite of the requirement.

rules: [- level: Metadata, resources: [group: '', resources: [secrets]], namespaces: [kube-system], omitStages: [RequestReceived], - level: RequestResponse]

Why wrong: omitStages does not set the audit level; it omits logs at certain stages.

The `securityContext` is misspelled

Why wrong: Misspelling a field name in the Pod spec is a schema-level mistake that the Kubernetes API server would normally reject during resource creation or update, typically with a validation error like 'unknown field'. Even if the API server were configured to prune unknown fields (e.g., via Server-Side Apply), the securityContext would be silently dropped, meaning the intended read-only root filesystem would never be enabled. Neither outcome produces a runtime write failure inside the container; a misspelling would fail at admission time or simply have no effect, not cause the EROFS error observed.

The container image does not have `/tmp` directory

Why wrong: Whether the container image happens to include a /tmp directory is a red herring. When readOnlyRootFilesystem is enabled, the entire root filesystem is mounted read-only, including any existing /tmp directory; therefore, even if /tmp exists in the image, a write to it would still be rejected with a read-only file system error. The application needs a writable volume like an emptyDir mounted at /tmp, not simply a directory that already exists in the image. In fact, the image's filesystem layout is irrelevant to the mount-read-only behavior.

The container is running as a non-root user

Why wrong: The root filesystem's read-only status is enforced at the kernel mount layer, not through Unix user permissions. When readOnlyRootFilesystem is true, the vfsmount is marked MS_RDONLY, and the kernel rejects write operations on that mount for every UID, including root. Changing the container to run as a non-root user would not help because the failure is caused by the mount flag, not a lack of filesystem write permission for the current user; in fact, even a non-root user with full rwx permissions on a directory within the rootfs would still be unable to write when the mount is read-only.

open

Why wrong: The open syscall is used to access filesystem objects, returning a file descriptor that can be read from or written to. While curl may call open to load configuration files, CA certificates, or the executable's shared libraries, the network connection itself is established through socket syscalls on a separate descriptor. Therefore open is unrelated to the actual TCP connection and would not reveal the curl request unless a file operation is being traced.

execve

Why wrong: The execve syscall replaces the current process image with a new executable; when the shell runs curl, it forks a child and calls execve to load curl into memory. This only marks the start of the curl process and does not involve any socket operations or network traffic. The actual TCP connection to the remote server is handled by subsequent socket-related syscalls, so execve would not capture the HTTP request itself.

bind

Why wrong: The bind syscall attaches a socket to a specific local IP address and port, and it is primarily used for listening sockets in server applications such as nginx or sshd. For an outbound HTTP request, curl does not need to bind explicitly because the kernel automatically assigns an ephemeral source port during the connection setup. Since the question is about an outgoing curl request, bind would not be the syscall that initiates the network connection.

container.name=shadow and evt.type=open

Why wrong: The `container.name` field in Falco refers to the container's name metadata (e.g., as set by `--name` or Docker's naming), not to file paths. Setting `container.name=shadow` would match only if a container itself is literally named "shadow" — it does not filter on the file being opened. Additionally, there is no `fd.name` condition, so this rule would fire for every open syscall in that container, flooding alerts. To detect access to /etc/shadow, you must filter on the file descriptor's name, not the container's name.

evt.type=read and fd.name=/etc/shadow

Why wrong: The `read` syscall receives an already-open file descriptor number, not a path, so `fd.name` is often unavailable or stale at that point in Falco's event stream. Falco populates `fd.name` reliably on the `open` syscall, where the path is explicitly passed as an argument; on `read`, the fd may refer to a pipe, socket, or an fd that was closed before the event is processed. Relying on `read` would also miss attempts to open the file for writing or other operations. The correct approach is to trap the `open` syscall, which is the point where the file path is first declared.

proc.name=cat and fd.name=/etc/shadow

Why wrong: Restricting the rule to `proc.name=cat` makes it trivially evadable: an attacker can use `grep`, `less`, `head`, `vi`, or a shell builtin like `echo` with redirection to read /etc/shadow without spawning a process literally named `cat`. Even renaming a `cat` binary to something else bypasses the filter, as would using a custom tool. The event of interest is the file access itself, not the actor's process name, so tying detection to `cat` creates a narrow, fragile rule that misses the vast majority of real attempts.

PodSecurityPolicy

Why wrong: PodSecurityPolicy is an admission controller resource that historically enforced constraints on pod security contexts, such as whether a pod could run privileged or use host namespaces. It was deprecated and removed in Kubernetes v1.25, and it has nothing to do with auditing or logging. Audit logging concerns the recording of API server request metadata and payloads, not the enforcement of pod-level security policies.

ConfigMap in kube-system

Why wrong: A ConfigMap placed in the kube-system namespace is a standard Kubernetes resource for storing key-value configuration data. However, the kube-apiserver does not consume the audit policy from a ConfigMap; it requires a local file path supplied with `--audit-policy-file`. Even if you mount a ConfigMap as a volume into the apiserver's pod, the apiserver only sees a file on disk, and the ConfigMap itself is never interpreted as the audit policy.

AuditPolicy CRD

Why wrong: In upstream Kubernetes, there is no built-in `AuditPolicy` Custom Resource Definition (CRD). The audit policy is not represented as a CRD; it is purely a YAML file on the apiserver's filesystem. While you could theoretically create your own CRD named AuditPolicy, the apiserver would have no built-in logic to read or apply it for auditing, so it would not satisfy the requirement.

Modify the rule condition to add `and not user.name in (admin, root)`

Why wrong: This would exclude all admin and root users, which might be too broad and miss real threats from those accounts.

Change the rule priority to `INFO`

Why wrong: Changing priority does not reduce the number of alerts; it only changes the severity level.

Disable the rule

Why wrong: Disabling the rule would stop detecting the activity entirely, which may not be desired.

The emptyDir volume is not mounted correctly in the pod spec

Why wrong: If the volume were not mounted, the container would fail to start or the directory would be empty, but the issue is permissions.

The emptyDir volume is using a read-only storage class

Why wrong: emptyDir volumes are ephemeral and always writable; there is no storage class.

readOnlyRootFilesystem also applies to mounted volumes

Why wrong: readOnlyRootFilesystem only affects the root filesystem of the container, not mounted volumes.

None

Why wrong: The None audit level is a blanket suppression: it tells the kube-apiserver to omit any matching events from the audit log entirely, so neither request metadata nor body information is captured. It is commonly used to exclude health probes, token reviews, or other high-traffic endpoints that would otherwise bloat the audit trail.

Metadata

Why wrong: The Metadata level captures the request's identity and context—such as user, groups, verb, resource, namespace, and response status—but explicitly discards the request and response bodies. This gives a lightweight audit trail for authorization and access analysis without storing payloads that may contain secrets or sensitive data. It is the default level in many sample policies.

RequestResponse

Why wrong: The RequestResponse level is the most complete audit level, recording the full request metadata, the request body, and the response body. This is useful for detecting or debugging exactly what was sent and returned, but it carries significant performance and privacy costs because every payload is persisted. Consequently, production policies typically apply this level narrowly to privileged actions or suspicious operations.

RequestEvaluated

Why wrong: RequestEvaluated is not a valid audit stage in the Kubernetes API. The audit system in kube-apiserver only recognizes a fixed set of stages defined in the audit.k8s.io/v1 API, and 'RequestEvaluated' is not one of them. This name might be confused with admission evaluation, but the actual stages are strictly RequestReceived, ResponseStarted, ResponseComplete, and Panic.

All of the above

Why wrong: Selecting 'All of the above' is incorrect because the option list contains invalid stages, specifically RequestEvaluated and ResponseSent. The Kubernetes audit stages are limited to four canonical values, and since two of the listed options (ResponseStarted and RequestReceived) are the only valid ones, 'All of the above' picks up fabricated entries and is therefore wrong.

ResponseSent

Why wrong: ResponseSent is not an official audit stage in Kubernetes. The final stage after a response is fully written is called ResponseComplete, not ResponseSent; the latter name may appear in other systems but not in the Kubernetes audit API. Using ResponseSent would be invalid in an audit policy, and the correct terminal stage to reference is ResponseComplete.

High CPU usage

Why wrong: High CPU usage is a resource-utilization metric, not a syscall event, so Falco—which analyzes kernel syscall streams—cannot directly observe or alert on it. A CPU spike may result from legitimate workload bursts, a busy-loop bug, or denial-of-service, and without a syscall context it is not a specific sign of compromise. Tools like Prometheus metrics or Kubernetes Horizontal Pod Autoscaler are designed to track this signal, but they lack the fine-grained execution context that Falco provides.

Pod restarting in a loop

Why wrong: A pod restart loop is a Kubernetes control-plane state transition reported by the kubelet, not a syscall, and Falco operates at the kernel syscall layer on the host or in the container itself. Restart loops can stem from failed health probes, OOM kills, or bad startup configuration, none of which necessarily indicate a security breach, so they are unreliable as a security signal. For this kind of event, you would use Kubernetes audit logging or custom controllers, not Falco.

PreProcessing

Why wrong: There is no audit stage called PreProcessing in Kubernetes. The valid audit stages are strictly RequestReceived, ResponseStarted, ResponseComplete, and Panic. This term might be confused with the admission controller phases, but audit events are not generated during a preprocessing phase. The request lifecycle is captured only at the four defined stages, and any other name is not a real audit stage.

None

Why wrong: 'None' is not an audit stage; it is one of the four valid audit levels (None, Metadata, Request, RequestResponse) that determine how much information is logged. Setting the level to None disables audit logging for a particular rule, but it does not represent a point in the request lifecycle. Stages identify when an event is emitted, while levels specify the verbosity of the event, so confusing them is a common mistake.

PostProcessing

Why wrong: There is no PostProcessing stage in Kubernetes audit. The closest valid stage is ResponseComplete, which is emitted after the entire response has been written to the client. The term might imply a phase after request processing, but Kubernetes uses ResponseComplete instead, and audit events are not generated after the request has been fully handled. Therefore, PostProcessing is not a recognized stage in the audit policy.

kubectl run --stdin --tty --image=busybox

Why wrong: kubectl run --stdin --tty --image=busybox creates a new pod from the busybox image rather than interacting with an existing pod. It lacks a name argument (e.g., kubectl run shell --image=busybox), and even when complete, it only schedules a fresh pod with no relation to the running container you want to access. This command is for launching workloads, not for entering an existing container's process namespace.

kubectl logs <pod>

Why wrong: kubectl logs <pod> streams the captured stdout/stderr from the container's logging pipeline; it is a one-way, read-only operation. It provides no stdin channel and cannot execute any command inside the container. Even with -f (follow), you only observe ongoing log lines, which never gives you a shell or an interactive terminal session for debugging.

kubectl attach <pod>

Why wrong: kubectl attach <pod> connects the local terminal to the already-running primary container process (PID 1) and its standard streams, but it does not create a new process such as /bin/sh. If the main process (e.g., a web server) does not read from stdin or respond to TTY input, the session appears hung and useless. Attach can only interact with whatever the existing process does with its stdio; it cannot run arbitrary commands.

falco --load /path/to/rules.yaml

Why wrong: Falco's CLI does not define a `--load` flag; invoking it will cause an argument parsing error before any rules are evaluated. In Falco, rule files are supplied through the `-r` option, so `--load` is unrecognized and distinct from `-l`/`--list` (which lists rules). Thus this command would fail rather than load a custom rule set.

falco --rules /path/to/rules.yaml

Why wrong: While `--rules` sounds like the long-form of `-r`, Falco's actual long-form option is `--rulesfile`, not `--rules`. The `-r` short flag is the only shorthand, and `--rules` is not recognized by the CLI, so Falco would reject it as an unknown argument. This is a subtle but important distinction from the correct invocation.

falco --config /path/to/rules.yaml

Why wrong: This flag is used to point Falco to its configuration file (`falco.yaml`), which contains settings like output channels, buffering, and rule-related configuration—not the rules definitions themselves. Using `--config` to pass a rules YAML would misdirect Falco and fail to load the rules, because the configuration schema expects a different structure. The rules file must be specified separately with `-r`.

proc.name contains "shadow" and evt.type=read

Why wrong: This filter checks for a process whose name contains "shadow" combined with any read syscall event. Process names like "shadow" or "shadowd" are unrelated to file access; /etc/shadow can be opened by any process such as "cat" or "python". It fails to inspect the target file, so a direct read of /etc/shadow by a normal process would not match this rule.

evt.type=read and fd.name contains "shadow"

Why wrong: Filtering on read syscalls with fd.name containing "shadow" detects only post-open reads and can be bypassed if the file is read through a hardlink or an inherited file descriptor; "contains" also matches files such as /etc/shadow.backup. More importantly, failed open attempts (e.g. permission denied) never reach the read stage, so the opening attempt is the better event to trace.

container and fd.name=/etc/shadow

Why wrong: The token "container" is a boolean-ish field indicating the event occurred in a container, not an event selector; it neither restricts the syscall type nor the file path. Without evt.type=open and fd.name=/etc/shadow, such a filter would match every container event, producing noise and failing to identify reads of the shadow file.

The container runtime is not Docker

Why wrong: The verdict is wrong because crictl is not a Docker-specific tool; it is a generic CRI client that can talk to any runtime exposing the Kubernetes CRI API. Whether the runtime is Docker, containerd, or CRI-O, the only requirement is that the --runtime-endpoint is correctly set to the runtime's gRPC socket. Therefore, 'not Docker' alone cannot explain empty crictl output, and Docker itself would actually require a CRI shim to be usable by kubelet and crictl.

The pod uses a different container runtime than CRI-O

Why wrong: This is incorrect because crictl interacts with the node's configured CRI runtime socket, not with the runtime of any particular pod. Even if a pod is scheduled to use a different runtime class (for example, Kata Containers or gVisor via a RuntimeClass), crictl still lists all CRI containers from the node's primary runtime socket unless you explicitly point it to a different endpoint. The pod's runtime choice has no effect on the connection between crictl and the default CRI socket, so this option does not explain the absence of output.

The containers are in a different namespace

Why wrong: Kubernetes namespaces are logical cluster-scoped objects, not Linux namespaces, and crictl ps displays containers from all Kubernetes namespaces running on that node. Container runtimes do not segregate CRI containers by Kubernetes namespace, so there is no 'namespace filter' that would hide running containers from crictl. If crictl ps shows nothing, the cause is almost always a communication failure with the runtime socket, not containment within a different logical namespace.

ResponseComplete

Why wrong: This stage is triggered only after the entire response body has been transmitted to the client, so it represents the end of the response lifecycle rather than its start. If the goal is to capture the initial response event, this stage is too late because it waits for the full body to finish, missing the timing of the response start. It is more suitable for capturing full response content for audit purposes.

Panic

Why wrong: The Panic stage is a special audit stage that occurs when a panic is recovered during request handling, not as a standard phase in the normal request/response flow. Since it only activates on exceptional errors, it cannot be the stage that captures a normal response start. Therefore it is incorrect for this question because panics are not part of the regular response lifecycle.

RequestReceived

Why wrong: This stage is triggered as soon as the kube-apiserver receives the HTTP request, before any request processing or response generation occurs. At this point, no response exists yet, so it cannot capture any aspect of the response, including the start of it. It is meant for logging incoming requests, not outgoing responses, making it the wrong choice.