Monitoring and runtime security is the set of tools and processes that watch what your Kubernetes cluster is doing every second, looking for bad behaviour like a guard watching a security feed. It matters for CKS because the exam tests whether you can detect an active attack and respond correctly — not just set up static defences, but react when things go wrong in real time.
Jump to a section
A simple way to picture Monitoring and Runtime: Threat Detection and Incident Response
3:47 AM. Your phone buzzes with a notification from your apartment building’s security system: "Motion detected in the basement hallway — camera 4." You check the live feed on your phone and see a stranger trying the doors to the storage lockers. You have two choices: ignore it and hope it’s nothing, or act immediately.
This is exactly what threat detection and incident response feels like in Kubernetes. Every minute, your cluster (your apartment building) is watched by security cameras (monitoring tools). These cameras aren’t just recording — they are analysing every person who walks by, every door that opens, every unusual pattern. When something looks wrong (a strange person at 3:47 AM), the system sends an alert. You, the security guard (the IT professional), must decide: Is this a false alarm (a sensor glitch) or a real break-in (a cyberattack)?
Just like you would check the live feed, lock down the basement remotely, and call the police, in Kubernetes you would isolate the compromised container, block the suspicious IP address, and analyse forensic logs to understand how the attacker got in. Without this camera-and-alert system, you would never know someone broke in until you found your belongings gone. In Kubernetes, without monitoring and runtime security, you would never know an attacker was stealing data or installing malware inside your containers until it was too late.
Let’s start from the ground up. Kubernetes is a system that runs your applications inside containers. A container is like a lightweight, portable box that holds your code and everything it needs to run. Normally, you trust that nothing bad will happen inside that box. But attackers can break in. They can use a vulnerability in your application to gain access to the container, then use that access to steal data, install malware, or attack other parts of your cluster.
Monitoring means collecting data from your cluster: logs (records of events), metrics (numbers like CPU usage), and traces (records of requests as they travel through services). You use tools like Falco, Sysdig, or the Kubernetes audit log to gather this data. Runtime security means looking at that data as it happens — not after the fact — to spot suspicious activity. For example, if a container suddenly starts running a command like 'curl' to download a file from the internet, that might be an attacker downloading a hacking tool.
Threat detection is the process of identifying that something dangerous is happening. It uses rules or machine learning to decide: ‘Is this normal or abnormal?’ A simple rule might be: ‘If a container tries to write to the /etc/ directory, alert.’ The /etc/ directory is where system configuration files live; normal applications should rarely write there. If one does, it could be an attacker changing settings.
Incident response is what you do after you detect a threat. You must contain the damage, investigate how the attacker got in, and remove them. In Kubernetes, this can mean immediately killing a pod (the group of one or more containers running your application), blocking network traffic to or from a compromised container, or taking a forensic snapshot of the container’s memory for later analysis.
Let’s define the key terms you will see in CKS:
Falco: An open-source tool that sits inside your Kubernetes cluster and watches system calls (the requests a container makes to the Linux kernel). If a container tries to open a network socket or read a sensitive file, Falco can alert you.
Audit log: A record of every request made to the Kubernetes API (the control centre). This includes who tried to create a pod, who deleted a secret, and what command they used. The audit log is the first place to look when investigating an incident.
Runtime security: The practice of monitoring containers while they are running — not just when they start. This catches attacks that happen after the container has been deployed.
Forensics: The process of collecting and analysing evidence after an attack. You might take a memory dump of a compromised container or copy its filesystem to examine it offline.
Incident response plan: A written procedure that says exactly what to do when a threat is detected. Who gets alerted? Which containers get killed first? How do you preserve evidence?
Why does this exist? Before runtime security, organisations relied on static scanning. They would scan container images before deployment to check for known vulnerabilities, but that didn’t catch attackers who exploited a bug after the container was running. Runtime security closes that gap. It watches the live system.
What does it replace? It doesn’t replace static scanning — it adds a second layer of defence. Think of it like a bank: static scanning is checking everyone’s ID at the door, while runtime security is the security guard watching for people who sneak in through a window or who are already inside but acting suspiciously.
For CKS, you need to know how to configure Falco, how to read its alerts, how to use admission controllers (which act like bouncers that check a pod before it is allowed to run) together with runtime detection, and how to perform basic forensic steps such as gathering logs and taking a container memory dump.
Deploy a runtime security tool (e.g., Falco)
Install Falco in your cluster, either as a DaemonSet (one pod per node) or directly on each node. Configure it with a set of rules that define what behaviour is suspicious, such as a container spawning a shell or reading sensitive files. Without this tool, you have no visibility into container behaviour.
Enable Kubernetes audit logging
Configure the API server to send audit logs to a file or a webhook. Define the audit policy to specify which events to log (e.g., all requests to secrets, all pod creations). This gives you a record of who did what in the cluster, which is essential for investigating an incident.
Monitor and detect a suspicious event
When Falco or another tool fires an alert, review the alert details: what command was run, in which container, and at what time. Check the audit log to see if the pod or the user that created the pod looks suspicious. This step determines whether the alert is a real threat or a false positive.
Isolate the compromised resource
Cordon the node to prevent new pods from being scheduled on it. Create a network policy that denies all traffic to and from the compromised pod. Optionally add a taint to the node to drive existing pods away. This stops the attack from spreading while you investigate.
Collect forensic evidence
Use 'crictl' to list running containers and find the container ID of the compromised pod. Use 'crictl export' to copy the container's filesystem, or use 'kubectl exec' to run a memory dump tool inside the container. Save the evidence to a secure location. This preserves data for analysis after the container is terminated.
Terminate the compromised pod and analyse evidence
Delete the pod using 'kubectl delete pod --force'. Offline, analyse the filesystem and memory dump to determine the attack vector, exfiltrated data, and installed malware. Update your incident response plan and security policies based on what you learn.
One morning, an IT professional named Maria gets an alert from Falco: a container running an e-commerce application is executing the shell command 'bash' interactively inside the pod. That is unusual. The application is a Node.js web server — it should never need a shell.
Maria’s incident response plan kicks in. She opens the Kubernetes dashboard and looks at the pod details. She sees that the pod was started 10 minutes ago by a user account called 'jenkins-deploy'. That account should only be used by the CI/CD pipeline to deploy new versions, but Maria checks the audit log and finds that the API request came from an IP address that does not belong to her company’s network.
Step by step, Maria does the following:
She immediately cordons the node where the pod is running. Cordoning means marking the node as unschedulable so no new pods are placed on it. This prevents the attack from spreading to other pods on the same machine.
She creates a network policy that denies all ingress and egress traffic to and from the compromised pod. This isolates it from the rest of the cluster.
She takes a forensic snapshot: she uses the 'kubectl exec' command to run a tool that copies the container’s entire filesystem to a secure storage bucket. She also takes a memory dump by using a tool like 'gcore' inside the container.
She kills the pod by running 'kubectl delete pod <pod-name> --force'. This immediately terminates the running container.
She then analyses the filesystem and memory dump offline. She finds a script that was downloading a cryptocurrency miner and connecting to an external command-and-control server. She shares the indicators of compromise (IOCs) like the IP address and file hashes with her team’s threat intelligence feed.
The next step is to figure out how the attacker got access. Maria reviews the audit log again and sees that the 'jenkins-deploy' service account had been assigned a role with too many permissions. Specifically, it had the 'cluster-admin' role, which gives full control over the entire cluster. The attacker had compromised the Jenkins server and used the stored credentials to create the malicious pod.
Maria then updates the incident response plan to require immediate rotation of all service account tokens when a compromise is suspected. She also implements a policy to never use 'cluster-admin' for CI/CD pipelines — only the minimum permissions needed.
This real-world scenario shows that monitoring and runtime security are not just about tech — they are about process. You need to have a plan, know your tools, and be able to move fast.
The CKS exam tests your ability to apply runtime security in a hands-on, practical way. It is not a multiple-choice theory exam — you will be given a terminal and asked to configure tools, respond to incidents, and fix security issues. Here is exactly what they test and how to prepare.
They love Falco. Expect a question where you must install Falco on a node, configure a custom rule, and test that it fires an alert. The rule might be: 'Alert if a container writes to /etc/shadow' or 'Alert if a container uses the 'apt' command to install packages.' You need to know the Falco rule syntax: rules are written in YAML, and they use fields like 'evt.type' (event type, like 'open' or 'execve') and 'container.id' to specify conditions.
They also test the Kubernetes audit log. You may be given a raw audit log file (a JSON file with a list of 'Event' objects) and asked to find which user created a certain pod or what command they ran. Learn to spot the 'user.username', 'verb' (e.g., 'create', 'delete'), and 'objectRef.resource' fields. A common trap is that the audit log can have a field 'responseStatus.code' — if the code is 403 (Forbidden), the request was denied, so it is not a threat.
Another must-know: admission controllers. You will need to configure an admission controller that validates pods before they run. For example, a custom admission webhook that checks if a container wants to run as root (privileged) and denies it. The exam may ask you to deploy a 'validate' or 'mutate' webhook. Traps often involve forgetting to set the 'failurePolicy' to 'Fail' if you want to enforce the rule strictly, or not handling 'namespaceSelector' correctly.
For incident response, they test the 'kubectl' commands: 'kubectl cordon', 'kubectl drain', 'kubectl taint', and 'kubectl delete'. You may be asked to isolate a compromised node. The trap: if you drain a node without first cordoning it, new pods might be scheduled on it, defeating the purpose. Know the order: cordon first, then drain.
They also expect you to know how to take a forensic image. Use 'crictl' (the CLI for CRI-O or containerd) to inspect containers. For example, 'crictl ps' lists running containers, 'crictl inspect <container-id>' gives details, and you can use 'crictl export <container-id>' to get the filesystem. The trap: they might give you a pod name but you need to convert it to a container ID using 'crictl ps --name <pod-name>'.
Finally, they test your understanding of 'RuntimeClass'. This is a Kubernetes resource that defines a runtime handler (like gVisor or Kata Containers) for extra isolation. They may ask you to create a pod that uses a specific RuntimeClass to run in a sandboxed environment. The trap: you must ensure the RuntimeClass exists in the cluster before referencing it in a pod.
To memorise, focus on:
Falco rule syntax (conditions, output, priority)
Audit log fields (user, verb, objectRef, responseStatus)
Incident response chain (cordon, isolate, forensics, delete)
'crictl' commands vs 'docker' commands (CKS uses containerd, not Docker)
Admission webhook configuration (failurePolicy, sideEffects, timeoutSeconds)
Runtime security watches containers as they run, catching attacks that static image scanning misses.
Falco alerts on system calls like 'execve' or 'open' inside containers, not just network traffic.
The Kubernetes audit log only captures API server requests, not commands run inside a pod.
When a threat is detected, first isolate the node (cordon) and the pod (network policy), then collect forensic evidence, then terminate.
Forensics in Kubernetes requires using 'crictl' or 'kubectl exec' to capture container filesystem and memory before the pod is deleted.
An admission webhook can prevent malicious pods from ever starting, but it does not replace runtime monitoring for attacks that occur after start.
Always rotate service account tokens and review RBAC permissions after a suspected compromise.
The CKS exam expects you to configure Falco, parse audit logs, and use 'crictl' for forensics in a live terminal environment.
These come up on the exam all the time. Here's how to tell them apart.
Runtime Security (Falco)
Watches containers while they run in real time
Catches zero-day attacks and dynamic exploits
Uses rules on system calls and kernel events
Static Security (Image Scanning)
Scans container images before deployment only
Only catches known vulnerabilities (CVEs)
Uses signature databases of known vulnerabilities
Kubernetes Audit Log
Records API server requests (create, delete, update)
Captures user identity and source IP
Structured JSON output, stored centrally
Application Log (stdout/stderr)
Records what the application prints to stdout/stderr
Does not capture who made the request
Unstructured text output, usually in pod logs
Cordon Node
Marks node as unschedulable (no new pods)
Does not evict existing pods
Used to stop further attacks immediately
Drain Node
Evicts all pods from the node gracefully
Moves workloads to other nodes
Used for node maintenance or full shutdown
gVisor RuntimeClass
Runs containers in a sandboxed kernel
Adds an extra security layer for untrusted code
May have performance overhead
Default runc RuntimeClass
Runs containers directly on the host kernel
No extra isolation beyond standard container security
Better performance but less isolation from host exploits
Admission Webhook (Mutating)
Can modify the pod spec before it is created
Used to inject sidecar containers or add labels
Runs before validations
Admission Webhook (Validating)
Only checks if the pod spec meets criteria
Can deny pod creation but cannot modify it
Runs after mutations
Mistake
Once I scan my container images for vulnerabilities, I don't need runtime monitoring — the containers are safe.
Correct
Image scanning only catches known vulnerabilities at build time. Runtime monitoring catches attacks that exploit zero-day vulnerabilities or misconfigurations that happen after the container starts, such as an attacker using stolen credentials to execute commands inside a running container.
Beginners often think security is a one-time check. In reality, attacks happen dynamically, and runtime is the only place you can see active malicious behaviour.
Mistake
If Falco alerts me, the container is definitely compromised and I should immediately kill it.
Correct
Not all Falco alerts indicate a real attack. Many are false positives caused by normal application behaviour (e.g., a legitimate tool that updates its configuration). You must investigate the alert by checking logs and context before taking action.
This mistake is common because beginners see alerts as absolute truth. In real operations, alerts are signals, not verdicts.
Mistake
The Kubernetes audit log records everything that happens inside a pod, including the commands users run in the terminal.
Correct
The audit log only records API server requests — things like creating pods, deleting secrets, or reading configmaps. It does not record shell commands executed inside a container via 'kubectl exec'. For that, you need a runtime tool like Falco or auditd inside the node.
Beginners confuse the API audit log with a system audit log. The names sound similar, but they monitor completely different things.
Mistake
If I set up a network policy that blocks all traffic to a pod, that pod is fully isolated and cannot be attacked.
Correct
Network policies only control network traffic at the Kubernetes layer. If the attacker is already inside the pod (e.g., through a vulnerability in the app), they can still execute code, exfiltrate data by writing to a shared volume, or communicate via host networking if the pod uses 'hostNetwork: true'.
This arises because beginners think network security equals total security. They forget that containers are still isolated by the host kernel, and a compromised process inside the container can still use other channels.
Mistake
Forensics in Kubernetes is as simple as taking a snapshot of the virtual machine the node runs on.
Correct
Containers are ephemeral — they can be killed and recreated automatically by Kubernetes. You must gather forensic evidence from the running container before it disappears, using tools like 'crictl' or 'kubectl exec' to copy out files or memory dumps. Node-level VM snapshots are too slow and may not capture container-level data.
This comes from experience with traditional servers where forensics is done at the VM or physical hardware level. Beginners don't realise that containers are designed to be short-lived and stateless.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Falco monitors system calls (like a process opening a file or executing a command) inside containers, while the Kubernetes audit log records API server requests (like creating a pod or deleting a secret). They cover different layers of security.
No, CKS uses containerd or CRI-O as the container runtime, not Docker. You must use 'crictl' instead of 'docker' commands. For example, 'crictl ps' lists containers, not 'docker ps'.
An admission webhook is a gatekeeper that checks a pod before it is allowed to run (static validation). Runtime detection watches the pod after it starts. A webhook can prevent a malicious pod from entering, but runtime detection catches attacks that exploit a vulnerability after the pod is running.
Check the context: what application is running in the container? Is it a legitimate tool that uses shell commands (e.g., a database backup script)? Review the pod logs and consult with the application owner. If the behaviour matches the app's normal operations, it is likely a false positive.
Isolate the pod by applying a network policy that denies all traffic, then cordon the node to prevent new pods from being scheduled there. Collect forensic evidence before killing the pod. Never kill first, or you lose evidence.
'kubectl cordon' marks a node as unschedulable (no new pods), but existing pods keep running. 'kubectl drain' evicts all pods from the node gracefully. For incident response, use cordon first to stop new pods, then drain if you need to take the node offline.
You've finished Monitoring and Runtime: Threat Detection and Incident Response. Continue through the CKS study guide to build a complete picture of the exam.
Done with this chapter?