Without logs and audits, a security breach in your Kubernetes cluster is essentially invisible — you will never know who broke in, what they took, or how they did it. Monitoring, logging, and auditing are the only way to detect attacks, investigate incidents, and prove compliance with security standards. For the CKS exam, you must understand how to configure these tools so that when something goes wrong, you can find the evidence and fix the hole.
Jump to a section
A simple way to picture Monitoring and Runtime: Logging and Auditing
A hotel night manager is responsible for the security and smooth operation of a hotel while guests sleep. They do not just sit at the front desk; they actively walk the corridors, check that fire doors are closed, and note any unusual sounds or smells. This is the monitoring part — keeping an eye on what is happening right now.
When a guest leaves a wet towel on the floor or a light is left on in an empty room, the manager makes a brief note in the logbook: 'Room 204, towel on carpet, 02:15.' Over the week, these log entries form an audit trail. If the hotel owner later asks why the carpet in Room 204 is mouldy, the manager can point to the logbook and see exactly when the water damage started. The logbook does not prevent the problem, but it records what happened so the owner can trace the cause and hold someone accountable.
For the CKS exam, logging and auditing work the same way. Kubernetes containers are like the hotel rooms — they run applications that generate records (logs) of what they did. The audit log records who accessed the cluster, when, and what they changed. Without these logs, a security incident is like a mysterious stain on the carpet with no record of how it got there.
Let’s break down the three parts of this exam objective: monitoring, logging, and auditing. They sound similar but each serves a different purpose.
Monitoring is watching your cluster in real time to spot problems as they happen. Think of it as a security camera in a shop. You look at the live feed to see if someone is stealing a product right now. In Kubernetes, monitoring tools like Prometheus scrape metrics — numbers that show CPU usage, memory, and network traffic. A sudden spike in CPU might mean an attacker is running a cryptocurrency miner inside a pod. Monitoring alerts you to that spike immediately.
Logging is the permanent record of events that have already happened. Every action a user or application takes can generate a log line — a text entry with a timestamp and details. For example, a containerised web app logs every HTTP request: 'GET /login 200 OK at 14:32:05'. This is useful for debugging, but also for security. If an attacker tries to brute-force passwords, the logs will show hundreds of failed login attempts in a row. You can search logs to find the attacker’s IP address and block it.
Auditing is a specific type of logging for security-relevant actions inside the Kubernetes API server. The API server is the brain of Kubernetes — everything that changes the cluster (creating a pod, deleting a secret, modifying a deployment) goes through it. Audit logging records every request to the API server: who made it (the user or service account), what they did (the verb: create, update, delete), what object they acted on (a pod, a namespace, a role), and the response code (success or failure). This is the evidence for compliance. If someone says 'I never touched that secret', the audit log proves otherwise.
Why does this matter for CKS? The exam expects you to know how to enable audit logging, configure the audit policy (which actions to log), and send logs to a central location. You also need to understand log collection from containers — because by default, container logs are ephemeral (they disappear when the pod is deleted). To persist logs, you must use a logging agent that runs inside the cluster and forwards logs to a storage backend like Elasticsearch or a cloud logging service.
Key components:
kube-apiserver: the gateway for all cluster changes. Audit logs are generated here.
Audit Policy: a YAML file that tells the API server which events to log and at what verbosity level. Levels include Metadata (log only the request metadata, not the body), Request (log metadata + request body), and RequestResponse (log everything).
Logging agent: a sidecar container or DaemonSet (a pod that runs on every node) that reads container logs and sends them to a central storage.
Fluentd and Logstash: popular logging agents that collect, parse, and forward logs.
Elasticsearch: a database that stores logs and makes them searchable. Often used with Kibana (a visual dashboard) — the combination is called the ELK stack.
How it all fits together in a cluster:
A user runs 'kubectl delete secret db-password'.
The kube-apiserver processes the request and, because your audit policy says to log 'delete secret' events, it writes an audit log entry to a file (e.g., /var/log/kubernetes/audit.log).
Simultaneously, the logging agent (running on the control plane node) notices the new audit log file content and forwards it to Elasticsearch.
A security analyst later searches Elasticsearch for 'verb=delete' and 'objectRef.resource=secrets' and finds the record. They see the username 'john.doe@example.com' deleted the secret at 3:15 AM. John gets a phone call.
What this replaces in older IT systems:
Before Kubernetes, system administrators manually configured log rotation and storage on each server. Logs were often kept on the same machine they were generated on, so if the server crashed, logs were lost. Kubernetes standardises log collection using the logging agent pattern, so logs survive pod and node failures. Audit logging is built into the API server, so you do not need third-party tools to track who did what — just configure the policy.
Important exam nuance:
The CKS exam does not require you to set up a full ELK stack from scratch. Instead, you must know how to enable audit logging, write a basic audit policy YAML, and verify that logs are being written. You should also know how to view container logs with 'kubectl logs' and understand that 'kubectl logs' only shows current container output — for historical logs, you need a logging backend.
Security events to watch for:
Unauthorised attempts to delete or modify RBAC (Role-Based Access Control) bindings.
Creation of pods in the kube-system namespace (where system components run).
Access to Secrets (which contain passwords or tokens).
Repeated failed API calls (brute-force detection).
Unusual API calls from a service account that normally only reads logs, not writes to secrets.
Audit logging is your only record of these events. Without it, you are blind.
Enable Audit Logging on the API Server
Edit the kube-apiserver static pod manifest (usually /etc/kubernetes/manifests/kube-apiserver.yaml). Add the flags --audit-log-path=/var/log/kubernetes/audit.log and --audit-policy-file=/etc/kubernetes/audit-policy.yaml. This tells the API server where to write logs and which policy to use. Without these flags, no audit records are generated.
Create the Audit Policy YAML File
Write a YAML file (e.g., /etc/kubernetes/audit-policy.yaml) containing the rules. For example, define a rule that logs all requests to Secrets at level Request. Include a catch-all rule at level None at the end. This policy file is read by the API server every time a request is processed.
Restart the API Server
Because the API server runs as a static pod, simply saving the manifest file triggers an automatic restart. Wait a few seconds, then verify the pod restarted with 'kubectl get pods -n kube-system'. Check the audit log file exists: 'ls -l /var/log/kubernetes/audit.log'.
Test That Logs Are Being Written
Make a real API request, such as 'kubectl get secrets -n default'. Then inspect the audit log file with 'tail -f /var/log/kubernetes/audit.log'. You should see a JSON entry with the verb 'list', resource 'secrets', and the user who made the request. If the file is empty, the policy may be too restrictive or the file path is wrong.
Configure Log Rotation
Add flags --audit-log-maxage=30 (retain log files for 30 days), --audit-log-maxbackup=10 (keep 10 old log files), and --audit-log-maxsize=100 (rotate when file reaches 100 MB). This prevents the audit log from filling the disk. For container logs, configure the kubelet's containerLogMaxSize and containerLogMaxFiles in the kubelet configuration file.
A real IT professional, say a Security Operations (SecOps) engineer at a mid-size e-commerce company, receives an alert from their monitoring system: 'API server request rate spiked 500% in the last 5 minutes.' This is the first sign of a potential attack. She opens Prometheus (monitoring tool) and sees that the spike is coming from a single IP address making thousands of GET requests to the 'secrets' endpoint. This looks like a credential scraping attack.
Here is the step-by-step process she follows:
Check the audit log. She runs a query on the Elasticsearch index for the last 10 minutes, filtering by the suspicious IP. The audit log shows that the requests were made by a service account called 'scraper-bot'. This account does not normally exist in the cluster.
Dig deeper. She looks at the audit log entries for the exact API calls. The entries show 'verb=get' and 'resource=secrets' with a response code of 200 (success) for the first few requests, then 403 (forbidden) for later ones. The attacker successfully read two secrets before the RBAC restriction kicked in.
Determine the blast radius. She examines the audit log to see what objects the service account created. It created three pods in the default namespace, all named 'malware-xyz'. These pods are likely running malicious code. The audit log gives her the exact timestamps and node assignments.
Contain the incident. She uses 'kubectl delete pod' to remove the malicious pods. She also deletes the 'scraper-bot' service account. But she must ensure the attacker cannot come back, so she checks the audit log for any role bindings that might have been altered. She finds that the attacker had created a ClusterRoleBinding (a global permission binding) that gave themselves admin access.
Forensic investigation. She exports the relevant audit log entries and attaches them to the incident report. The compliance team needs these logs to satisfy PCI DSS (Payment Card Industry Data Security Standard) requirements, which mandate audit trails for any access to cardholder data environments. Without audit logs, she would have no evidence to prove which secrets were exposed and when.
Improve defences. She modifies the audit policy to log all access to secrets at the 'RequestResponse' level, so the actual secret value (if exposed) can be seen in logs. She also turns on monitoring alerts for any new service account creation.
Key tools the engineer uses:
kubectl: to query the cluster and delete resources.
audit2rbac: a tool that reads audit logs and suggests RBAC permissions needed.
Prometheus: for real-time metrics.
Elasticsearch + Kibana: for log storage and visualisation.
Sysdig or Falco: runtime security tools that can detect anomalous behaviour like a shell running inside a container (sometimes tested in CKS).
This scenario shows why CKS tests audit logging so heavily. Without it, you cannot investigate, contain, or learn from attacks. The logs are your only source of truth.
The CKS exam tests logging and auditing in a very practical, hands-on way. You will not get multiple-choice questions about theory — you will be given a terminal and a partially configured cluster, and you must fix or improve the logging setup.
Exactly what the exam asks you to do:
Enable audit logging on the kube-apiserver. The typical exam task gives you a control plane node (e.g., 'controlplane' or 'master') and asks you to modify the static pod manifest for the API server. You must add the '--audit-log-path' flag and point it to a file (usually /var/log/kubernetes/audit.log).
Create an audit policy file. You will write a YAML file that defines which events to log. The exam loves to test the rule 'use a specific verb for a specific resource'. For example: 'log all requests to secrets with verb=delete at level RequestResponse'. The trap is that if you set the level too low (e.g., 'None'), events are not logged. If you set it too high (e.g., 'RequestResponse' for namespace listing), you fill the disk with noise. The exam expects you to find the balance.
Verify that audit logs are being written. After editing the API server manifest, you need to check that the audit log file exists and contains entries. Use 'cat' or 'tail' on the file. If the file is empty, the API server might not have restarted or the policy is too restrictive.
Configure container log rotation. The exam might ask you to ensure that container logs do not fill the disk. You must set the '--log-rotate' or configure the kubelet's container log max size and max files options.
Use 'kubectl logs' to retrieve logs from a specific pod. They might ask you to find an error message in a pod's logs and then trace it using the audit log.
Traps the exam sets:
The audit policy file path typo. You write the path to the policy file correctly in the manifest, but the file itself has a syntax error (e.g., missing a dash or an indentation error). The API server will start but logs nothing. The exam will not tell you the error — you must check the API server logs or the container logs of the API server pod.
Only logging successful requests. Beginners often think you should only log 'failure' events. But the exam wants you to log both. For example, logging a 'get' on a secret that returns '200' (success) is critical because it shows the attacker successfully stole data. Logging a '403' shows they tried but failed. Both are important.
Forgetting to restart the API server. On a static pod, updating the manifest file in /etc/kubernetes/manifests automatically restarts the pod. But if you edit the wrong file, the pod does not restart. Always verify with 'kubectl get pods -n kube-system | grep kube-apiserver' and check the AGE column to confirm it restarted.
Using the wrong level for secrets. The exam often asks you to log all requests to secrets. Many candidates set level: 'Metadata' to save space, but 'Metadata' only logs the request header, not the object body. For secrets, you often need the body to see which secret was accessed. The correct level is 'Request' (logs the object metadata plus the request body).
Key exam topics:
Audit policy rules structure (rules array with 'level', 'verbs', 'resources', 'namespaces').
Static pod manifest location for the API server.
Difference between 'kubectl logs' (live log stream) and audit logs (historical API records).
Required flags: --audit-log-path, --audit-policy-file, --audit-log-maxage, --audit-log-maxbackup, --audit-log-maxsize.
The /var/log/pods/ directory for container logs at the node level.
What you must memorise:
The exact YAML structure of an audit policy file (it is a single 'apiVersion: audit.k8s.io/v1' object with a 'rules' array).
The flag format: '--audit-log-path=/var/log/kubernetes/audit.log' (note the equal sign and path).
That 'kubectl logs' does not work after a pod is deleted — logs are lost unless forwarded to a backend.
The levels from least to most verbose: None, Metadata, Request, RequestResponse.
Audit logging must be explicitly enabled by adding the --audit-log-path flag to the kube-apiserver static pod manifest.
Container logs are ephemeral and disappear when the pod is deleted unless forwarded to a persistent backend like Elasticsearch or Cloud Logging.
An audit policy YAML file defines which API server requests to log, using levels from None (no log) to RequestResponse (full request and response).
Use kubectl logs to view real-time container output, but never rely on it for security incident investigation after a pod is gone.
Always include a catch-all rule at level: None as the last rule in your audit policy to prevent logging every trivial request.
The CKS exam tests audit logging by giving you a broken cluster and asking you to fix the API server configuration to enable logging with a custom policy.
These come up on the exam all the time. Here's how to tell them apart.
Audit Logs
Record every API request to the cluster (create, delete, update).
Written by the kube-apiserver to a file on the control plane node.
Used for security and compliance investigation.
Container Logs
Record stdout/stderr output of application processes inside containers.
Written by the container runtime (Docker, containerd) to node-local files.
Used for debugging application errors and behaviour.
Monitoring (Prometheus)
Shows real-time metrics like CPU, memory, request rate.
Triggers alerts when thresholds are crossed (e.g., high error rate).
Data is numerical time-series (numbers at each timestamp).
Logging (Elasticsearch)
Stores event text with timestamps (who, what, when).
Used for post-incident analysis and forensics.
Data is unstructured text (JSON log entries).
Audit Policy Level: Metadata
Logs only the request metadata (timestamp, user, resource type).
Does not include the request body (e.g., which secret name was accessed).
Lower disk usage, good for high-volume read operations.
Audit Policy Level: RequestResponse
Logs metadata plus the full request and response bodies.
Includes sensitive details like secret names or configuration values.
High disk usage, use only for sensitive resources like Secrets.
Mistake
Audit logging is enabled by default in Kubernetes.
Correct
Audit logging is disabled by default. You must explicitly enable it by adding the --audit-log-path flag to the kube-apiserver configuration.
Beginners think Kubernetes 'audits everything' out of the box because other systems like Windows Event Viewer log by default. But Kubernetes is designed to be lightweight, so audit logging is opt-in.
Mistake
kubectl logs gives you historical logs of a pod that was deleted.
Correct
kubectl logs only shows current logs of a running pod. Once a pod is deleted, its logs are lost unless they were forwarded to an external logging backend during the pod's lifetime.
People confuse kubectl logs with 'journalctl' on a Linux system, which does persist logs. Kubernetes does not store container logs permanently by default.
Mistake
If an audit policy rule does not match a request, the request is not logged at all.
Correct
If no rule matches, the request is logged at the default level (which is 'Metadata' if not specified, or 'None' if you set a default rule). The exam expects you to always include a catch-all rule at level 'None' to avoid logging everything.
The rules array works like a firewall: the first matching rule applies. If you forget the last rule (level: None), everything gets logged at the default level, which can overwhelm storage.
Mistake
Container logs and audit logs are the same thing.
Correct
Container logs are stdout/stderr from application processes inside pods. Audit logs are records of API server requests (who did what to the cluster). They are completely separate data streams.
Both are called 'logs', but the exam uses them for different purposes. Mixing them up leads to incorrect configuration.
Mistake
You should set the audit policy level to 'RequestResponse' for all resources to be safe.
Correct
Setting everything to 'RequestResponse' generates enormous volumes of data and will fill the disk quickly. The correct practice is to log only sensitive resources (Secrets, RBAC changes) at high verbosity, and low-verbosity (Metadata) or None for read-only resources.
Newcomers think 'more logging = more security', but excessive logging causes performance issues and makes it harder to find meaningful events.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
'kubectl logs' shows the stdout/stderr output of a running container, useful for debugging application issues. Audit logs record every API request to the cluster, including who made it and what they changed. They serve completely different purposes.
No. Audit logging is built into the Kubernetes API server. You only need to enable it with flags and provide an audit policy file. The logs are written to a local file on the control plane node by default.
Minimal impact for normal traffic. However, setting the level to 'RequestResponse' for every request can cause performance degradation due to the volume of data written. Use selective logging for sensitive resources only.
Audit logs are plain JSON files on the control plane node. Use a log shipper like Filebeat or Fluentd to read the file and forward it to Elasticsearch, Splunk, or a cloud logging service. This is not tested in CKS but is good to know for real-world deployments.
The API server will fail to start or start without logging, depending on the error. Always check the API server container logs for errors: 'kubectl logs -n kube-system kube-apiserver-controlplane'.
No. 'kubectl get events' shows Kubernetes events (like pod schedule failures), not audit logs. Events are a subset of API server activity and are not a substitute for audit logging.
You've finished Monitoring and Runtime: Logging and Auditing. Continue through the CKS study guide to build a complete picture of the exam.
Done with this chapter?