What Does kubectl Debug Mean?
Also known as: kubectl debug, kubernetes debug, ephemeral containers, cka troubleshooting, kubernetes troubleshooting commands
On This Page
Quick Definition
kubectl debug is a command that lets you add a special helper container to an existing pod so you can inspect what is happening inside. It is like opening a small side door into a running server without disturbing the main service. This is useful when an application is failing and you need to check logs, network settings, or file system contents.
Must Know for Exams
The CNCF Certified Kubernetes Administrator (CKA) exam includes troubleshooting as a major topic. According to the official CKA curriculum, approximately 30% of the exam covers troubleshooting of cluster components, nodes, and applications. kubectl debug appears in the exam objectives under the section 'Troubleshoot application failures, misconfigurations, and connectivity issues'.
Candidates are expected to know how to debug pods that are crash looping, failing readiness probes, or experiencing network issues. The exam does not test theory alone but requires hands-on skills in a live terminal environment. You might be given a pod that is running but failing to serve traffic, and you need to diagnose why.
kubectl debug is often the most efficient tool for this because it allows you to add a container with networking tools to check DNS resolution, connectivity to other services, or file system permissions. The CKA exam also tests your ability to use ephemeral containers correctly. You need to know the syntax: kubectl debug pod-name --image=busybox -- copy-to=debug-pod or kubectl debug pod-name -it --image=ubuntu --share-processes.
You may also be asked to debug a node-level issue, such as a NodePort service not working, and kubectl debug can help you create a temporary pod on a specific node to test connectivity. Beyond CKA, the Certified Kubernetes Application Developer (CKAD) exam also references debugging in the context of application troubleshooting, though it focuses more on pod logs and exec commands. However, kubectl debug is more administrative and therefore more relevant to CKA.
The exam will not ask you to explain the history of the feature, but you must be able to type the correct command under time pressure. Knowing the difference between kubectl debug --copy-to and kubectl debug without cloning is essential. If you accidentally modify the original pod spec or use the wrong flag, you may lose valuable time.
Simple Meaning
Imagine you are in a large office building where each department works in a closed room. One day, the finance team says their computer is acting strangely, but you cannot just walk into their room and start looking around because that might interfere with their work. kubectl debug is like having a special pass that lets you enter the room through a separate, temporary door, look at the computer screen, check the cables, and run a few tests, all without touching the employee's keyboard or disrupting their workflow.
In Kubernetes, every application runs inside a container, which is like a sealed box containing everything the app needs. If something goes wrong, you might need to see what is inside that box, but the box is designed to run only one application and nothing else. kubectl debug allows you to inject a second, temporary container into the same box.
This debug container shares the same network, storage, and process space as the original application, so you can run diagnostic commands like checking network connectivity, inspecting files, or testing DNS resolution. You are not changing the application or stopping it. You are just adding a temporary observer that can look around and report back.
Once you finish, you remove that temporary container, and the original application continues running as if nothing happened. This is a safer and more isolated way to troubleshoot compared to installing debugging tools inside the production container itself, which could cause instability or security issues. The command is part of Kubernetes version 1.
20 and later, and it is a standard tool used by cluster administrators and developers to solve problems without breaking anything.
Full Technical Definition
kubectl debug is a built-in kubectl command introduced in Kubernetes v1.20 (alpha) and promoted to beta in v1.23. It is used to create ephemeral containers inside running pods for debugging purposes.
Ephemeral containers are a special type of container that are temporary, do not restart on failure, and are not part of the pod's original specification. They are intended for interactive troubleshooting sessions. When you run kubectl debug, the Kubernetes API creates an EphemeralContainers subresource under the target pod.
This subresource is part of the Pod spec but is treated separately from regular containers and init containers. The debug command can operate in two primary modes: copying a pod or attaching to an existing pod. In the copy mode, kubectl debug creates a new pod that is a clone of the target pod but with a modified container image that includes debugging tools.
This is useful when the original pod's image is minimal and lacks utilities like curl, ping, or netstat. In the attach mode, kubectl debug adds an ephemeral container directly to the running pod. This ephemeral container can use a custom image with debugging tools, and it shares the pod's network namespace, IPC namespace, and optionally the process namespace if shareProcessNamespace is enabled.
The ephemeral container's lifecycle is tied to the pod. If the pod is deleted, the ephemeral container is also removed. The kubectl debug command supports several flags, including --image to specify the debug container image, --target to select a specific container within the pod, and --copy-to to create a copy pod with a new name.
It also supports --share-processes to share the process namespace, which allows the debug container to see all processes running in the other containers. Under the hood, kubectl debug uses the Kubernetes API to patch the pod's spec with an ephemeral container definition. The cluster must have the EphemeralContainers feature gate enabled, which is default in most modern distributions.
The container runtime, such as containerd or CRI-O, must support ephemeral containers, which most do as of Kubernetes 1.23. Once the ephemeral container is running, you can exec into it using kubectl exec to run shell commands.
The ephemeral container's stdout and stderr are not logged to the pod's logs by default, but you can capture them manually. This tool is critical for debugging production issues because it avoids modifying the original application container, reducing the risk of configuration drift or security breaches.
Real-Life Example
Think of a large apartment building with many units. Each unit has a single door and a single resident inside. One day, one resident reports that their internet is not working, but you cannot just barge into their apartment and start rearranging their furniture or unplugging cables because that would be invasive and might break things further.
Instead, the building manager uses a special system. They slide a small, temporary service panel into the wall outside the unit. Inside that panel is a set of diagnostic tools: a network tester, a signal meter, and a cable scanner.
The panel is connected to the resident's internal wiring but does not require entering their apartment. The technician can use these tools to check whether the problem is inside the unit or in the building's main network. Once the test is done, the panel is removed, and the resident's home remains untouched.
This is exactly how kubectl debug works. The production pod is the apartment, the application container is the resident, and the debug container is that temporary service panel. You inject a short-lived container that shares the same network and storage as the application, but you do not change any files inside the application container itself.
You run your checks, collect data, and then remove the debug container. The application continues to run exactly as before. This approach is far safer than editing the production container to install a debugger or logging into the container itself, which might accidentally alter its state.
Why This Term Matters
In real IT work, especially in cloud-native environments, applications run in containers that are intentionally stripped down to reduce size and attack surface. Containers often run with minimal images that do not include bash, curl, netstat, ping, or other traditional debugging utilities. When a service goes down or behaves unexpectedly, engineers need a way to inspect the pod's network, file system, and processes without compromising the security or stability of the production environment.
kubectl debug solves this problem by allowing you to add a temporary container with all the tools you need. This is far better than the older approach of SSHing into a virtual machine or modifying a Dockerfile to include debugging tools, both of which introduce risks. Another reason this matters is incident response speed.
When a critical microservice is failing, minutes matter. Instead of rebuilding the pod with a debug image and redeploying, which can take time and might change the pod's IP address or volume mounts, kubectl debug works on the running pod instantly. You can inspect the actual state of the failing application as it happens.
This is crucial for diagnosing transient issues like network timeouts, disk space problems, or misconfigured environment variables. Furthermore, kubectl debug supports ephemeral containers that are visible to audit logs and Kubernetes RBAC, so you can control who is allowed to debug which pods. This provides a secure, auditable debugging workflow suitable for regulated industries.
In summary, kubectl debug is a fundamental troubleshooting tool for any Kubernetes administrator or developer. It reduces risk, saves time, and gives you a clear window into the inner workings of your applications without breaking them.
How It Appears in Exam Questions
Exam questions that involve kubectl debug generally fall into three categories: scenario-based debugging, configuration correction, and pod inspection. In a scenario-based question, you are told that a pod named web-frontend is running but not responding to HTTP requests. You are asked to identify the cause.
The correct approach is to use kubectl debug to add a container with curl and test connectivity to the service endpoint from within the pod's network. For example, you might run kubectl debug pod/web-frontend -it --image=curlimages/curl -- curl localhost:8080 to see if the application responds on the expected port. Another common pattern is a pod that crashes immediately after startup.
The exam might present a CrashLoopBackOff state, and you need to check the file system or environment variables before the pod exits. You could use kubectl debug to create a copy of the pod with a command that keeps the container alive, like sleep 3600, and then exec into that copy to inspect the configuration. Configuration correction questions might require you to edit a pod manifest to include a debug init container, but the exam expects you to know the kubectl debug command as a faster alternative.
Architecture questions are less common, but you might be asked how ephemeral containers differ from regular containers in terms of lifecycle and resource limits. Sometimes the exam presents a trick: a candidate tries to use kubectl exec to run commands inside a container that lacks a shell, resulting in an error. The correct solution is to use kubectl debug to attach a container with a shell.
Another tricky question involves pod security policies or security contexts that prevent attaching debug containers. You need to understand that ephemeral containers are subject to the same PodSecurityPolicy as the pod, so you may need to adjust permissions or use a copy pod with relaxed security settings. The exam may also test your knowledge of the --target flag to debug a specific container in a multi-container pod.
In that case, you would use kubectl debug pod/my-pod --target=my-container. Questions may also combine kubectl debug with other commands like kubectl logs, kubectl describe pod, and kubectl get events to form a complete troubleshooting workflow.
Study cncf-cka
Test your understanding with exam-style practice questions.
Example Scenario
Your company runs a Kubernetes cluster for a customer-facing e-commerce application. One day, the payment service pod named payments-5d4f8b7c6-abc12 starts returning 503 Service Unavailable errors. The pod status shows Running, but the readiness probe is failing.
You suspect the pod cannot connect to the database service. Because the container image is based on a minimal Alpine image, it does not include tools like nslookup or curl. You decide to use kubectl debug to troubleshoot.
You run kubectl debug pod/payments-5d4f8b7c6-abc12 -it --image=busybox -- cp /us/bin/nslookup /tmp/nslookup. Actually, you would use a full debug image. You run kubectl debug pod/payments-5d4f8b7c6-abc12 -it --image=ytanazawa/netutils.
This adds an ephemeral container named payments-5d4f8b7c6-abc12-debug to the pod. You then run nslookup database-service from within the debug container and see that the DNS resolution succeeds. Next, you run telnet database-service 5432 and discover a connection timeout.
This points to a network policy or firewall rule blocking traffic. You report the issue to the networking team, who updates the policy. The readiness probe passes, and the pod begins serving traffic again.
All this was done without modifying the original payment container, without restarting the pod, and without installing any permanent tools.
Common Mistakes
Thinking kubectl debug replaces kubectl exec for all debugging needs.
kubectl exec is used to run a command in an existing container, but if the container lacks a shell or the needed tools, exec will fail or be useless. kubectl debug adds a container with tools, allowing you to run commands there instead.
Use kubectl exec when the container already has the tools you need. Use kubectl debug when the container image is minimal or when you need to avoid modifying the container's state.
Assuming kubectl debug works on pods that are in CrashLoopBackOff state because they start and stop quickly.
Ephemeral containers can only be added to running pods. If a pod crashes immediately, you cannot attach an ephemeral container to it. You must first create a copy of the pod with a command that keeps it alive.
For CrashLoopBackOff pods, use kubectl debug pod-name --copy-to=debug-pod --container=mycontainer -- sleep 3600 to create a copy that stays running, then debug that copy.
Forgetting to specify the --image flag and relying on the default image, which may not have the tools you need.
The default debug image used by kubectl debug might be minimal and lack tools like curl or netstat, making the debugging session ineffective.
Always specify --image with an image that contains the debugging tools you need, such as busybox, ubuntu, or a custom debug image.
Using kubectl debug to add an ephemeral container and then trying to use kubectl logs to see its output.
Ephemeral containers are not part of the pod's log stream by default. Their stdout is not captured in the pod logs. You must use kubectl attach or kubectl exec interactively to see output.
Run kubectl debug with the -it flags and interact with the container directly. If you need to capture output, redirect it to a file within the container or use a sidecar logging approach.
Believing that ephemeral containers can run in any pod regardless of security settings.
Ephemeral containers are subject to PodSecurityContext, PodSecurityPolicy, and RBAC rules. A pod with restricted security contexts may reject ephemeral containers that try to run with privileged capabilities.
Check the pod's security context and, if needed, create a copy pod with relaxed security settings using kubectl debug --copy-to and modifying the spec.
Confusing kubectl debug's --copy-to mode with simply describing a pod or getting its logs.
--copy-to creates a new pod that is a duplicate of the original, which consumes additional resources. Some candidates misuse it when a simple kubectl logs or kubectl describe would suffice.
Use --copy-to only when you need a running instance of the pod with modifications, such as a different command or image. For reading logs or metadata, use kubectl logs or kubectl describe pod.
Exam Trap — Don't Get Fooled
The exam presents a scenario where a pod is in 'CrashLoopBackOff' state, and the candidate is asked to debug it. The candidate runs kubectl debug pod-name without any flags, expecting to add an ephemeral container to the crashed pod. Remember that ephemeral containers can only be added to pods that are in the 'Running' state.
For CrashLoopBackOff pods, you must use the --copy-to flag to create a copy that you can modify to stay alive (e.g., changing the command to sleep). In the CKA exam, always check the pod status first with kubectl get pods.
If the pod is not Running, use --copy-to with a custom command.
Commonly Confused With
kubectl exec runs a command inside an existing container within a pod. It does not add a new container. If the original container lacks debugging tools, exec is useless. kubectl debug adds a whole new container with the tools of your choice, sharing the same network and volumes.
If you need to check DNS inside a pod that only has a binary and no shell, kubectl exec -it pod-name sh will fail. kubectl debug pod-name --image=busybox will give you a shell with nslookup.
kubectl logs retrieves the stdout and stderr output of a container. It is for reading already-written logs, not for interactive troubleshooting. kubectl debug is for live inspection and running commands inside the pod's environment.
If an application prints errors, kubectl logs will show them. If you need to test connectivity to a database, kubectl logs cannot help, but kubectl debug can.
kubectl run creates a brand new pod, unrelated to any existing workload. It does not attach to a running application or share its network. kubectl debug either attaches to an existing pod or creates a copy of it, preserving its configuration.
To test network policies, you could use kubectl run test-pod --image=busybox -- sleep 3600, but this pod is separate. To test connectivity from the exact same network as a failing pod, kubectl debug is better because it shares the pod's network namespace.
kubectl describe pod provides metadata and status information about a pod, like events, conditions, and container statuses. It does not allow any interactive debugging. kubectl debug is a hands-on tool for when the information from describe is insufficient.
If describe shows a pod is failing due to a readiness probe failure, you still need kubectl debug to explore why the probe is failing by checking the application's actual response.
Step-by-Step Breakdown
Check Pod Status
Use kubectl get pods to see the current state of the pod. Is it Running, CrashLoopBackOff, or Pending? This determines whether you can attach an ephemeral container or need to create a copy. If the pod is not Running, you will use the --copy-to flag.
Choose Debug Mode
Decide between attach mode (add ephemeral container to the running pod) and copy mode (clone the pod with modifications). Attach mode is for live pods that are running but misbehaving. Copy mode is for pods that crash on startup or have restrictive security contexts.
Select the Debug Image
Pick a container image that includes the debugging tools you need. Common choices are busybox, ubuntu, curlimages/curl, or nicolaka/netshoot. Use the --image flag to specify it. If you need specific tools, consider a custom image you have access to.
Run the kubectl debug Command
Execute the command. For attach mode: kubectl debug pod/my-pod -it --image=busybox. For copy mode: kubectl debug pod/my-pod --copy-to=my-debug-pod --container=my-container -- image=busybox -- sleep 3600. The -it flag makes the session interactive.
Execute Diagnostic Commands
Once inside the debug container, run commands to diagnose the issue. For example, use nslookup to test DNS, curl to test HTTP endpoints, ping to test basic connectivity, df to check disk space, and ps to see processes. Share the process namespace if needed with --share-processes.
Analyze Output and Fix the Issue
Based on the diagnostic output, identify the root cause. It could be a DNS misconfiguration, a network policy blocking traffic, a missing volume mount, or an environment variable error. Apply the fix to the original deployment or pod spec, not the debug container.
Clean Up the Debug Container
After debugging, exit the container. The ephemeral container is automatically removed when you exit. If you used copy mode, delete the copy pod with kubectl delete pod my-debug-pod to free resources and avoid confusion.
Practical Mini-Lesson
kubectl debug is more than just a command; it is a design pattern for safe troubleshooting in production environments. The core principle is isolation: never modify a running production container if you can avoid it. Instead, create a temporary environment that mirrors the same context.
In practice, you should have a set of prepared debug images stored in your container registry. These images should contain common networking tools, file system utilities, and scripting languages. For example, a netshoot image might include curl, jq, dnsutils, tcpdump, and vim.
Having these images pre-pulled to your cluster nodes speeds up debug session startup. When debugging, consider the following typical failure modes. First, application crash loops: the pod restarts repeatedly.
Use kubectl debug with --copy-to and change the command to sleep 3600. Then inspect the container's file system, environment variables, and config maps. Second, network issues: a pod cannot reach a service.
Attach an ephemeral container with networking tools and test connectivity from within the pod's network namespace. Third, configuration errors: the application reads the wrong configuration. Use kubectl debug to mount the same volumes and look at the configuration files directly.
Fourth, resource exhaustion: the pod is killed due to memory limits. Use kubectl debug with a tool like top or free to see resource usage from within the pod's cgroup. One common pitfall is forgetting that ephemeral containers do not inherit the pod's resource limits.
They are subject to their own resource requests and limits, which default to no limits. This could cause resource contention. You can set requests and limits using the --requests and --limits flags.
Another point is security. Ephemeral containers can be a vector for privilege escalation if not controlled. Use Kubernetes RBAC to restrict who can create ephemeral containers. The kubectl debug command requires the pods/ephemeralcontainers resource permission.
You can assign this to specific users or service accounts. Finally, in a GitOps or CI/CD workflow, kubectl debug can be integrated into automated debugging pipelines. For example, a failed deployment could trigger a script that creates a debug pod, gathers logs, and deletes the pod.
This is useful for post-mortem analysis. The broader concept connects to observability. While kubectl debug is great for interactive troubleshooting, it should be complemented with proper logging, monitoring, and tracing to reduce the need for manual intervention in the first place.
Memory Tip
Think 'D.E.B.U.G.' - Don't Enter a Box, Use a Guest. You are not entering the application container; you are adding a guest container that can look around.
Covered in These Exams
Related Glossary Terms
802.1Q is the networking standard that allows multiple virtual LANs (VLANs) to share a single physical network link by tagging Ethernet frames with VLAN identification information.
802.1X is a network access control standard that authenticates devices before they are allowed to connect to a wired or wireless network.
5G is the fifth generation of cellular network technology, designed to deliver faster speeds, lower latency, and support for many more connected devices than previous generations.
An A record is a DNS record that maps a domain name to the IPv4 address of the server hosting that domain.
Frequently Asked Questions
What is the difference between kubectl debug and kubectl exec?
kubectl exec runs a command inside an existing container. kubectl debug adds a new temporary container to the pod. Use exec when the container has the tools you need. Use debug when the container image is minimal or you want to avoid touching the application container.
Can I use kubectl debug on a pod that is not running?
No, ephemeral containers can only be added to running pods. For pods in CrashLoopBackOff or Pending state, use the --copy-to flag to create a copy of the pod with a modified command that keeps it alive.
What image should I use for kubectl debug?
Use an image that contains the debugging tools you need. Common choices include busybox, ubuntu, nicolaka/netshoot, or curlimages/curl. You can also use a custom image stored in your private registry.
Does kubectl debug work in all Kubernetes versions?
kubectl debug was introduced in Kubernetes v1.20 (alpha) and became beta in v1.23. It is enabled by default in most modern distributions. Check that your cluster has the EphemeralContainers feature gate enabled.
How do I see the output of an ephemeral container?
Ephemeral containers are not logged to the pod's log stream by default. Use kubectl attach or run the debug session interactively with -it to see output in real time. You can also redirect output to a file inside the container.
Is kubectl debug safe for production use?
Yes, it is designed for production use. It isolates the debugging activity from the application container, reducing risk. However, you should control access via RBAC and use secure debug images to avoid introducing vulnerabilities.
Can I debug a specific container in a multi-container pod?
Yes, use the --target flag followed by the name of the container you want to debug. For example: kubectl debug pod/my-pod --target=my-container -it --image=busybox.
Summary
kubectl debug is a powerful command in Kubernetes that lets you add a temporary, isolated container to a running pod for troubleshooting purposes. It solves the common problem of debugging minimal container images that lack standard tools like curl, ping, or netstat. Instead of modifying the production container or redeploying with a different image, you inject a debug container that shares the pod's network, storage, and process namespace.
This provides a safe, auditable, and efficient way to diagnose application failures, network issues, configuration errors, and resource problems. For certification exams like the CKA, knowing how to use kubectl debug in both attach mode and copy mode is essential. You must understand when to use each mode, how to choose the right debug image, and how to interpret the results.
Common pitfalls include using the command on non-running pods without the --copy-to flag, forgetting to specify an appropriate image, and confusing it with kubectl exec. Remember the memory tip 'D.E.
B.U.G.' to reinforce the concept of adding a guest container rather than entering the application container. With kubectl debug, you can solve real-world Kubernetes problems faster, safer, and with minimal disruption to your services.