EX280 Troubleshooting And Diagnostics — 40 Questions

Question 1

An OpenShift administrator observes that the 'authentication' ClusterOperator is reporting Available=False, Progressing=True, and Degraded=True. During troubleshooting, they notice that the kube-apiserver cannot validate tokens because the OAuth server routes are unreachable due to a misconfigured ingress controller. Which log collection method is best suited to pinpoint the OAuth metadata retrieval failure?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

oc logs -n openshift-authentication deployment/oauth-openshift. The best method is to check the oauth-openshift deployment logs in the openshift-authentication namespace. These logs will contain errors related to OAuth metadata retrieval, such as failures to reach the OAuth server routes. This directly pinpoints the issue described.

Answer

oc get events -n openshift-config-managed

Answer

oc get clusteroperator authentication -o yaml

Answer

oc adm must-gather --image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:...

Question 2

A cluster administrator suspects that an API rate-limiting issue is causing intermittent client timeouts. Which component logs should be inspected to confirm that clients are exceeding their API request limits?

Accepted Answer

oc logs -n openshift-kube-apiserver pod/kube-apiserver-<node-name>. The kube-apiserver logs on the control plane nodes should be inspected to confirm API rate-limiting issues. These logs contain entries about client requests exceeding limits, such as 'rate limit exceeded' messages. This directly confirms the suspected issue.

Answer

oc get events --all-namespaces

Answer

oc logs -n openshift-ingress-operator deployment/ingress-operator

Answer

oc logs -n openshift-kube-apiserver-operator deployment/cluster-kube-apiserver-operator

Question 3

A developer reports that their pod is unable to start because it cannot mount a persistent volume claim (PVC). Where should the administrator look to diagnose volume attachment issues related to the CSI driver?

Accepted Answer

oc describe pvc my-pvc. `oc describe pvc my-pvc` is the correct first stop because the PVC's Events section surfaces the CSI driver's provisioning/attachment failures — including 'failed to provision volume', 'waiting for a volume to be created', or attach-detach controller errors that name the specific CSI driver and node. The PVC status conditions (Pending, Lost, or bound-but-unattached) plus the event stream give the administrator the exact failure reason without needing node-level access.

Answer

oc get events -n openshift-storage

Answer

oc debug node/worker-0

Answer

oc logs -n openshift-cluster-storage-operator deployment/csi-snapshot-controller-operator

Question 4

A pod in the 'production' namespace is crashing repeatedly due to a segmentation fault in the application binary. The administrator needs to inspect the file system of the crashing container without altering the deployment spec permanently. Which command is most appropriate?

Accepted Answer

oc debug pod/my-app-6b79f-abcde. The 'oc debug pod/my-app-6b79f-abcde' command creates a debug pod that copies the target pod's specification but overrides the command to start a shell, allowing the administrator to inspect the file system and troubleshoot without modifying the deployment. This is the standard OpenShift troubleshooting approach for crashing containers.

Answer

oc logs pod/my-app-6b79f-abcde --previous

Answer

oc rsh deployment/my-app

Answer

oc exec -it pod/my-app-6b79f-abcde -- /bin/sh

Question 5

A node in an OpenShift cluster goes into 'NotReady' status. The administrator uses 'oc debug node/<node-name>' to inspect the node's root filesystem. Where are the host's system logs mounted inside the debug chroot environment?

Accepted Answer

/host/var/log. When using 'oc debug node/<node-name>', the host's root filesystem is mounted at /host inside the debug pod, and the host's system logs are accessible at /host/var/log. This is the standard path for inspecting node logs in OpenShift.

Answer

/rootfs/var/log

Answer

/mnt/host/log

Answer

/var/log

Question 6

An OpenShift cluster experiences severe etcd latency spikes, causing leader elections to fail. Which metric or diagnostic tool should be used to inspect etcd peer round-trip times directly from the command line within the cluster?

Accepted Answer

oc rsh -n openshift-etcd pod/etcd-<node-name> etcdctl endpoint health. The etcdctl binary is embedded in the etcd static pods running in the openshift-etcd namespace, and 'etcdctl endpoint health' queries each peer's /health endpoint, returning the round-trip latency to that member. Running it via 'oc rsh' inside the etcd pod gives direct, low-level visibility into peer RTT without relying on aggregated Prometheus metrics. This is the canonical CLI method for diagnosing etcd leader-election failures caused by slow peer communication.

Answer

oc adm inspect etcd

Answer

oc adm diagnostics etcd-latency

Answer

oc get etcd -o yaml

Question 7

An administrator wants to stream logs from all pods belonging to a specific Deployment named 'web-frontend' in real-time. Which command achieves this?

Accepted Answer

oc logs deployment/web-frontend -f. oc logs supports targeting a Deployment directly via the resource type, streaming logs from all associated pods.

Answer

oc watch deployment/web-frontend

Answer

oc logs pods -l app=web-frontend

Answer

oc rsh deployment/web-frontend

Question 8

An administrator needs to check the cluster events across all namespaces sorted by timestamp to identify recent infrastructure warnings. Which command should they use?

Accepted Answer

oc get events --sort-by='.metadata.creationTimestamp'. The correct command is `oc get events --sort-by='.metadata.creationTimestamp'`. This uses the standard `oc get` verb to list all events in the current namespace (or all namespaces if `--all-namespaces` is added), and the `--sort-by` flag sorts the output by the creation timestamp of each event. This directly addresses the requirement to view events sorted by timestamp to identify recent warnings.

Answer

oc adm events --all

Answer

oc show events --recent

Answer

oc cluster-events

Question 9

An administrator needs to tail the logs of all pods with the label 'tier=frontend' across the current namespace. Which command should be used?

Accepted Answer

oc logs -l tier=frontend. The `oc logs` command supports the `-l` (or `--selector`) flag to aggregate logs from all pods matching a label selector in the current namespace. `oc logs -l tier=frontend` streams the logs of every pod with the label `tier=frontend`, which is exactly what the administrator needs. This is the documented OpenShift/Kubernetes pattern for multi-pod log tailing without listing pods individually.

Answer

oc watch logs tier=frontend

Answer

oc logs pods --selector=tier=frontend -f

Answer

oc tail -l tier=frontend

Question 10

An administrator observes that a specific node has high CPU utilization and wants to identify which processes or containers are consuming the most resources on that host directly. Which command-line tool executed inside an oc debug session on the node is best suited for interactive process monitoring?

Accepted Answer

oc debug node/ -- chroot /host top. oc debug node/ -- chroot /host top launches a debug pod on the node, chroots into the host filesystem, and runs top to show live per-process CPU/memory usage on the node itself. This gives interactive, real-time visibility into host-level processes and containers, which is what the question asks for.

Answer

oc exec node/<node-name> -- ps aux

Answer

oc adm monitor node

Answer

oc adm top node --use-protocol-buffers

Question 11

An administrator wants to check the exit code of a container that terminated earlier today in a pod that has since been restarted. Which command displays the previous termination details?

Accepted Answer

oc logs <pod-name> --previous. The --previous flag on oc logs retrieves logs from the previous instance of a container in the pod — i.e., the container that terminated before the current restart. This is the correct way to inspect output from a crashed container, and combined with oc describe pod (which shows Last State and exit code), it gives the full termination picture.

Answer

oc describe pod <pod-name> --history

Answer

oc adm pod-logs --last

Answer

oc get pod <pod-name> -o exit-code

Question 12

A pod is stuck in the 'ContainerCreating' state for an extended period. The administrator suspects a persistent volume mount timeout. Which resource events should they examine to confirm if the volume plugin is failing to attach the disk?

Accepted Answer

oc describe pvc . When a pod is stuck in ContainerCreating due to a volume attach/mount problem, the PersistentVolumeClaim's events record the storage plugin's attach/detach and mount failures. Running `oc describe pvc ` shows the Events section with messages like 'failed to attach volume' or 'MountVolume.WaitForAttach failed', which directly confirm whether the volume plugin is failing to attach the disk. This is the fastest way to correlate the pod's stuck state with the underlying storage error.

Answer

oc adm diagnostics storage

Answer

oc logs deployment/<deployment-name>

Answer

oc get pv -o json

Question 13

An OpenShift cluster upgrade stalled because a MachineConfigPool is paused. The administrator needs to verify which MachineConfigPool is paused and unpause it. Which command accomplishes this?

Accepted Answer

oc get machineconfigpools. Checking the MachineConfigPool resource YAML or status shows the paused state, which can be modified using oc patch or oc edit.

Answer

oc adm upgrade resume

Answer

oc get mcp --show-paused

Answer

oc resume machineconfigpool/worker

Question 14

An administrator needs to troubleshoot an application that fails to resolve internal cluster service names. They decide to run an interactive shell inside a temporary debugging pod with DNS tools installed. Which command creates this diagnostic pod?

Accepted Answer

oc run net-debug --rm -it --image=quay.io/openshift/origin-cli:latest -- /bin/bash. 'oc run' creates a pod from an image and, with --rm -it, attaches an interactive TTY that is deleted on exit — exactly what you want for a throwaway DNS troubleshooting container. The origin-cli image ships with dig, nslookup, and curl, so you can immediately test resolution of internal service names like my-svc.my-namespace.svc.cluster.local. This is the standard pattern for ephemeral network debugging in OpenShift.

Answer

oc debug service/my-svc

Answer

oc adm tcp-dump

Answer

oc rsh net-debug

Question 15

An administrator wants to check the resource requests and limits across all pods in a specific namespace. Which command provides this summary?

Accepted Answer

oc get pods -o custom-columns=NAME:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu. `oc get pods -o custom-columns=...` uses a custom-columns output format to extract the resources.requests.cpu field from each container spec, producing a per-pod summary of CPU requests directly from the API. This is the only option that actually surfaces resource requests and limits across all pods in a namespace in a single command.

Answer

oc get quota

Answer

oc describe namespace my-namespace

Answer

oc adm top pod -n my-namespace

Question 16

An administrator notices that a node is marked as 'SchedulingDisabled' (Cordoned). Which command should be executed to allow pods to be scheduled on this node again?

Accepted Answer

oc adm uncordon . The `oc adm uncordon ` command is the correct way to reverse the cordon operation. When a node is cordoned, the Kubernetes scheduler marks it as unschedulable by setting the `spec.unschedulable` field to `true`. The `uncordon` command explicitly sets this field back to `false`, allowing new pods to be scheduled onto the node. This is the standard administrative command for this purpose in OpenShift, built on top of the Kubernetes API.

Answer

oc patch node <node-name> --type=json -p='[{"op": "replace", "path": "/spec/unschedulable", "value": false}]'

Answer

oc adm drain <node-name> --undo

Answer

oc scale node <node-name> --replicas=1

Question 17

A worker node's root filesystem reaches 100% capacity due to accumulated container images and logs, causing the kubelet to enter an eviction state. After freeing up disk space on the node, the administrator notices the node status remains NotReady. What action is required to clear the kubelet's node pressure condition?

Accepted Answer

Restart the kubelet service on the affected node or reboot the node.. When a node's root filesystem fills up, the kubelet triggers disk pressure eviction and marks the node NotReady. Even after disk space is freed, the kubelet may not automatically clear the pressure condition because its internal state remains stuck; restarting the kubelet service (or rebooting the node) forces it to re-evaluate node conditions and clear the pressure taint, allowing the node to become Ready again. This is a known behavior in OpenShift 4 where kubelet does not always recover from eviction state without a restart.

Answer

Run 'oc adm uncordon' on the node.

Answer

Delete and recreate the clusterversion resource.

Answer

Scale the machineconfigpool to zero and back.

Question 18

An application pod is failing to write data because its mounted PersistentVolume is mounted as ReadOnly. The administrator checks the PVC definition and sees accessMode is set correctly to ReadWriteOnce. What is the most likely cause of the read-only mount?

Accepted Answer

The storage backend experienced an I/O error and remounted the block device as read-only.. When a PVC is correctly set to ReadWriteOnce but the pod still sees a read-only mount, the most likely cause is that the underlying storage backend or node experienced an I/O error and the kernel remounted the filesystem or block device read-only to prevent data corruption. Kubernetes does not automatically change the mount to read-only based on access modes, so the read-only state must originate from the storage layer or kernel. This is a classic symptom of a failed block device or full filesystem triggering ext4/xfs read-only remount.

Answer

The ServiceAccount lacks storage-admin permissions.

Answer

The namespace quota restricts write operations.

Answer

The pod security standard blocked write permissions.

Question 19

An application pod is experiencing intermittent network drops. The administrator wants to capture network packets passing through the pod's interface for analysis. Which tool can be used inside a debugging session attached to the pod namespace?

Accepted Answer

oc debug pod/ -- tcpdump -i any -w /tmp/packet.pcap. The correct command uses 'oc debug' to create a debugging session attached to the pod's namespace, then runs tcpdump inside that session to capture packets on any interface and write to a pcap file. This is the standard OpenShift way to capture network traffic for a pod without installing additional tools on the node. The 'oc debug pod/' command creates a copy of the pod with a shell, and the '--' passes the tcpdump command to run inside.

Answer

oc network capture <pod-name>

Answer

oc exec pod/<pod-name> -- wireshark

Answer

oc adm tcpdump pod/<pod-name>

Question 20

An administrator needs to collect logs from all router pods in the 'openshift-ingress' namespace. Which TWO methods can achieve this? (Choose TWO)

Accepted Answer

oc logs -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default -n openshift-ingress. Option A is correct because `oc logs -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default -n openshift-ingress` uses a label selector to aggregate logs from all pods matching the ingress controller label in the openshift-ingress namespace, which is exactly how router pods are identified. Option E is correct because `oc logs deployment/router-default -n openshift-ingress` retrieves logs from all pods belonging to the router-default deployment, which is the standard deployment name for the default Ingress Controller's router pods. Option B is incorrect because `oc describe deployment` only shows deployment metadata, events, and status, not container logs. Option C is incorrect because `oc get routes` lists OpenShift Route objects, which is unrelated to collecting pod logs. Option D is incorrect because `oc rsh` opens an interactive shell into a pod rather than collecting logs.

Answer

oc describe deployment router-default -n openshift-ingress

Answer

oc get routes -n openshift-ingress

Answer

oc rsh -n openshift-ingress deployment/router-default