Troubleshooting Scenario Questions

The API server service is missing endpoint IPs.

Why it fails: A missing kubernetes API service endpoint breaks in-cluster API access, not external DNS forwarding, since CoreDNS forwards external queries to upstream resolvers configured in the Corefile. It tempts because endpoint failures do disrupt cluster services, but internal resolution here already works, ruling it out.

NodePort range exhaustion prevents external DNS packets from traversing worker nodes.

Why it fails: NodePort exhaustion affects inbound service exposure on the 30000-32767 range; CoreDNS forwards external queries outbound via the forward plugin, which uses ephemeral source ports, not NodePorts. It tempts because port exhaustion does cause connectivity failures, but it cannot selectively block external DNS while internal resolution succeeds.

The cluster ingress controller is blocking external UDP port 53.

Why it fails: Ingress controllers handle inbound HTTP/HTTPS traffic to cluster services; they do not intercept pod egress DNS queries, which traverse the pod network directly to CoreDNS. It tempts because ingress misconfiguration does break name-based access, but external DNS resolution from pods bypasses ingress entirely.

oc get routes

Why it fails: Routes expose ingress hostnames, not operator installation state; OLM failures surface in the Subscription, ClusterServiceVersion and InstallPlan resources. Route inspection is tempting because operators with webhooks or console plugins publish routes, so it fits connectivity debugging once the operator is already running.

oc get pv

Why it fails: PersistentVolumeClaims are storage resources unrelated to OLM reconciliation of an operator installation. It is tempting because failing operators sometimes stall on storage, but OLM diagnostics require the Subscription, ClusterServiceVersion, and InstallPlan or operator pod, which expose the actual installation state.

Delete all pending MachineConfig objects from the cluster.

Why it fails: Deleting pending MachineConfig objects discards the desired configuration rather than deferring it, so the changes are lost and never applied. It is tempting when an administrator wants an immediate halt, but pausing the MachineConfigPool preserves the objects and applies them once troubleshooting finishes.

Scale down the machine-config-operator deployment in openshift-machine-config-operator to zero replicas.

Why it fails: Scaling the operator deployment to zero stops reconciliation entirely, so the operator cannot report status or resume cleanly, and the cluster marks it degraded. It is tempting as a blunt stop, but the supported mechanism is pausing the specific MachineConfigPool, leaving the operator running and other pools unaffected.

Annotate all worker nodes with 'machineconfiguration.openshift.io/paused=true'.

Why it fails: The pause annotation is applied to a MachineConfigPool, not to individual nodes, so annotating worker nodes has no effect on the Machine Config Operator's rollout. It is tempting because node annotations commonly influence other operators, but only the pool-level annotation halts updates to that pool.

The cluster image registry configuration

Why it fails: The cluster image registry configuration governs image pull and push behaviour, not PersistentVolumeClaim binding or attachment, so describing it exposes registry storage settings rather than the failing volume's events. It is tempting because registry storage itself uses PVCs, and would be correct when diagnosing registry pod storage or image pull failures.

The OAuth client credentials secret

Why it fails: OAuth client credentials secrets authenticate API service accounts, not storage volumes, so 'oc describe' on them reveals token metadata rather than PersistentVolumeClaim binding events or attachment errors. It is tempting because secrets are inspectable resources, and would be the right target when debugging authentication failures for an operator or OAuth-protected route.

The default Kubernetes events in the default namespace.

Why it fails: Kubernetes events are namespaced, short-lived and typically expire after an hour, so they rarely capture the operator's reconciliation errors for a CR. Events suit transient scheduling or image-pull failures; the CR's own status conditions persist the controller's reported error state.

The global clusterversion object status conditions.

Why it fails: ClusterVersion status conditions report the Cluster Version Operator's own reconciliation of OpenShift updates, not the operand CR's controller loop. It is tempting because ClusterVersion aggregates cluster operator health, but the specific CR's error messages live in that CR's status field.

The /var/log/messages file on the master nodes.

Why it fails: Node syslog captures kubelet and container runtime output, not the operator controller's reconciliation result for a CR. It is tempting when pods crash, but controller-loop errors are written to the CR's status conditions, not to /var/log/messages on masters.

Check the MachineConfigPool for paused reconciliation flags.

Why it fails: MachineConfigPool paused flags affect node configuration rollout, not OLM CSV installation. It is tempting because paused reconciliation stalls cluster updates, but the relevant object is the ClusterServiceVersion or InstallPlan, which records why the Operator cannot install in the restricted namespace.

Review the ClusterVersion resource status for pending updates.

Why it fails: ClusterVersion reports the cluster's own update channel and version state, not Operator installation. The CSV and InstallPlan hold the OLM resolution error caused by the AllNamespaces mode clashing with the restricted target namespace. ClusterVersion is the correct resource when diagnosing stalled OpenShift platform upgrades or version reconciliation.

Inspect the KubeletConfig custom resource for validation errors.

Why it fails: KubeletConfig governs node-level kubelet parameters, not OLM resolution. The Subscription's InstallPlan and CSV status carry the resolution failure, so inspecting node configuration cannot reveal why the AllNamespaces mode conflicts with the restricted target namespace. KubeletConfig is the right resource when tuning kubelet settings such as pod limits or eviction thresholds on nodes.

bob needs an explicit RoleBinding for the 'admin' role in the default namespace

Why it fails: An 'admin' RoleBinding in the default namespace grants full control of resources inside that namespace only; it confers no cluster-scoped permission to create new projects. It tempts because namespace-scoped admin is the usual fix for managing workloads, which is the correct scenario when bob needs to administer one existing project.

bob's user object is missing the 'project-creator' annotation

Why it fails: User objects carry no 'project-creator' annotation; project creation is governed by the self-provisioner cluster role binding, which bob lacks. The tempting idea of per-user annotations reflects how labels and annotations tag resources, but authorisation in OpenShift flows exclusively through RBAC roles and bindings, never annotations.

bob must be added to the cluster-admin group in the oauth configuration

Why it fails: Cluster-admin grants unrestricted rights cluster-wide, far exceeding what project creation requires, and group membership is managed in the identity provider rather than the oauth configuration. It tempts because cluster-admin does enable 'oc new-project', making it the right answer only when a user genuinely needs full cluster control.

Run 'oc adm prune crds --force' to automatically purge all stuck custom resources.

Why it fails: No such subcommand exists; 'oc adm prune' handles images, builds and deployments, not CRDs, so the command fails outright. It tempts administrators wanting a single forceful purge, but orphaned CRDs must be removed individually with 'oc delete crd', after stripping the OLM owner reference.

Restart the kube-apiserver pods to bypass CRD validation checks.

Why it fails: Restarting kube-apiserver pods does not alter stored CRD objects or their conflicting versions; the API server revalidates the same etcd data on startup. It tempts those seeking to clear transient API errors, but restarting control-plane components is for recovery from crashes, not for removing orphaned CRDs.

Delete the openshift-operator-lifecycle-manager namespace to reset all CRDs.

Why it fails: Deleting that namespace removes OLM's own controllers and subscriptions, not cluster-scoped CRDs, which survive namespace deletion. It tempts administrators wanting a full reset, yet reinstalling OLM leaves the same conflicting CRDs, so the CSV stays Pending; individual CRD deletion is required.

ClusterRoles must be defined inside the same project request template YAML file.

Why it fails: ClusterRoles are cluster-scoped objects resolved from the cluster, not embedded within the template YAML; defining one inside the template does not create it. It is tempting because templates bundle namespaced objects, so administrators assume every referenced object must be declared in the same file — correct for namespaced Roles, not ClusterRoles.

Project request templates do not support RoleBindings; only ResourceQuotas are allowed.

Why it fails: Project request templates do support RoleBindings alongside ResourceQuotas and LimitRanges; the failure stems from ClusterRole resolution, not from RoleBindings being unsupported. It is tempting because ResourceQuotas are the most familiar template content, so administrators assume templates are limited to quota objects — true only when no RoleBinding is defined.

The RoleBinding must use a Namespaced Role instead of a ClusterRole.

Why it fails: A RoleBinding may legitimately reference a ClusterRole via roleRef; substituting a namespaced Role is unnecessary and would restrict permissions to that namespace. It is tempting because the missing ClusterRole appears to be a scoping problem, so switching to a Role seems to avoid it — yet the real issue is that the ClusterRole must already exist in the cluster.

Switch the upgrade channel to 'fast-4.14' to bypass the degraded check.

Why it fails: Changing channels alters which versions are offered; it does not clear a Degraded=True ClusterOperator, and the new channel still blocks upgrades while the operator is unhealthy. Channels suit selecting update streams. The operator's fault must be diagnosed and resolved first.

Delete the Cluster Version Operator deployment to force a reset of the upgrade state machine.

Why it fails: Deleting the Cluster Version Operator destroys the component that reconciles ClusterVersion state and reports operator conditions; it cannot clear a Degraded ClusterOperator, which needs its own fault diagnosed. It is tempting because forcing a reset can clear stuck states, but that applies to the CVO's own reconciliation, not a degraded operator.

Force the upgrade by running 'oc adm upgrade --to=<version> --force-ignore-degraded'.

Why it fails: The force flag overrides the ClusterVersion's degraded gate but leaves the faulty ClusterOperator unrepaired, risking a partially upgraded, unsupported cluster. It suits recovering a stuck upgrade after the underlying fault is fixed. Here the operator's error must be investigated and corrected first.

Reinstall the entire cluster storage operator.

Why it fails: Reinstalling the storage operator addresses persistent volume provisioning, which is unrelated to whether a Route's hostname resolves or its backend service answers. It is tempting because reinstallation feels like a clean reset, but it would be correct when storage classes or volume claims are failing, not ingress routing.

Modify the cluster kube-apiserver certificates.

Why it fails: Route resolution depends on the ingress controller, router shards and DNS, not on API server serving certificates; regenerating those certificates would disrupt the control plane without fixing the Route. It is tempting because certificate errors do break TLS, and it would be correct when the API server's own serving certificate has expired.

Run 'oc adm diagnostics storage'.

Why it fails: oc adm diagnostics storage checks cluster-level storage configuration and node health, not the external provisioner's controller output. It is tempting because it is the standard OpenShift troubleshooting command, and would be correct for diagnosing a misconfigured storage class or unhealthy CSI node plugin.

Inspect the container image registry logs.

Why it fails: The container image registry stores images, not provisioner controller output, so its logs contain nothing about PVC provisioning failures. It is tempting because registry logs are a familiar troubleshooting source, and would be correct when image pulls or pushes fail during a deployment.

Check /var/log/messages on the control plane nodes.

Why it fails: External provisioners run as pods, so their messages go to container stdout, not the control plane node's /var/log/messages. It is tempting because that file is the traditional place for system daemon errors, and would be correct for diagnosing kubelet or CRI-O failures on a node.

oc cluster-version approve-csr <csr-name>

Why it fails: No such subcommand exists under oc cluster-version; that command manages cluster version upgrades, not certificate approval. It is tempting because it sounds administrative, but CSR approval belongs to the certificates API, where the correct verb operates on the csr resource.

oc patch csr <csr-name> --type=json -p='[{"op": "replace", "path": "/status/conditions", "value": "Approved"}]'

Why it fails: Patching status conditions directly bypasses the approval subresource and is rejected by the API server, which controls status writes. It is tempting because JSON patches can mutate many resources, but CSR approval must go through the dedicated approve action rather than a raw status replace.

oc approve csr <csr-name>

Why it fails: oc approve is not a valid subcommand; approval is exposed as a subresource action on the csr resource, not a top-level verb. It is tempting because the wording matches the task, but the actual syntax requires the certificates API group and the approve subresource.

The PersistentVolumeClaim is stuck in Pending.

Why it fails: A Pending PersistentVolumeClaim concerns storage binding and blocks pod scheduling, not image retrieval. It is tempting because Pending pods often signal storage problems, and this is the right check when pods stall before scheduling due to unavailable or unbound volumes.

The route hostname is invalid.

Why it fails: Route hostnames govern OpenShift ingress routing of external HTTP traffic to services; they play no part in a kubelet pulling an image from a registry. Checking routes is correct when external clients cannot reach a published application, not when image pulls fail.

The liveness probe timeout is configured too short.

Why it fails: A short liveness probe timeout causes containers to be restarted after they start running; it cannot prevent the kubelet from pulling an image. Probe tuning is correct when containers repeatedly restart despite successful image pulls, not when pulls themselves fail.

/var/lib/etcd/cert

Why it fails: etcd stores the cluster's key-value state, not kubelet credentials; its certificates live only on control-plane nodes, so an oc debug session on a worker finds nothing there. It is tempting because etcd certificates are also X.509 files that expire, making etcd the right target when diagnosing control-plane certificate failures rather than kubelet ones.

/etc/origin/node/certificates

Why it fails: This path belongs to OpenShift 3's node configuration; OpenShift 4 kubelets read client certificates from /var/lib/kubelet/pki, so the expired files are not here. It is tempting because the directory name explicitly references node certificates, making it the correct location when troubleshooting legacy OpenShift 3 worker nodes.

/etc/kubernetes/pki

Why it fails: /etc/kubernetes/pki holds the cluster's control-plane PKI, including the API server and CA certificates, not the kubelet's client credentials. It is tempting because that directory is the standard PKI location on control-plane nodes, but on worker nodes the kubelet client certificate lives in /var/lib/kubelet/pki.