Hard Difficulty Questions

oc adm upgrade status

Why it fails: Cluster upgrade status reports version and channel progress, not node-level packet paths. It is tempting because oc adm commands run against the cluster, but this one inspects ClusterVersion, not interfaces, routes or OVN flows. The scenario needs tools such as ip, tcpdump or ovs-ofctl inside the debug pod.

etcdctl endpoint health

Why it fails: etcdctl endpoint health checks control-plane datastore latency and quorum, which remain healthy even when pod traffic between nodes fails. It is tempting because etcd underpins the cluster, but the fault lies in the data plane, so ip, tcpdump or ovs-ofctl inside the debug session are required.

External identity provider records are automatically deleted from LDAP or OIDC servers.

Why it fails: Deleting an OpenShift User object only removes the cluster-side record; the external LDAP or OIDC directory is never contacted, so provider entries persist. It tempts because identity providers do hold the authoritative credentials, but synchronising deletions back to them is not part of 'oc delete user'.

All namespaces owned or created by the user are permanently deleted.

Why it fails: Namespaces are independent resources with their own lifecycle; removing a User object leaves every project and namespace it created intact. The temptation is that ownership implies cascade, yet OpenShift performs no such garbage collection, so an administrator must delete namespaces explicitly.

All RoleBindings and ClusterRoleBindings referencing the user are automatically deleted.

Why it fails: RoleBindings and ClusterRoleBindings persist after the User object is removed, leaving orphaned subjects that still match a recreated username. It tempts because bindings reference the user by name, but OpenShift does not cascade-delete RBAC objects; they must be cleaned up manually.

The API server service is missing endpoint IPs.

Why it fails: A missing kubernetes API service endpoint breaks in-cluster API access, not external DNS forwarding, since CoreDNS forwards external queries to upstream resolvers configured in the Corefile. It tempts because endpoint failures do disrupt cluster services, but internal resolution here already works, ruling it out.

NodePort range exhaustion prevents external DNS packets from traversing worker nodes.

Why it fails: NodePort exhaustion affects inbound service exposure on the 30000-32767 range; CoreDNS forwards external queries outbound via the forward plugin, which uses ephemeral source ports, not NodePorts. It tempts because port exhaustion does cause connectivity failures, but it cannot selectively block external DNS while internal resolution succeeds.

The cluster ingress controller is blocking external UDP port 53.

Why it fails: Ingress controllers handle inbound HTTP/HTTPS traffic to cluster services; they do not intercept pod egress DNS queries, which traverse the pod network directly to CoreDNS. It tempts because ingress misconfiguration does break name-based access, but external DNS resolution from pods bypasses ingress entirely.

Htpasswd file path

Why it fails: OIDC authenticates against an external issuer using client ID, client secret and issuer URL, so no local htpasswd file is referenced. An htpasswd file path is required only when configuring the htpasswd identity provider, which validates credentials stored in a local file rather than through OIDC token exchange.

LDAP bind password

Why it fails: OIDC delegates authentication to an external identity provider via tokens and a client secret, so no LDAP bind password is collected. The bind password belongs to LDAP identity provider configuration, where OpenShift queries a directory server directly; it would be correct only if the stem specified LDAP rather than OIDC.

Placing the MachineConfig object directly inside the GPU namespace.

Why it fails: MachineConfigs are cluster-scoped resources, not namespaced.

A field in the spec named 'targetPool: gpu-worker'.

Why it fails: There is no 'targetPool' field in MachineConfig specs.

An annotation specifying 'machineconfigpool.openshift.io/target: gpu-worker'.

Why it fails: Targeting is performed via labels defined in the pool's nodeSelector, not annotations.

openshift-storage.rbd.csi.ceph.com

Why it fails: Ceph RBD provides block storage and typically only supports ReadWriteOnce (RWO) in standard configurations.

gp2-csi.storage.k8s.io

Why it fails: General Purpose SSD EBS provisioner only supports RWO.

ebs.csi.aws.com

Why it fails: AWS EBS CSI only supports ReadWriteOnce (RWO).

oc adm upgrade resume

Why it fails: CVO upgrades are managed via clusterversion, not machineconfigpools directly with this command.

oc get mcp --show-paused

Why it fails: There is no --show-paused flag for oc get mcp.

oc resume machineconfigpool/worker

Why it fails: 'oc resume' is not a valid OpenShift CLI verb.

The PVC size is updated in Kubernetes metadata, but physical storage is left unchanged.

Why it fails: The controller updates the resizing status conditions rather than silently desynchronizing metadata and backend.

OpenShift automatically converts the PVC to use a default storage class that supports expansion.

Why it fails: OpenShift never mutates storage classes or migrates PVC data automatically in this manner.

The pod is immediately terminated and restarted on a node with more disk space.

Why it fails: PVC expansion has no relationship to node disk space availability.

HTTP 401 Unauthorized forcing the user to re-authenticate

Why it fails: Quota violations are authorization/admission failures, not authentication failures.

HTTP 200 OK, but the ConfigMap is placed in a pending state until quota frees up

Why it fails: Kubernetes resource creation requests that exceed quotas are rejected immediately rather than queued.

HTTP 504 Gateway Timeout because the controller manager is blocked

Why it fails: Quota checks happen synchronously at admission time, returning immediate rejection, not timeouts.

oc scale deployment/cluster-version-operator --replicas=3

Why it fails: Scaling the cluster-version-operator changes its replica count; it neither reveals why a MachineConfigPool is degraded nor repairs node configuration drift. It is tempting because the operator orchestrates upgrades, but diagnosis requires inspecting MachineConfigPool status, node conditions, and machine-config daemon logs instead.

oc adm upgrade cancel

Why it fails: Cancelling the upgrade aborts the in-progress rollout; it removes the symptom rather than exposing why the MachineConfigPool degraded. It is tempting because a stalled upgrade invites rollback, but diagnosis needs MachineConfigPool conditions, node status, and machine-config daemon logs, not cancellation.

containerRuntime

Why it fails: containerRuntime belongs to the ContainerRuntimeConfig or KubeletConfig resource, not StorageClass, which accepts provisioner, parameters, reclaimPolicy, volumeBindingMode and allowedTopologies. It is tempting because runtime settings do govern how pods execute, but that configuration is applied to nodes, not to dynamic provisioning of persistent volumes.

nodeSelector

Why it fails: nodeSelector is a Pod spec field controlling scheduling, not a StorageClass parameter; StorageClass uses allowedTopologies to constrain where volumes provision. It is tempting because both restrict placement, yet nodeSelector targets pod scheduling while allowedTopologies filters topology zones during dynamic provisioning.

When the PVC object is successfully created in the namespace.

Why it fails: With WaitForFirstConsumer, the PersistentVolume is provisioned only when a pod using the PVC is scheduled, not at PVC creation. The PVC-creation answer is tempting because dynamic provisioning normally triggers then, but delayed binding defers it until scheduling reveals the topology.

When the user executes 'oc adm pv create'.

Why it fails: 'oc adm pv create' manually creates a PersistentVolume object; it plays no part in dynamic provisioning under WaitForFirstConsumer. The mode defers the CSI provisioner's volume creation until a pod is scheduled. Manual PV creation suits pre-provisioned static volumes, not this dynamic StorageClass scenario.

When the CSI driver operator starts up during cluster bootstrap.

Why it fails: The CSI driver operator merely installs and manages the driver; it does not provision volumes. With WaitForFirstConsumer, provisioning is deferred until a pod using the PVC is scheduled, so the volume is created at that point, not at bootstrap. Operator startup is unrelated to per-claim binding.

The cluster image registry configuration

Why it fails: The cluster image registry configuration governs image pull and push behaviour, not PersistentVolumeClaim binding or attachment, so describing it exposes registry storage settings rather than the failing volume's events. It is tempting because registry storage itself uses PVCs, and would be correct when diagnosing registry pod storage or image pull failures.

The OAuth client credentials secret

Why it fails: OAuth client credentials secrets authenticate API service accounts, not storage volumes, so 'oc describe' on them reveals token metadata rather than PersistentVolumeClaim binding events or attachment errors. It is tempting because secrets are inspectable resources, and would be the right target when debugging authentication failures for an operator or OAuth-protected route.

StartupProbe

Why it fails: A StartupProbe only determines whether a container's application has started, gating liveness and readiness checks; it does not itself signal readiness to receive traffic. It is tempting because both run before traffic arrives, but readiness is reported by a ReadinessProbe, which controls Service endpoint membership.

LivenessProbe

Why it fails: A LivenessProbe detects whether a running container is healthy and restarts it on failure; it does not gate traffic, so a failing pod stays in Service endpoints. It is tempting because both are periodic HTTP or TCP checks, but readiness, not liveness, governs whether the pod receives requests.

HealthProbe

Why it fails: HealthProbe is not an OpenShift/Kubernetes probe type; the readiness check is a readiness probe, which controls whether a pod receives Service traffic. HealthProbe describes no API field, so it cannot be defined. It is tempting because the name sounds like a health check, but it maps to no actual mechanism.

The source namespace must disable its LimitRange objects.

Why it fails: LimitRange objects cap per-pod or per-container resource requests and limits; they have no bearing on cross-namespace API authorisation. It is tempting because LimitRange is a namespace-scoped policy object, and would matter if the question concerned resource quota enforcement.

An administrative override annotation must be placed on the target namespace.

Why it fails: Kubernetes RBAC has no override annotation mechanism; authorisation is evaluated solely through Roles, ClusterRoles and their bindings. It is tempting because annotations do configure other namespace behaviours, such as scheduling or sidecar injection, but no annotation bypasses the RBAC authorisation chain for a ServiceAccount.

The ServiceAccount must be granted cluster-admin rights cluster-wide.

Why it fails: Cluster-admin grants full cluster-wide access, far exceeding the namespace-scoped RoleBinding or ClusterRoleBinding plus Role required here. It is tempting because cluster-admin genuinely solves cross-namespace access, but by granting every permission everywhere rather than the specific API resources in namespace-b.

Hardcode credentials in the Dockerfile

Why it fails: Hardcoding credentials in the Dockerfile exposes them in image layers and source control, and OpenShift cannot use them for registry authentication during the build. It is tempting because it appears to make credentials available, and would be correct only if security and auditability were irrelevant.

Use an environment variable in the deployment

Why it fails: Deployment environment variables apply to the running pod, not the build process, so the build still cannot authenticate to the private registry. It is tempting because environment variables commonly pass configuration, and would be correct for supplying runtime settings to a deployed application rather than build-time registry credentials.

Mount a ConfigMap

Why it fails: A ConfigMap holds non-confidential configuration data, and OpenShift builds cannot read registry credentials from one, so authentication still fails. It is tempting because ConfigMaps do inject build settings, and would be correct for supplying ordinary configuration values rather than secret credentials.

The default Kubernetes events in the default namespace.

Why it fails: Kubernetes events are namespaced, short-lived and typically expire after an hour, so they rarely capture the operator's reconciliation errors for a CR. Events suit transient scheduling or image-pull failures; the CR's own status conditions persist the controller's reported error state.

The global clusterversion object status conditions.

Why it fails: ClusterVersion status conditions report the Cluster Version Operator's own reconciliation of OpenShift updates, not the operand CR's controller loop. It is tempting because ClusterVersion aggregates cluster operator health, but the specific CR's error messages live in that CR's status field.

The /var/log/messages file on the master nodes.

Why it fails: Node syslog captures kubelet and container runtime output, not the operator controller's reconciliation result for a CR. It is tempting when pods crash, but controller-loop errors are written to the CR's status conditions, not to /var/log/messages on masters.

mountOptions: [ "raw" ]

Why it fails: mountOptions applies filesystem mount flags passed to the kubelet when mounting a volume, so it presupposes a filesystem; it cannot request raw block access. volumeMode: Block is the correct field. It tempts because 'raw' sounds like a mount flag, yet raw block volumes are never mounted at all — they appear as a device node inside the pod.

accessMode: RawBlock

Why it fails: Kubernetes defines raw block access through the volumeMode field set to Block on the PersistentVolumeClaim, not through accessMode, which governs read/write sharing (ReadWriteOnce, ReadOnlyMany). accessMode: RawBlock is not a valid value and would be rejected by the API server. It tempts administrators because both fields sit alongside each other in the PVC spec.

storageType: Unformatted

Why it fails: The PVC schema has no storageType field; raw block volumes are declared with volumeMode: Block. storageType: Unformatted would fail validation and never reach the CSI driver. It tempts because 'unformatted' describes the desired outcome — a device without a filesystem — but Kubernetes expresses this through volumeMode, not a storage type selector.

bob needs an explicit RoleBinding for the 'admin' role in the default namespace

Why it fails: An 'admin' RoleBinding in the default namespace grants full control of resources inside that namespace only; it confers no cluster-scoped permission to create new projects. It tempts because namespace-scoped admin is the usual fix for managing workloads, which is the correct scenario when bob needs to administer one existing project.

bob's user object is missing the 'project-creator' annotation

Why it fails: User objects carry no 'project-creator' annotation; project creation is governed by the self-provisioner cluster role binding, which bob lacks. The tempting idea of per-user annotations reflects how labels and annotations tag resources, but authorisation in OpenShift flows exclusively through RBAC roles and bindings, never annotations.

bob must be added to the cluster-admin group in the oauth configuration

Why it fails: Cluster-admin grants unrestricted rights cluster-wide, far exceeding what project creation requires, and group membership is managed in the identity provider rather than the oauth configuration. It tempts because cluster-admin does enable 'oc new-project', making it the right answer only when a user genuinely needs full cluster control.

Review the OperatorHub catalog source logs in openshift-marketplace.

Why it fails: OperatorHub catalog source logs record marketplace operator installation activity, not Machine Config Daemon reconciliation errors on nodes. It is tempting because catalog sources do surface operator failures, and would be correct when a subscribed operator fails to install or upgrade in openshift-marketplace.

Run 'oc adm node-logs --role=worker' to view kernel panic messages.

Why it fails: Kernel panic messages are unrelated to MachineConfig application failures; the Machine Config Daemon logs its errors to the node journal under the machine-config-daemon unit. oc adm node-logs is the correct tool, but the wrong unit is queried here. Kernel logs would be the right choice when diagnosing a node that has crashed or rebooted unexpectedly.

Inspect the Cluster Version Operator logs in openshift-cluster-version.

Why it fails: The Cluster Version Operator reports cluster-wide update progress, not per-node MachineConfig application errors. The Machine Config Daemon writes its failures to the affected node's journal, retrievable via oc adm node-logs or oc describe machineconfigpool. CVO logs would be the right place to diagnose a stuck cluster upgrade, not a degraded pool.