Hard Difficulty Questions

Install a plugin that logs all kubectl commands.

Why it fails: Logging kubectl commands provides an audit trail after the fact but leaves manual writes fully permitted, so it cannot ensure resources are unmodifiable. Command logging suits forensic or compliance visibility requirements, not enforcement of a Git-only change path.

Add a firewall rule to block kubectl.

Why it fails: Blocking kubectl at the network layer does not prevent modification: any client, CI job or direct API call from an allowed source still writes to the API server. Firewall rules suit restricting access to specific CIDR ranges, not enforcing that all changes flow through Git.

Disable the Kubernetes API server.

Why it fails: Disabling the API server halts all cluster operations, including the GitOps controller's own reconciliation, so nothing can be deployed at all. Taking the API server offline is appropriate only for decommissioning or isolating a cluster, never for governing change.

Only having one cluster.

Why it fails: Consolidating to a single cluster removes the regional variation the scenario requires, so identical configuration is achieved by eliminating the regions rather than managing them. It is tempting because one cluster trivially guarantees uniformity, and that would be correct where no regional presence, latency or data-residency obligations exist.

Duplicating the configuration for every cluster.

Why it fails: Duplicating configuration per cluster breaks the single source of truth, so a shared change must be edited in every copy and drift accumulates. Duplication suits small, static estates where clusters never share configuration and change rarely.

Using global variables in a shell script to modify manifests.

Why it fails: Shell scripts mutating manifests bypass Git as the source of truth, so drift and audit history are lost, and regional overrides cannot be reconciled declaratively. Scripting suits one-off imperative patching, such as bootstrapping a legacy cluster, but not continuous GitOps reconciliation across a fleet.

Provision new clusters

Why it fails: Kyverno policies validate, mutate, generate and verify Kubernetes resources; provisioning clusters is the job of cluster lifecycle tooling such as eksctl or Terraform. Kyverno is tempting because it is cluster-scoped and declarative, but it operates on API resources, not infrastructure provisioning.

Manage Git repository webhooks

Why it fails: Git webhooks are configured in the Git provider or CI system, not by Kyverno, which validates, mutates, generates and verifies Kubernetes resources at admission time. Kyverno is tempting because it is policy-as-code stored in Git, but it never manages repository-side integrations.

Automatically upgrade cluster nodes

Why it fails: Node upgrades are handled by managed node group versioning or Karpenter, not admission policies; Kyverno mutates, validates, generates and verifies Kubernetes resources. Kyverno is tempting because it enforces cluster-wide desired state, but it cannot orchestrate node lifecycle operations.

Copying the entire deployment configuration for every single environment.

Why it fails: Duplicating whole deployment configurations per environment creates divergent copies that drift, defeating the single-source-of-truth model GitOps depends on. It is tempting because isolated per-environment manifests feel safe, and that approach is correct when environments genuinely differ in topology and must be versioned as separate, deliberate overlays.

Storing all configurations in a single root-level file.

Why it fails: One root-level file serialises every change through a single artefact, so unrelated environments become coupled and merge conflicts block promotion. It is tempting because a single file appears tidy and atomic, and that pattern is correct for a small, single-environment repository where no per-team ownership boundaries exist.

To perform parallel builds

Why it fails: App-of-Apps structures parent and child Argo CD Applications for declarative deployment; parallel builds are a CI pipeline concern outside Argo CD's GitOps reconciliation. It is tempting because both involve many applications, but the pattern governs deployment topology, not build execution.

To increase the memory capacity of the cluster

Why it fails: App-of-Apps orchestrates nested Argo CD Applications; cluster memory is provisioned by the underlying nodes, not by an Application manifest pattern. It is tempting because large application sets consume resources, but the pattern's purpose is declarative multi-application deployment and synchronisation.

To allow multiple users to edit the same file

Why it fails: App-of-Apps manages a hierarchy of Argo CD Applications from a parent Application; concurrent file editing is handled by Git branching and merge controls. It is tempting because both concern managing many manifests, but the pattern addresses declarative application orchestration, not user access.

During the Git commit process using local pre-commit hooks.

Why it fails: Pre-commit hooks run on the developer's machine and can be skipped with --no-verify, so they cannot enforce policy for every commit reaching the cluster. They are useful for fast local linting feedback, but admission control must sit where manifests are actually applied.

In the CI pipeline during the image scanning phase.

Why it fails: Scanning images in CI validates container contents, not Kubernetes manifest fields such as privileged mode or hostPath, which OPA/Gatekeeper evaluates at admission. CI scanning is the right place for vulnerability and provenance checks, but it cannot block non-compliant resource specs at apply time.

By manually reviewing the Git diffs before merging the pull request.

Why it fails: Manual diff review is a human gate, not automated policy enforcement, so it cannot guarantee every manifest is evaluated against OPA/Gatekeeper constraints before admission. It is tempting because peer review catches intent-level issues, but it belongs alongside, not instead of, admission-time automated checks.

Because it eliminates the need for Git.

Why it fails: Pull-based reconciliation depends on Git as the source of truth; the agent continuously compares cluster state against the Git repository. Eliminating Git contradicts the entire GitOps model. The option is tempting because pull-based agents do reduce direct human access to clusters, but Git remains the declarative store they reconcile against.

Because Git requires passwords for every pull.

Why it fails: Git does not require passwords for every pull; it commonly uses SSH keys, tokens or anonymous HTTPS reads. Pull-based security derives from the agent initiating outbound connections, so no inbound credentials are exposed. The option is tempting because authentication does feature in Git workflows, but it misstates how Git access actually works.

Because it removes the need for container images.

Why it fails: Pull-based reconciliation still deploys container images; it merely has the cluster fetch the desired state from Git rather than receiving a push. Removing images is unrelated to the direction of reconciliation. The option is tempting because pull models do shift trust to the cluster, but images remain central to deployment.

It manually restarts the cluster

Why it fails: An Image Updater watches a registry and commits updated image tags back to the Git repository, letting the GitOps operator reconcile the change; it never restarts clusters. Restarting nodes is what a rolling-update strategy or a manual kubectl rollout restart does, so this option confuses image-tag automation with workload restarts.

It creates a backup of the cluster

Why it fails: The Image Updater commits new image tags to git; it does not snapshot cluster state. Tempting because GitOps relies on git history, and a backup would be the correct choice when protecting cluster resources against loss, not when automating image tag updates.

It bypasses the git repository entirely

Why it fails: The Image Updater writes updated image tags back to the git repository, so the operator can reconcile the change; bypassing git removes the declarative source of truth. Tempting because direct cluster edits appear faster, and would be correct only for imperative tooling that does not reconcile from git.

To replace the GitOps operator

Why it fails: A CI pipeline builds and tests code and pushes images or manifests; it does not perform the operator's continuous state reconciliation against Git. Replacing the operator is tempting because pipelines also run on commits, yet the operator is what watches the cluster and enforces desired state.

To execute the reconciliation loop

Why it fails: The reconciliation loop belongs to the GitOps operator, which continuously compares cluster state against Git and applies drift correction; a CI pipeline builds, tests and publishes artefacts. It is tempting because CI and GitOps both react to commits, but the loop is the operator's controller, not the pipeline's job.

To handle user authentication for the cluster

Why it fails: Cluster authentication is handled by the API server's identity providers and RBAC, not by a CI pipeline, which compiles, tests and publishes artefacts. It is tempting because pipelines hold credentials to push to registries, but they authenticate to registries, not cluster users.

Git is no longer the source of truth

Why it fails: Git becomes the single source of truth in GitOps, with Flux continuously reconciling cluster state against it. Abandoning Git describes no GitOps model at all; the tempting confusion is that CI still builds artefacts, yet desired state is declared and versioned in Git.

The cluster no longer requires an API server

Why it fails: The API server remains essential; Flux reconciles by watching Git and applying changes through it. Removing it is tempting when contrasting pull with inbound push access, but the architectural shift is that nothing external initiates connections, not that the control plane endpoint disappears.

The operator is now a push-based agent

Why it fails: Flux runs inside the cluster as a controller that pulls manifests from Git, so it is not a push-based agent reaching outward. A push agent describes the Jenkins model being replaced, where the pipeline holds credentials and initiates connections to the cluster API server.

The operator ignores the change to prevent production downtime.

Why it fails: Automatic Sync means the operator continuously reconciles drift without human intervention, so the manual change is reverted rather than left in place. It is tempting because operators do avoid destructive actions, but ignoring drift would defeat the reconciliation loop; only a suspended or manual-sync configuration leaves the change untouched.

The operator updates the Git repository to match the new manual change.

Why it fails: A GitOps operator treats Git as the single source of truth and reconciles the cluster towards it, so the manual edit is reverted, not written back to Git. It is tempting because bidirectional sync tools exist, but GitOps deliberately keeps Git authoritative and never commits cluster drift into the repository.

The operator prompts the administrator for manual approval before reverting.

Why it fails: Automatic Sync reconciles immediately and silently; approval prompts belong to manual sync, where the operator reports OutOfSync and waits. It is tempting because approval gates exist in GitOps, but they are a property of the sync policy, not of automatic reconciliation, which reverts drift without asking.

The network latency to the Git server is too high.

Why it fails: High latency delays reconciliation but does not itself produce OutOfSync; the controller still compares desired and live state once it fetches. It is tempting because connectivity problems commonly cause GitOps faults, and latency would be the cause when syncs time out or lag rather than diverge.

The Git repository is empty.

Why it fails: An empty repository yields no manifests to compare, so the controller would report no resources rather than a persistent OutOfSync against a correctly running cluster. It is tempting because missing desired state sounds like drift, and an empty repo would be the cause when nothing deploys at all.

The controller is broken.

Why it fails: A broken controller would stop reconciling entirely, so it would not reliably report OutOfSync while the cluster runs correctly. It is tempting because a faulty component seems to explain odd status, and a broken controller would be the cause when no status updates appear at all.

Running 'helm upgrade' manually.

Why it fails: Manual helm upgrade runs are imperative and leave no reconciled desired state, so drift is never corrected automatically. It tempts because Helm itself is central to GitOps tooling, and manual upgrades would be acceptable for one-off releases outside a GitOps pipeline.

Only storing the chart source code.

Why it fails: Storing only chart source omits the rendered manifests that Argo CD or Flux actually reconciles against the cluster, so drift between values and deployed state goes undetected. It is tempting because versioning chart templates is genuine GitOps practise; that approach suits developing a reusable chart, not deploying environment-specific releases.

Delete the pods and let the operator recreate them.

Why it fails: Deleting pods only triggers the operator to recreate them from the same failing manifest, reproducing the fault. Pod deletion suits clearing crashed or stuck pods under a healthy revision; rollback requires reverting the Git commit so the operator reconciles to the prior working state.

Stop the GitOps operator to prevent it from correcting the state.

Why it fails: Stopping the operator halts reconciliation but leaves the broken revision deployed and the desired state unchanged in Git, so drift persists. Pausing an operator suits planned maintenance windows, not rollback; reverting the Git commit is what actually restores the previous working state.

Run 'kubectl rollout undo' on the deployment.

Why it fails: Running kubectl rollout undo mutates the cluster directly, so the GitOps operator reconciles the live state back to the failing manifest still declared in Git. Rollback belongs in Git by reverting the commit; imperative kubectl commands suit non-GitOps clusters without a reconciling controller.

Logging into the node via SSH

Why it fails: SSH access to a node changes nothing the GitOps controller observes, since reconciliation compares Git commits with live cluster state, not node sessions. It is tempting because SSH is a familiar operational entry point, and it would be correct for manually inspecting or repairing a node's runtime.

Restarting the Kubernetes API server

Why it fails: Restarting the Kubernetes API server disrupts the control plane but does not reconcile cluster state against Git; the GitOps controller watches Git and cluster events, not API server restarts. It is tempting because restarts do reinitialise components, and restarting a controller pod would legitimately trigger reconciliation.

SOPS version mismatch

Why it fails: Flux decrypts using the sops-age key held in its decryption secret, so a SOPS binary version difference does not prevent decryption; the encrypted data key and Age recipient remain unchanged. Version alignment matters when the local sops CLI must re-encrypt files with a newer format, not when Flux reads them.

Network latency

Why it fails: SOPS decryption happens locally inside the Flux controller using the Age private key; it makes no network call to a key service, so latency cannot cause failure. Network conditions matter for KMS or Vault-backed keys, where the controller must reach a remote endpoint to unwrap the data key.

Incorrect git branch

Why it fails: Flux reconciles whatever branch the GitRepository source references; pointing at a different branch simply changes which manifests are read, and any encrypted file on that branch still decrypts. Branch selection is the correct concern when promoting environment-specific overlays, not when decryption itself fails.

ArgoCD Notifications

Why it fails: ArgoCD Notifications sends alerts about sync and health events; it takes no admission decision and cannot block a pod image. It is tempting because it is a GitOps observability component, but it would be correct for notifying teams of deployment changes, not enforcing registry allowlists at admission.

Sealed Secrets

Why it fails: Sealed Secrets encrypts Kubernetes Secrets for safe Git storage; it performs no admission-time validation of image registries. It is tempting because it is a Flux-friendly security tool, but it would be correct for committing credentials to Git, not for restricting namespaces to an internal registry.

Flux Image Automation Controller

Why it fails: Image Automation Controller updates image tags in Git; it does not evaluate admission requests. Namespace-scoped registry restriction at admission time needs a validating admission policy, such as a Kyverno ClusterPolicy or Gatekeeper constraint. Flux's automation is tempting because it already watches registries, but it writes manifests rather than blocking pods.

Maintaining a manual 'deployment' document in a wiki.

Why it fails: GitOps stores desired state declaratively in Git, so a wiki document cannot drive reconciliation and drifts from the cluster. It is tempting as lightweight documentation, and would suit recording runbooks or onboarding notes, but not the authoritative deployment source.

Increasing the number of manual approvals in the CI pipeline.

Why it fails: GitOps removes manual approval gates by letting merge requests and automated reconciliation control deployment, so adding approvals contradicts the model. It is tempting where regulated releases demand human sign-off, but that scenario is change management, not GitOps transition.

A tool that converts YAML to JSON.

Why it fails: Converting YAML to JSON is a serialisation task, not continuous reconciliation of cluster state against a Git repository. It is tempting because GitOps manifests are YAML and tooling often parses them, so a conversion utility appears related; however, an operator watches Git and applies drift correction, which no format converter performs.

A database that stores configuration history.

Why it fails: A GitOps operator is a controller that watches Git and reconciles cluster state; it is not a datastore, though Git itself holds configuration history. It is tempting because Git does store history, but the operator acts on that history rather than storing it.

A human user who manually pushes Git commits.

Why it fails: A GitOps operator is software running in the cluster that continuously reconciles live state with the Git-declared desired state; a human committing manually is not an operator. It is tempting because humans do commit changes, but the operator automates that reconciliation loop.

Share secrets in cleartext via team chat

Why it fails: Team chat offers no access control, audit logging, or encryption guarantees, and messages persist on third-party servers, breaching secret-handling policy. It is tempting for speed when teams coordinate during incidents, but approved channels are encrypted secret managers with scoped, auditable retrieval, not chat.

Store base64 encoded secrets in Git

Why it fails: Base64 is reversible encoding, not encryption, so committing such secrets exposes credentials to anyone with repository read access and to Git history permanently. It is tempting because Kubernetes Secrets manifest as base64 and tools like Sealed Secrets encrypt them, but plain base64 in Git is never a secrets-management practise.