Troubleshooting Scenario Questions

Store the credentials in the Automation Hub token and reference it.

Why it fails: Automation Hub tokens authenticate collection and image registry access, not pip against a private Python index, so pip still receives no credentials. It tempts because it is the standard secret mechanism in this toolchain, and would be correct when the failure involves pulling collections or base images rather than installing packages.

Add the credentials directly to the `Containerfile` that `ansible-builder` generates.

Why it fails: The Containerfile is regenerated by ansible-builder on each build, so hand-edited credentials are overwritten and also persist in image layers. It tempts because editing the generated file directly appears to fix the failing pip command, and would be acceptable only for a throwaway local build with no secret involved.

Include the credentials in the `execution-environment.yml` under `dependencies: python:`.

Why it fails: Credentials written into execution-environment.yml are committed to source control and baked into build context, exposing secrets to anyone with repository access. It tempts because that file already lists Python dependencies, so adding an index URL there feels natural, and it would suit a public repository needing no authentication.

The private automation hub is configured to allow unauthenticated access; change the hub configuration to disable authentication.

Why it fails: A 403 means the registry rejected the request as unauthorised, so disabling authentication would remove the credential check rather than satisfy it; the controller needs valid registry credentials configured. Anonymous pull is tempting for a fully public registry, but a private automation hub requires authentication for image pulls.

SELinux on the controller node is blocking container pulls; temporarily set SELinux to permissive.

Why it fails: SELinux denials produce permission errors or AVC messages, not an HTTP 403 returned by the registry itself; the registry answered, so the block is authorisation. Permissive mode is tempting when diagnosing container runtime denials, but here the controller must authenticate to the private automation hub.

The execution environment definition in the controller is missing the 'pull' field; add 'pull: always' to the job template.

Why it fails: A 403 from the registry indicates authentication or authorisation failure, so the controller needs valid registry credentials configured for the execution environment; the pull field only controls image-fetch behaviour. It is tempting because pull policy does govern image retrieval, but it cannot resolve a permissions rejection.

Read role

Why it fails: The Read role grants view-only access to a job template, so operators could inspect job output but could not launch jobs, failing the execution requirement. Read suits auditors or stakeholders who need visibility without operational control; here the team must both launch templates and view output.

Admin role

Why it fails: The Admin role grants full control of the job template, including editing its configuration, which directly violates the requirement to prevent modifications. Admin suits template owners who must manage credentials, surveys and settings, not operators restricted to launching and reading output.

Update role

Why it fails: The Update role permits editing the job template's configuration, contradicting the requirement that operators must not modify it. Update suits engineers who maintain template definitions; operators needing only launch and output visibility require the Execute role instead.

Verify the inventory contains the correct host IP

Why it fails: An incorrect host IP yields connection timeouts or unreachable-host errors, not 'Permission denied', which indicates the SSH credential was rejected after reaching the host. Verifying inventory addresses is the right first step when jobs fail with unreachable or connection-refused errors instead.

Check the vault credential used in the job template

Why it fails: Vault credentials decrypt secrets inside playbooks at runtime; they do not authenticate the SSH connection, so a vault problem surfaces as decryption errors, not 'Permission denied'. Checking vault credentials is correct when playbooks fail while reading encrypted variables or files.

Check the project sync status

Why it fails: Project sync status governs whether playbook content is current, not SSH authentication; a stale sync produces outdated or missing tasks, not 'Permission denied'. It is tempting because sync failures commonly break job runs, and checking it would be right when templates fail with missing roles or modules rather than credential rejection.

Change the script's file extension to .json so Ansible recognizes it as a JSON inventory.

Why it fails: Ansible does not rely on file extensions to determine inventory type. It checks if the file is executable and attempts to run it with --list. Renaming to .json would make Ansible treat it as a static JSON file, which would fail unless the content is valid static JSON. This does not address the dynamic script requirements and likely worsens the problem.

Make the script non-executable and pass it as a static inventory file.

Why it fails: Dynamic inventory scripts must be executable; Ansible runs them to get inventory data. Making it non-executable would cause Ansible to try to read it as a static file, which would fail because it is a script, not an INI or YAML inventory. This action directly contradicts the requirement for dynamic scripts and would not resolve the parse error.

Add the script path to the `inventory` setting in ansible.cfg and remove the -i option from the command.

Why it fails: Specifying the inventory in ansible.cfg is a valid configuration method, but removing -i does not fix the underlying parse error. The error indicates the script itself is not producing valid output or not handling --list. Changing how it is specified does not correct the script's behavior. The problem would persist as long as the script is malformed.

Run `podman login registry.redhat.io` to authenticate.

Why it fails: Authentication happens after the hostname resolves, so credentials cannot fix a failed lookup of registry.redhat.io. It is tempting because login errors are common with Red Hat registries, and authenticating is correct when the failure is an authorisation or token error rather than DNS.

Restart the container runtime service.

Why it fails: Restarting the runtime does not correct a misconfigured or unreachable resolver, so the lookup still fails. It is tempting after a daemon crash or hung socket, where a restart genuinely restores container operations, but DNS configuration persists across restarts.

Use the `--no-cache` flag to force a fresh build.

Why it fails: Bypassing the layer cache does not alter name resolution, so the DNS failure recurs. It is tempting when stale cached layers mask a changed Dockerfile, which is a genuine build problem, but here the resolver itself is failing before any layer is fetched.

`serial`

Why it fails: `serial` controls the number of hosts updated per batch, not the failure threshold. While it can limit the impact of a failure by reducing batch size, it does not automatically abort the play when a certain percentage of hosts fail. It only defines how many hosts are targeted at once, so it does not satisfy the requirement to pause based on failure rate.

`any_errors_fatal`

Why it fails: `any_errors_fatal` causes the entire play to fail immediately when any host encounters an error, regardless of the number of hosts. This is too strict for a rolling update where a single host failure might be acceptable within the allowed unavailability. It does not provide a percentage-based threshold and would abort on the first error.

`ignore_errors`

Why it fails: `ignore_errors` allows a task to continue even if it fails, which is the opposite of what you need. It would mask failures and not pause the play, potentially leaving hosts in an inconsistent state. It does not provide any mechanism to stop the rolling update based on failure percentage.

`max_fail_percentage: 0`

Why it fails: Setting max_fail_percentage to 0 would abort the play if any host fails, but it is evaluated per batch and would also abort on the first batch if any host fails, which may be too aggressive. More importantly, the scenario asks for immediate stop on a task failure in any batch, which is better controlled by a different keyword. This value can also cause unexpected aborts on unreachable hosts.

`ignore_errors: false` on the service restart task.

Why it fails: ignore_errors defaults to false, so explicitly setting it to false does not change behavior. It also only affects whether the specific task is treated as failed; it does not cause the entire play to stop when a failure occurs on one host. The rollout would continue for other hosts and batches.

`serial: 1`

Why it fails: Reducing the serial to 1 would update hosts one at a time, which slows the rollout but does not cause the play to stop when a task fails on one host. The remaining batches would still be processed, so the desired immediate halt is not achieved by changing the batch size.

The control node's firewall is blocking outbound DNS queries to the automation hub host.

Why it fails: The engineer already verified that the hostname is resolvable from the control node, which means outbound DNS queries are working. A firewall blocking DNS would prevent resolution entirely, contradicting the scenario. The error likely originates from the target host's inability to resolve its own hostname, not from the control node's DNS path.

The automation hub host is not listed in the `[automationhub]` group of the installer inventory.

Why it fails: The error explicitly indicates a DNS resolution failure, not a missing inventory group. If the host were absent from the `[automationhub]` group, the installer would report a missing required variable or group, not a DNS resolution error. The scenario states the hostname is correct and resolvable, so the issue lies elsewhere, such as missing `/etc/hosts` entries on the target or a required variable like `automationhub_admin_password`.

The `[automationhub]` group is missing the `automationhub_admin_password` variable, causing the installer to fail before DNS resolution.

Why it fails: A missing `automationhub_admin_password` would produce a specific error about an undefined variable, not a DNS resolution failure. The installer validates required variables early and would report a missing password before attempting network connections. The scenario describes a DNS error, so the cause is related to name resolution, not credential configuration.