Troubleshooting Scenario Questions

Into ~/.config/containers/policy.d/keys/

Why it fails: Podman reads signature verification keys from the GPG keyring configured in policy.json, not from a policy.d keys directory, which is not a recognised lookup path. A per-user policy.d location is tempting because policy.json itself lives under /etc/containers, suggesting sibling directories hold related configuration.

Into /etc/containers/certs.d/keys/

Why it fails: /etc/containers/certs.d/ holds registry TLS certificate authorities for transport trust, not GPG keys for image signature verification. It is tempting because both directories sit under /etc/containers and both concern trusting a registry, but certs.d governs TLS, not signedBy.

Into /var/lib/containers/sigstore/

Why it fails: /var/lib/containers/sigstore/ stores signature and identity data for sigstore-based verification, not the GPG public keys referenced by signedBy. It is tempting because sigstore is a genuine container-signing mechanism, and would be the correct location when policy.json uses sigstore rather than GPG signedBy.

podman output auth_api

Why it fails: podman output is not a valid subcommand, so it cannot retrieve the container's stdout stream. It is tempting because the name suggests exactly that function, and a log-retrieval command would be correct here; the actual subcommand for reading auth_api's standard output is podman logs.

podman events auth_api

Why it fails: podman events reports daemon-level lifecycle events such as container start, stop and die, not the application's stdout stream. It is tempting because it does surface container activity, and would be correct when auditing when auth_api started or exited rather than reading its runtime log output.

podman history auth_api

Why it fails: podman history lists the image layers and build commands behind a container's image, not its runtime stdout. It is tempting because it inspects container-related artefacts, and would be correct when tracing which Dockerfile instruction produced a layer, not for troubleshooting a failing running service.

podman run --healthcheck="curl -f http://localhost/" --interval=30 --timeout=5 --retries=3 webapp

Why it fails: --healthcheck and --interval are not podman run flags; podman uses --health-cmd, --health-interval, --health-timeout, --health-retries and --health-start-period. It is tempting because the values match the stem, and this style would be correct for Docker Compose healthcheck keys rather than podman CLI flags.

podman run --health-probe="curl -f http://localhost/" --health-delay=10s webapp

Why it fails: --health-probe is not a podman run flag, and the command omits the || exit 1 exit-status logic plus the timeout and retries settings the stem requires. It is tempting because it names a health check, and it would suit a runtime where a single probe command with defaults is acceptable.

podman run --check-command="curl -f http://localhost/" --check-period=30s webapp

Why it fails: --check-command and --check-period are not podman run options, so the health check never registers with the required interval, timeout or retries. It is tempting because the flag names sound like health-check configuration, and such syntax would be correct in a different orchestrator's manifest format.

podman inspect --format='{{.Metadata.ExitCode}}' container_name

Why it fails: ExitCode sits under the top-level State object, not Metadata, so this path resolves to nothing and prints blank output. It is tempting because Metadata does exist in the inspect output and holds container creation details, but the exit code is a runtime state attribute reached via {{.State.ExitCode}}.

podman inspect --format='{{.Status.Running}}' container_name

Why it fails: Running is a boolean under State, not Status, so this expression yields no value and cannot report whether the container is active. It is tempting because Status appears in inspect output for other container runtimes, but Podman nests runtime state under State; the working form is {{.State.Running}}.

podman inspect --format='{{.Exit}}' container_name

Why it fails: The inspect JSON exposes the exit code under State.ExitCode, so .Exit matches no field and returns an empty result. It is tempting because the name reads naturally and the surrounding syntax is valid Go templating, but the correct expression for this attribute is {{.State.ExitCode}}.

nscd

Why it fails: nscd caches NSS lookups for the host, not container-to-container DNS on a Podman network, so it cannot resolve container names. It is tempting because nscd relates to name service caching, but Podman's custom networks rely on aardvark-dns, not the host cache daemon.

dnsmasq

Why it fails: dnsmasq is not the default backend in modern rootless Podman; aardvark-dns handles container name resolution on custom networks. It is tempting because dnsmasq is widely used for DNS forwarding, but Podman replaced it with aardvark-dns for per-network container DNS.

bind9

Why it fails: Bind9 is a standalone DNS server, not the backend Podman spawns for container name resolution; rootless Podman runs its own resolver process per network. It is tempting because bind9 is a common DNS daemon, but Podman does not invoke it for aardvark-style container DNS.

The container executed an invalid system call blocked by seccomp.

Why it fails: A seccomp-blocked syscall usually returns EPERM to the process, causing an application-level error rather than SIGKILL; 137 specifically denotes termination by signal 9. Seccomp filtering is relevant when hardening containers against specific syscalls, not for diagnosing memory-driven kills.

The container encountered an application-level syntax error.

Why it fails: Application syntax errors surface as non-zero exit codes from the interpreter itself, commonly 1 or 2, not 137. Code 137 reflects SIGKILL (128+9), so it indicates external termination such as memory exhaustion, not a fault within the application's own parsing.

The container startup script had a permission denied error.

Why it fails: Exit code 137 means the process was terminated by SIGKILL, typically the OOM killer or a docker stop timeout, not a permission failure. Permission denied produces exit code 126 or 127, so this option confuses an authorisation error with a signal-based termination.

podman stream

Why it fails: podman stream does not exist as a Podman subcommand, so it cannot attach to or read container output streams. The name suggests live output, but attaching to a running container's streams is done with podman attach, and log retrieval with podman logs.

podman view-logs

Why it fails: podman view-logs is not a valid Podman subcommand, so it cannot retrieve container output. The plausible-sounding name mimics logging tools, yet the real command is podman logs, which reads the container's captured stdout and stderr.

podman output

Why it fails: podman output is not a Podman subcommand; no such command exists in the CLI. It is tempting because the name suggests retrieving container output, but the actual commands are podman logs for captured stdout/stderr and podman attach for live streams.

podman inspect-logs

Why it fails: `podman inspect-logs` is not a real Podman subcommand, so it cannot retrieve container output or events. It is tempting because `podman inspect` genuinely returns container metadata, and `podman logs` streams stdout/stderr, so the name looks like a plausible blend. `podman logs` and `podman events` are the actual commands required here.

podman history

Why it fails: podman history displays image layer build history, not a running container's output or events. It is tempting because it inspects container-related artefacts, and would be correct when auditing how an image was constructed from its Dockerfile instructions.

podman trace

Why it fails: podman trace records and displays system-call activity for a running process, not container stdout logs or lifecycle events. It is tempting because tracing sounds diagnostic, but examining output logs and event history requires podman logs and podman events, which read those specific sources.

--network isolated

Why it fails: Docker has no 'isolated' network driver; the value is rejected or falls back, so the container still gains connectivity. A driver of this name would be correct if Docker shipped such a mode, but none exists, so the container must join the none network instead.

--network bridge

Why it fails: The bridge driver attaches the container to Docker's default bridge network, giving it outbound connectivity and an IP address, so isolation fails. Bridge networking is correct when containers need to communicate with each other or reach external networks on a single host.

--network host

Why it fails: Host networking places the container directly in the host's network namespace, giving it full access to every host interface and port — the opposite of isolation. It is tempting because it removes NAT overhead and suits performance-sensitive workloads needing host-level throughput, but it cannot satisfy a no-network requirement.

podman exec-top

Why it fails: podman exec-top is invalid.

podman adjust

Why it fails: podman adjust does not exist.

podman modify

Why it fails: podman modify is not a valid subcommand.

podman logs payment_svc

Why it fails: podman logs displays the standard output and standard error captured from the container's main process. Since payment_svc fails to start and never runs, there is little or no log output, and logs do not expose mount points, namespace configuration, or the configured command. It cannot provide the low-level inspection data required.

podman top payment_svc

Why it fails: podman top lists the running processes inside a container along with selected fields. It requires the container to be running, and payment_svc is not running because it fails to start. Even if it were running, top shows process information rather than configuration details like mounts or namespaces.

podman stats payment_svc

Why it fails: podman stats reports live resource usage such as CPU, memory, and network I/O for running containers. It does not reveal static configuration such as mount points, namespace settings, or the configured command, and it cannot produce meaningful output for a container that is not running. It is unrelated to inspecting configuration.

Each service publishes its port to the host with 'ports:', which replaces the internal DNS records.

Why it fails: Publishing a port creates a host-side forward and does not remove or replace the DNS records for the container on its network. Containers on the same network still resolve each other by name even when ports are published. This condition would not cause name resolution to fail, so it does not explain the symptom.

The Compose file sets 'dns:' on each service to an external resolver that has no record of the service names.

Why it fails: Overriding 'dns:' changes the resolvers used by the container, but aardvark-dns is still consulted for names on the container's attached network in typical Podman configurations. More importantly, the described symptom is a complete failure of service-name resolution on the network, which points to network attachment or the DNS service itself rather than an external resolver setting.

The containers are running with '--network none' but have static /etc/hosts entries for each other.

Why it fails: With no network attached, containers have no connectivity at all, so they could not reach each other by IP either. The scenario states IP connectivity works, which rules out this condition. Static hosts entries would also be a manual workaround rather than the cause of a DNS failure on a user-defined network.

The binary was copied with incorrect permissions; it needs to be made executable with chmod +x during build.

Why it fails: A statically compiled Go binary needs no shared libraries, so missing permissions would produce a permission denied error, not file not found. The actual cause is a dynamic linker mismatch: the binary was built with CGO enabled or against glibc. chmod +x is the fix for permission errors, not execution failures.

The EXPOSE instruction was missing, preventing the binary from binding to network interfaces.

Why it fails: EXPOSE only documents which ports the image intends to listen on; it neither opens ports nor affects process execution, so its absence cannot produce 'file not found'. It is tempting because EXPOSE relates to container networking, but the error indicates the kernel cannot locate the executable or its interpreter.

The WORKDIR instruction was omitted, preventing Podman from locating the binary.

Why it fails: WORKDIR sets the working directory for subsequent instructions and container start; it does not affect how the runtime resolves the binary path given in ENTRYPOINT or CMD. It is tempting because a missing directory can break relative paths, but 'file not found' here signals a missing dynamic loader.

podman ps --format='{{.Command}}' app_run

Why it fails: Listing running containers shows only the command of an already-started container, and ps cannot report entrypoint plus arguments for a stopped container. It tempts because ps --format is the quick way to view a live container's command line, which would suffice if app_run were already running.

podman inspect --format='{{.State.Command}}' app_run

Why it fails: State.Command does not exist in Podman's inspect schema, so the template renders empty. It tempts because State holds runtime data such as Pid and Status, making it look like the natural place for command details, whereas Config carries Entrypoint and Cmd.

podman inspect --format='{{.Args}}' app_run

Why it fails: The .Args field holds only the command arguments, omitting the entrypoint, so it cannot reveal the full startup command. Inspecting both requires .Config.Entrypoint and .Config.Cmd. .Args is tempting because it sounds like the argument list, but it is not the container configuration field.

podman logs web

Why it fails: Container logs show output from the application running inside the container, such as nginx access or error messages. A host-side port conflict prevents the container from being attached to the host port at all, so the failure occurs before the containerized process produces any relevant log lines. Checking logs would not reveal the host port already in use.

podman port web

Why it fails: 'podman port' lists the port mappings that Podman successfully configured for a running container. If the mapping was rejected because the host port was already bound, this command would simply show no mapping or an incomplete one, but it does not identify which host process already holds port 8080, so it does not explain the root cause.

podman inspect web

Why it fails: 'podman inspect' returns the container's configuration, including the requested port bindings and network settings, but it reflects what was requested rather than what is currently bound on the host. It will not show the competing host process that owns port 8080, so it does not directly explain why the mapping could not be established.