Troubleshooting Scenario Questions

The kernel parameter nomodeset is set in /etc/default/grub.

Why it fails: nomodeset affects graphics driver loading at boot, having no bearing on interface addressing or DHCP fallback. It is tempting because kernel parameters can influence boot behaviour, and would be relevant when diagnosing display or GPU initialisation failures, but it cannot cause ens192 to obtain a DHCP lease.

The firewall is blocking the static IP assignment.

Why it fails: Firewall rules filter network traffic; they do not assign addresses or override ifcfg settings, so blocking cannot explain DHCP fallback after reboot. Firewall misconfiguration would be the cause when connectivity fails despite a correct static address, not when the interface itself requests DHCP.

The network service is not enabled to start at boot.

Why it fails: NetworkManager already started the interface at boot, so enabling the service would not change the DHCP fallback. The ifcfg file lacks ONBOOT=yes, which NetworkManager reads to decide whether to activate the profile automatically; without it, the interface defaults to DHCP. Enabling a service is the fix when a unit is genuinely disabled, not here.

/etc/sysconfig/network

Why it fails: /etc/sysconfig/network holds hostname and gateway settings, not resolver configuration, so it cannot cause hostname-only resolution failure. It is tempting because it is a network configuration file, and would be correct if the default gateway or static hostname were misconfigured.

/etc/hosts

Why it fails: A stale or incorrect entry in /etc/hosts would override DNS and break resolution for that specific hostname, but the stem gives no evidence of a local override. It is tempting because hosts entries are a common cause, and would be correct if only one hostname failed while others resolved.

/etc/nsswitch.conf

Why it fails: nsswitch.conf orders name-service lookups; a misordering there would still fall through to DNS, so it rarely causes total hostname failure. It is tempting because it governs resolution sources, and would be the answer if local files were consulted before DNS and returned stale entries.

The port forwarding rule should be added to the internal zone instead of the external zone.

Why it fails: Port forwarding for incoming external traffic should be in the external zone.

The internal server has a different default gateway.

Why it fails: The internal server's default gateway should be 192.168.1.1, which is the gateway server. If it were different, it would not route back correctly.

IP forwarding is not enabled on the system.

Why it fails: The stem states IP forwarding is enabled.

SSH is not allowed from any source

Why it fails: SSH is allowed from the local subnet.

The INPUT chain policy is ACCEPT, so SSH should be allowed

Why it fails: The explicit DROP rule overrides the default policy.

The DROP rule for SSH is not matching because of packet count zero

Why it fails: The packet count is zero because no packets have matched yet; it will match if traffic arrives.

The unit file has incorrect permissions

Why it fails: Exec format error means the kernel cannot parse the executable, typically a missing shebang, wrong architecture binary or CRLF line endings — file permissions produce 'Permission denied' instead. Permissions are tempting because unit files must be readable, but that failure mode is a different systemd error entirely.

The user does not have permission to start the service

Why it fails: Insufficient user privilege returns 'Access denied' or polkit authorisation failures, not Exec format error, which originates from the kernel's binary loader. Permission problems are tempting because they commonly block service starts, but they are the correct diagnosis when systemctl reports authorisation or interactive authentication required.

The service is disabled

Why it fails: A disabled unit simply is not started at boot; manual starts still work and never yield Exec format error. Disabling is tempting because it is a common reason services do not run, but it is the right diagnosis when a unit is unexpectedly absent after reboot, not when the binary itself is unparseable.

The CPU is overloaded, causing processes to wait for CPU time.

Why it fails: High await with low svctm points to I/O queueing, not CPU contention; CPU saturation would show high load or run-queue delays, not disk wait. It is tempting because slow peak-hour performance often stems from CPU, and would be correct if vmstat showed runnable processes waiting.

The system is using swap space heavily, causing disk I/O.

Why it fails: Heavy swap usage would cause high disk utilization, but the svctm would likely be higher because swapping involves random I/O, and the await would be high as well. However, the low svctm suggests the disk is serving requests quickly once they start, which is not typical of swap.

The disk is experiencing hardware errors.

Why it fails: Hardware errors would raise svctm, not leave it at 5 ms while await exceeds 1000 ms; the gap indicates queueing. It is tempting because disk faults cause slowness, and would be correct if iostat showed elevated service times or errors.

systemctl status myapp.service

Why it fails: Status shows current unit state, the main process exit code and only the tail of recent journal entries, not the full error output produced before exit. It is tempting because it is the quickest service check, and it would suit a first glance at whether the unit is active or failed.

systemctl reload myapp.service

Why it fails: Reload re-reads configuration for a running unit; on a failed service it cannot surface the earlier exit output and may itself fail. It is tempting because reload avoids downtime, and it would be correct for applying configuration changes to an already active daemon.

systemctl restart myapp.service

Why it fails: Restarting the unit re-runs the service and may clear the failed state, but it generates new output rather than retrieving the previous run's error text. It is tempting because restarting is a common first response to a fault, and it would suit recovering a service after the cause is known.

The administrator forgot to run `systemctl enable` before starting the service.

Why it fails: enable only creates symlinks for boot-time start; it does not affect unit discovery, so 'Unit not found' persists. It is tempting because forgetting enable is a frequent systemd mistake, and would be correct if the unit were found but failed to start automatically at boot.

The unit file is placed in /usr/lib/systemd/system/ instead of /etc/systemd/system/.

Why it fails: /usr/lib/systemd/system/ is a valid unit search path, so placement there does not cause 'Unit not found'. It is tempting because /etc/systemd/system/ is the conventional location for custom units, and would be correct if the file were in a non-searched directory.

The administrator is not in the 'systemd' group.

Why it fails: systemd has no 'systemd' group governing unit visibility; permissions are not group-based in this way. It is tempting because permission errors are common causes of service failures, and group membership would be correct if the unit file itself were unreadable by the invoking user.

Replace the power supply unit to ensure stable power.

Why it fails: The dmesg soft lockup traces identify a kernel thread monopolising CPU#0, so swapping the power supply addresses no fault the logs describe. It is tempting because hardware instability can freeze a server, but PSU replacement would be correct only where logs showed voltage, thermal or memory errors rather than a CPU stuck in kernel code.

Increase the soft lockup threshold via sysctl to reduce false positives.

Why it fails: Raising the soft lockup threshold through sysctl only delays when the watchdog reports a stuck CPU; the kernel thread still spins and the server still hangs. It is tempting to silence noisy warnings, but threshold tuning would be correct only when genuine stalls are shorter than the configured detection interval.

Add 'nosoftlockup' to the kernel boot parameters.

Why it fails: Adding nosoftlockup merely suppresses the watchdog's detection and reporting of CPU stalls, leaving the underlying kernel-thread hang unaddressed and recurrence silent. It is tempting as a way to stop alarming dmesg output, but it would be appropriate only when soft lockup warnings are confirmed false positives from virtualisation or heavy scheduling latency.

The default gateway is missing from the routing table.

Why it fails: The stem already shows 'default via 10.0.0.1 dev eth0' in the routing table, so the gateway is present. A missing default route would be the cause if 'ip route show' returned no default entry, leaving only the connected 10.0.0.0/24 route.

DNS resolution is not configured.

Why it fails: Ping to 8.8.8.8 is a raw ICMP test using an IP literal, so name resolution never occurs; DNS cannot explain this failure. Configuring resolvers in /etc/resolv.conf is the right fix when names such as google.com fail while numeric addresses succeed.

A firewall is blocking outbound ICMP traffic.

Why it fails: A default route via 10.0.0.1 exists, so routing is configured; ICMP being filtered would not prevent TCP-based internet access, and the stem gives no evidence of a firewall rule. Firewall diagnosis is correct when only specific protocols fail while others succeed.

Edit /opt/tomcat/bin/startup.sh and hardcode JAVA_HOME.

Why it fails: Hardcoding JAVA_HOME inside startup.sh bypasses systemd's environment handling and diverges from the packaged script, so the unit still starts with no JAVA_HOME exported. Editing the script is tempting because the error names that variable, but the correct fix sets Environment=JAVA_HOME= in the unit file.

Run the startup script manually with 'bash /opt/tomcat/bin/startup.sh'.

Why it fails: Running the script manually executes it in the administrator's interactive shell, which already has JAVA_HOME, so it neither fixes the unit's missing environment nor makes systemctl start succeed. Manual invocation is tempting for confirming the script itself works, but systemd services need Environment=JAVA_HOME= in the unit file.

Set JAVA_HOME globally using 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk' in /etc/profile.

Why it fails: /etc/profile is read only by login shells, so systemd's ExecStart never sources it and JAVA_HOME remains unset for the unit. Global exports are tempting because they fix interactive sessions, but the correct fix declares Environment=JAVA_HOME= in tomcat.service, which systemd reads directly.

systemctl status dhcpd

Why it fails: dhcpd is the server daemon, so systemctl status dhcpd reports on a DHCP server, not the client on eth0. It is tempting because systemctl status is the standard way to inspect a failing service, and would be correct on a host running isc-dhcp-server.

nmcli dev show eth0

Why it fails: nmcli dev show eth0 only displays current interface configuration and state; it does not expose the DHCP client's lease negotiation or errors. It suits confirming an address is absent, whereas inspecting dhclient or NetworkManager logs reveals why the request failed.

dhcpd -t

Why it fails: dhcpd -t validates the server's configuration file syntax; it does not query or trace a client's lease acquisition. It is tempting as a diagnostic flag, and would be correct when verifying dhcpd.conf before restarting a DHCP server.

The vm.dirty_ratio and vm.dirty_background_ratio are set too high.

Why it fails: Dirty ratios govern how much modified page-cache data may accumulate before writeback; they influence flush timing, not whether anonymous pages get reclaimed to swap. It is tempting because both are vm tunables, but the swap usage with free memory indicates an aggressive swappiness setting instead.

The database is configured to use huge pages, which are not swappable.

Why it fails: Huge pages are locked and non-swappable, so their presence would reduce swap consumption, not cause 6 GB of swap use. It is tempting because huge pages affect memory behaviour, but the observed swapping with free RAM points to swappiness or reclaim pressure, not page-size configuration.

The vm.vfs_cache_pressure is set too low.

Why it fails: vfs_cache_pressure controls reclaim of dentry and inode caches; lowering it retains those caches longer and does not push anonymous pages into swap. It is tempting because it is a reclaim-related tunable, but the symptom reflects swappiness driving anonymous-page eviction despite available RAM.

free -m

Why it fails: free -m reports total, used and available RAM and swap, but not the per-process or cgroup limits that trigger 'cannot allocate memory' after a kernel update. It is tempting because it is the standard first look at memory pressure; systemd unit limits or cgroup memory.max values are the actual constraint here.

cat /proc/meminfo

Why it fails: /proc/meminfo shows system-wide memory counters, not the per-service cgroup or systemd limit that produces 'cannot allocate memory'. It is tempting as a general memory overview, yet the service's own MemoryMax or cgroup memory.max is the binding constraint. Inspect the unit's limits directly rather than aggregate host figures.

sysctl vm.overcommit_memory

Why it fails: vm.overcommit_memory governs kernel allocation heuristics, not the cgroup or systemd MemoryMax ceiling that a service hits. It is tempting because overcommit settings can cause allocation failures, but reading it first ignores the per-unit limits introduced or changed by the kernel update. Check systemctl show or cgroup memory.max instead.

The server's DNS resolver is not configured

Why it fails: Ping uses IP address, not DNS.

The SSH rule is misconfigured and accidentally blocks all traffic

Why it fails: SSH works, so the SSH rule is fine.

The default policy on the INPUT chain is DROP

Why it fails: If the policy were DROP without any rules, SSH would not work either.