Troubleshooting Scenario Questions

fsck /dev/sda1

Why it fails: Running fsck against /dev/sda1 limits the check to a single partition and does not address the broader set of filesystems needed for the boot process. If the root filesystem is not on sda1, or if the boot error is actually caused by a missing or malformed entry in /etc/fstab, this command will not find the problem. It also does not honor the pass-order sequencing determined by the fifth column of /etc/fstab, which fsck -A enforces.

debugfs -R 'repair'

Why it fails: debugfs is an interactive ext2/3/4 debugger designed for manually examining and modifying inodes, block bitmaps, and superblocks, but it has no 'repair' command and is not a repair utility. The -R flag merely runs one debugfs command non-interactively, and 'repair' is not a recognized command, so the invocation will fail. Even if a valid command such as 'check' were used, debugfs lacks the automated scanning and recovery logic of e2fsck and is far too low-level and risky for ordinary boot recovery.

e2fsck -p

Why it fails: e2fsck -p invokes the ext-family checker in preen mode, which automatically fixes only the set of cleanly repairable problems and requires you to specify a device or filesystem explicitly. It does not scan /etc/fstab, so it cannot discover every filesystem that the boot sequence depends on; if another ext4 filesystem is also corrupted, it will remain unchecked. Moreover, preen mode is intentionally conservative and will stop and require a manual answer or -y for more severe corruption, making it less robust than fsck -A -y.

usermod -e 2024-12-31 bob; passwd -S bob

Why it fails: usermod -e 2024-12-31 bob assigns a fixed calendar date rather than a date relative to the current day, so it does not guarantee the required 30-day window; if today's date is past that date, the account is immediately expired. passwd -S bob merely reports the password status (for example, PS, LK, NP) and does not change any field in /etc/shadow, so it cannot unlock the account or reset its expiration. The password status may still show locked even after the first command.

chage -d 0 bob; usermod -L bob

Why it fails: chage -d 0 bob forces Bob to change his password at the next login by setting the last password change date to epoch, but this does not touch the account expiration or inactivity lock that caused the account to be disabled. usermod -L bob then explicitly locks the account by prepending '!' to the encrypted password hash in /etc/shadow, which is the opposite of reactivating it. The two commands together make the account more inaccessible, not less.

usermod -e $(date -d +30days +%Y-%m-%d) bob; passwd -u bob

Why it fails: While usermod -e $(date -d +30days +%Y-%m-%d) bob does correctly extend the account expiration to 30 days from now using shell command substitution, the second command fails to address the inactivity lock: passwd -u bob only removes the '!' or '!!' prefix from the password hash in /etc/shadow, essentially unlocking the password, but it does not reset the number of inactive days tracked by chage -I. As a result, the account still remains disabled under the inactivity policy, and Bob still cannot log in. This answer is incomplete because it only handles password locking, not expiration/inactivity.

The physical volume is corrupted and needs to be restored from backup

Why it fails: Physical volume corruption typically surfaces as I/O errors, failed metadata reads, or 'Read of PV failed' messages during vgscan, not as an 'unknown device' placeholder. The PV's identity and device path remain known; LVM would mark the PV as having errors or being unavailable while still displaying its name. Restoring from backup is relevant only after metadata loss or device failure, but it would not explain a filter-induced scanning exclusion.

The filesystem on the logical volume has become corrupted, preventing LVM metadata access

Why it fails: Filesystem corruption on a logical volume is unrelated to LVM metadata, because metadata lives in the physical volume's metadata area and in /etc/lvm/backup, not inside the filesystem on the LV. LVM activation is completely independent of filesystem health; a corrupted ext4/xfs filesystem does not prevent LVM from reading PV metadata or activating the VG. Therefore, it cannot be the cause of an 'unknown device' status.

The UUID of the physical volume has changed due to a disk replacement

Why it fails: When a disk is replaced, the new device's PV UUID differs from the original, so LVM treats it as a brand-new physical volume; the old PV is considered missing, appearing as 'unknown device' only if its device node is gone. If /dev/sdb1 is present with a changed UUID, LVM would display it as an unrecognized PV, not cloak it as 'unknown device.' A deliberate filter exclusion, not a UUID change, is what prevents the PV from being scanned and identified.

Add --force to the podman run command

Why it fails: The `--force` flag is not a valid option for `podman run`; it is used with `podman rm` or `podman stop` to forcefully terminate a running or stopped container without waiting for graceful shutdown. Even if you attempted to pass it, it would be rejected by Podman's CLI parser, and it would not address the underlying port conflict. To fix the error, you must either stop the process already bound to port 80 or map the container to a different host port, such as `-p 8080:80`.

Add --replace to the podman run command

Why it fails: The `--replace` option tells Podman to remove any existing container with the same name before creating the new one, so it only resolves name collisions, not port conflicts. If port 80 is already occupied, the new container will still fail to bind regardless of whether `--replace` is present, because the conflict is with an external process or another container's port forward. It is therefore the wrong tool for this scenario.

Use --net=host to bypass the port mapping

Why it fails: With `--net=host`, the container shares the host's network namespace and binds directly to host interfaces, bypassing the need for `-p` port mappings entirely. However, this does not grant any special ability to re-bind an already-occupied port: if a host process is listening on port 80, the container's service will encounter the same 'address already in use' error. The only way to avoid the conflict is to free the port or change the port that the container's service uses.

The file is owned by the wrong user.

Why it fails: If /etc/sudoers were owned by a non-root user, sudo would refuse to read it and issue a warning such as 'sudo: /etc/sudoers is owned by uid X, should be 0' rather than a generic 'unable to open' message. In this setup, the file is correctly owned by root:root, so ownership is not the cause. Sudo specifically checks the owner UID and aborts before attempting to open the file if ownership is invalid.

The sudo binary is missing the setuid bit.

Why it fails: A missing setuid bit on /usr/bin/sudo would prevent sudo from gaining root privileges to read the root-only sudoers file, but the immediate failure would be 'sudo: effective uid is not 0' or a similar privilege error, not 'unable to open /etc/sudoers'. The error described indicates sudo did start with root privileges and then failed while opening the policy file. Therefore, while the setuid bit is essential, its absence produces a different symptom.

SELinux is blocking access.

Why it fails: SELinux uses type enforcement, and /etc/sudoers has a default type (etc_t) that the sudo_t domain is allowed to read. An SELinux denial would appear in audit.log as an AVC message and would require a Boolean or context change; it would not be fixed by altering file mode. Since the scenario points to a permission problem after editing the file, and the file's SELinux context is almost always correct, SELinux is not the cause. If SELinux were blocking, you would also see denials even after the mode is corrected.

The filesystem is nearly full

Why it fails: A filesystem at 90% or even 99% capacity is irrelevant when 90G is free; write permission checks happen before any space allocation. Writing to a directory consumes space for a new directory entry, and an 'out of space' condition would typically surface as ENOSPC or a quota error, not as a permission denial. The reported 90G free clearly rules out a full filesystem as the cause.

The directory is owned by root and alice is not root

Why it fails: Unix permission evaluation first identifies the requester's relationship to the object — owner, group member, or other — but it is the mode bits that ultimately grant or deny access. Alice being a non-owner does not by itself prevent access, because group or other permissions could allow writing; for example, mode 770 would let group members write. In this case, the directory is set to 700, so 'others' (Alice) have no write permission, meaning the restrictive mode — not simple ownership — is what blocks her.

The directory has ACLs preventing access

Why it fails: The ls -l output shown in the exhibit has no '+' character after the permission string; a POSIX ACL, if present, is always indicated by a trailing plus sign, as in drwxrwx---+. Without that marker, only standard mode bits are in effect, and no ACL entries are masking or augmenting them. Therefore, the denial is fully explained by the ordinary 700 mode, not by any ACL rule.

Add 'selinux=0' to kernel boot parameters and reboot

Why it fails: Adding `selinux=0` to the kernel command line disables SELinux entirely by bypassing the LSM at boot, which masks the labeling mismatch but does not repair it. This leaves the system with no SELinux protection, violating a typical RHEL security baseline, and is considered only a temporary troubleshooting step. Moreover, after the next reboot, if the parameter is removed, the same mislabeled /run files would still prevent logind from working, so the root cause remains unresolved.

Edit the systemd-logind service unit to add 'Permissions=yes'

Why it fails: The systemd unit parser does not recognize a `Permissions=yes` directive; the only SELinux-related unit options are things like `SELinuxContext=` for explicitly specifying a context. SELinux enforcement is performed by the kernel's Linux Security Module hooks, independent of systemd, so a service option cannot grant permission beyond what the policy allows. Even if you attempted to edit the unit, it would neither relabel existing /run files nor add any SELinux allow rule, making this a fundamentally invalid approach.

Reinstall the systemd-logind package using 'yum reinstall systemd'

Why it fails: Reinstalling the systemd package via `yum reinstall systemd` only reinstalls the binaries, libraries, and unit files from the RPM, replacing the files that the package owns. It does not invoke restorecon, so the SELinux labels on runtime artifacts under /run, like /run/logind or /run/user/*, remain as whatever they were before. Because those mislabeled files are not part of the RPM package database, reinstalling systemd has no effect on their contexts, and logind would still be denied by SELinux.

Use `nice -n 19 ./backup_script` to start the process with lower priority next time.

Why it fails: The nice utility only assigns a modified nice value to a command before it is launched, so it has zero effect on the backup that is already running. You would have to re-execute the backup script with nice, which restarts the job and loses any progress, whereas the current process's priority can only be lowered using renice.

Use `chrt -i 0 <PID>` to set the scheduling policy to idle.

Why it fails: chrt -i 0 <PID> changes the process to the SCHED_IDLE scheduling class, which is an extremely low-priority policy intended for deferential background tasks. Because SCHED_IDLE processes yield almost entirely to every other runnable thread, the backup may stall for excessively long periods and I/O or network timeouts could cause it to fail, and this also requires root privileges.

Use `kill -STOP <PID>` to pause the process and then resume later.

Why it fails: kill -STOP <PID> sends SIGSTOP, which uncondionally halts the process — it cannot be caught or ignored by the application. The backup is frozen in place without flushing output files or updating metadata, so subsequent SIGCONT might resume from an invalid state, and a forgotten resume leaves the backup permanently stopped.

Disable SELinux with setenforce 0 and restart the service.

Why it fails: There is no AVC denial in /var/log/audit/audit.log or the container's journal, so SELinux is not blocking the container startup. Running setenforce 0 only toggles SELinux to permissive until the next reboot and would not create the missing /usr/bin/app.sh inside the image. Even if the container then started, the web application would still fail later because the required script does not exist in the image.

Remove the systemd service and regenerate it with 'podman generate systemd --new --name webapp'.

Why it fails: podman generate systemd reads the current container's configuration and emits a unit file; --new makes that unit create a fresh container from the same webapp image each time it starts. That image still lacks /usr/bin/app.sh, so the regenerated service reproduces the identical failure rather than fixing it. Removing the unit does not remove or rebuild the defective image, so the root cause persists.

Manually create the /usr/bin/app.sh file inside the container using podman exec.

Why it fails: podman exec attaches to a running container's namespaces, but this container is in an exited state because the service failed before reaching a running condition. Attempting to run podman exec webapp touch /usr/bin/app.sh returns an error like 'container is not running' and cannot inject the file. Even if the container could be started, writing to the container's writable layer is ephemeral: a subsequent restart from the same image would discard the change. The fix must be applied at image build time.

systemctl status cron

Why it fails: On RHEL-based systems, the cron daemon's systemd unit is named crond.service, not cron.service. Running 'systemctl status cron' causes systemd to look for cron.service, which does not exist, so it reports 'Unit cron.service could not be found' instead of showing crond's state. To diagnose a failed cron job, you must address the exact unit name; otherwise, you will be misled into thinking the service is absent.

systemctl list-units --type=service

Why it fails: 'systemctl list-units --type=service' outputs every loaded service unit on the system, such as sshd, httpd, and network.service, so it is a broad inventory rather than a targeted check of cron. To find cron in that output you would need to grep or paginate through hundreds of lines, and even then you get only a loaded/active state, not the specific error or journal messages needed to fix a failing job. The outage of one service cannot be efficiently diagnosed this way.

service crond status

Why it fails: The 'service crond status' command is a legacy SysVinit-style interface that modern RHEL systems map through systemctl, so it may display status, but it is deprecated and provides abbreviated information relative to native systemd commands. It typically does not show the process PID, recent journal output, or dependency failures in the same integrated view that systemctl status crond does. RHEL 7+ administrative tasks should use systemctl for accurate and complete service monitoring.

mount -o rw,remount /sysroot

Why it fails: /sysroot is a mount point used only inside the initramfs during early boot; it is not the rescue environment's target directory, which is /mnt/sysimage. Additionally, remounting with -o remount requires the filesystem to already be mounted, but /sysroot is typically not mounted in rescue mode. So this command either fails or operates on the wrong location, leaving the corrupt fstab inaccessible.

chroot /mnt/sysimage

Why it fails: The chroot command changes the apparent root directory for the current process, but it does not mount any filesystem; the target directory must already contain a valid Linux root with /bin/bash and other executables. If you run chroot /mnt/sysimage before mounting the root logical volume, the /mnt/sysimage directory is empty, so chroot fails with an error like "chroot: failed to run command '/bin/bash': No such file or directory". Mounting must always precede chroot.

systemctl rescue

Why it fails: systemctl rescue switches the running system into rescue mode, which is a minimal single-user target, but it relies on the already-mounted root filesystem of the current boot. It does not automatically mount the root filesystem from the hard disk in a recovery environment; in fact, if fstab is corrupted, systemctl rescue may not even boot properly. For fixing a broken fstab, you need to boot from rescue media and mount the disk, not just invoke systemctl rescue.

systemctl status local-fs.target

Why it fails: The systemctl status local-fs.target command reports the current state of the local-fs.target unit, such as whether it is active or failed, but it does not display the underlying cause of a boot problem. It shows only a summary of dependency units and may indicate that a mount unit failed without exposing the specific fstab entry, device node, or mount option that caused the failure. In an emergency shell, you need the actual error messages from the journal, not a high-level status view.

fsck -A

Why it fails: The fsck -A command checks all filesystems listed in /etc/fstab for structural integrity and can repair corruption, but a failure to boot is often caused by an incorrect UUID, missing device, or bad mount option rather than filesystem damage. Running fsck in the emergency shell may be unnecessary or even problematic if filesystems are already mounted, as fsck on a mounted filesystem can cause further corruption. The journal will show the precise reason for mount failure, whereas fsck only addresses a narrow class of issues.

mount -a

Why it fails: The mount -a command attempts to mount every filesystem entry in /etc/fstab and will print the error for the first failing entry, but it does not provide the full boot-time context and may silently succeed for already-mounted pseudo filesystems. In an emergency shell, many required kernel devices or service dependencies are not yet set up, so mount -a can produce misleading or incomplete output. The journal, captured by systemd during boot, records the exact ordering of mount attempts and all related kernel messages, which is far more useful than a single immediate attempt.

The logical volume 'lv_logs' overlaps with 'lv_db'.

Why it fails: LVM manages physical extents inside a volume group and records their allocation in on-disk metadata. The lvcreate and vgcreate commands refuse to allocate the same physical extent to more than one logical volume, so overlapping LVs cannot be created accidentally. If the metadata were manually corrupted to create an overlap, LVM would refuse to activate the volume group, and the system would show a different failure, not simply missing files under /logs. Therefore, an overlap is not a plausible explanation for this mount problem.

The physical volume /dev/sda is not recognized by LVM after reboot.

Why it fails: Physical volumes are identified by an LVM metadata label containing a unique UUID, not by the kernel device name such as /dev/sda. During boot, udev and lvm2-pvscan scan all available block devices and recognize a PV by that metadata, even if the device node name changed. A reboot alone would not make LVM forget a PV unless the device is physically removed, the metadata is wiped, or the device is genuinely unavailable. Since the scenario does not describe hardware loss or metadata corruption, this option is not a plausible cause of the missing /logs mount.

The volume group 'vg_app' is not automatically activated.

Why it fails: By default, LVM activates all volume groups automatically during system startup through the lvm2-activation systemd units. For vg_app not to be activated, the administrator would have to set auto_activation_volume_list in /etc/lvm/lvm.conf or the VG would need to be marked as foreign or have a missing PV. If vg_app were truly inactive, the logical volume would not appear under /dev/mapper, and any fstab entry for it would fail with 'device not found'. Because the described problem is a missing persistent mount, not an inactive device, the VG-activation issue is unlikely and also subordinate to the missing fstab entry.

rm \-f

Why it fails: In `rm \-f`, the backslash is a shell quoting character that escapes the hyphen, so the command receives the single argument `-f`. However, the shell's backslash handling only affects how the word is passed to the program; rm itself still sees an argument with a leading dash and parses `-f` as its force option. Escaping the dash may make the shell leave the argument intact, but it does not tell rm to stop option parsing or to treat the argument as a filename.

rm -f

Why it fails: In `rm -f`, the token `-f` is parsed by rm's option processor as the force flag, not as a file name. The command supplies no operand at all, so rm has nothing to delete; the intent to target a file literally named `-f` is lost because the leading dash is interpreted as an option marker. Without a `--` separator, any argument starting with `-` is treated as an option, and a bare `-f` cannot name a file.

rm "-f"

Why it fails: With `rm "-f"`, the double quotes group the hyphen and letter `f` into one shell word, but the shell's quote removal simply passes the resulting two-character string `-f` to rm. Quotes are processed by the shell before the program runs; rm cares only that the resulting argument begins with `-`, so it is dispatched to the option parser and interpreted as force. Quoting cannot emulate `--` because it has no effect on how the receiving program classifies the argument.

The /tmp is a separate filesystem

Why it fails: The exhibit displays mount points for / and /boot only, with no separate entry for /tmp. On RHEL systems, /tmp is part of the root filesystem unless explicitly mounted as its own device; a separate filesystem would appear as a distinct df row or mount point. Since the output shows no such mount, a dedicated /tmp filesystem is not the reason write access is failing.

There is a filesystem quota enabled

Why it fails: The df output in the exhibit provides no evidence of quota settings, such as the usrquota or grpquota mount options, nor does it show quota-related columns like Quota or Limit. Standard RHEL installations do not enable filesystem quotas by default, and quotas restrict per-user or per-group usage rather than blocking writes to /tmp for the root administrator. Without any quota enablulation in the mount flags, this is not a plausible cause.

The /boot partition is too small

Why it fails: The /boot partition is reported as only 31% full and is a separate, small partition dedicated to kernel and bootloader files. Its size or fullness has no bearing on writability of /tmp, because /tmp resides on the root filesystem. A nearly full /boot might cause kernel updates to fail, but it cannot affect the ability to write to /tmp, making this option incorrect.