# System Hardening: Host OS and Kernel Security

> Chapter 11 of the Courseiva CNCF-CKS curriculum — https://courseiva.com/learn/cncf-cks/system-hardening-host-os-security

**Official objective:** 4.1 — Harden the underlying host operating system and kernel parameters

## Introduction

System hardening is the practice of deliberately locking down the underlying operating system and the Linux kernel to eliminate unnecessary vulnerabilities before an attacker can exploit them. For the CKS exam, you must understand that even a perfectly configured Kubernetes cluster can be compromised if the host OS beneath it has weak passwords, open ports, or misconfigured kernel parameters. Mastering this concept means you can prevent attacks that target the server itself, not just the containers running on it.

## The Fortified Restaurant Kitchen Analogy

A busy restaurant kitchen on a Friday night. The head chef, Maria, has just received a health inspector warning about a nearby place that got shut down after a rat infestation and a gas leak. She decides to harden her own kitchen before it is too late.

First, she installs heavy-duty screens on every window and door — not just the main entrance. These screens are like the kernel parameters that control how the operating system handles network traffic. She blocks any ventilation shaft wider than a centimetre, which is like disabling unused ports and services on the host OS. She then fits all electrical sockets with childproof covers so that only specific plugs can fit — this mirrors setting filesystem permissions and using mandatory access controls like AppArmor or SELinux to restrict what processes can do.

Next, Maria creates a strict logbook for deliveries. Every supplier must sign in, and any ingredients left unattended get thrown out. This is analogous to auditing and logging all system calls and changes to critical system files. She assigns each chef a specific coloured apron: red for the grill, blue for pastries, green for vegetables. If a chef in a red apron tries to enter the pastry station, an alarm sounds. That is exactly what kernel security modules do — they enforce rules about which users and programs can interact with which resources.

Finally, she removes all but one key to the walk-in freezer and gives it only to herself. That single lock is like the principle of least privilege applied to the root account: only the admin gets the master key, and even then, only when absolutely necessary. The kitchen stays open, the food is safe, and the inspectors leave happy. Your host OS and kernel are that kitchen: they need the same kind of obsessive, layered defence to keep attackers out.

## Core explanation

When you run Kubernetes, every node is a physical or virtual server running an operating system — almost always Linux. The operating system, or host OS, manages all the hardware resources and runs all the software. The kernel is the absolute core of that operating system, a piece of software that controls everything: memory, CPU, devices, and network traffic. System hardening means making both the host OS and the kernel as resistant to attack as possible.

Let us start with the host OS. A fresh installation of Linux comes with thousands of programs, services, and features enabled by default. Many of these are unnecessary for running Kubernetes. For example, a service called CUPS lets you print documents. On a server that never prints anything, CUPS is just a potential entry point for an attacker. The same goes for things like FTP servers, email services, or the X Window System graphical interface. Hardening means removing or disabling every service, package, and user account that is not absolutely required. This is often called reducing the attack surface.

Next come kernel parameters. The Linux kernel has hundreds of settings you can adjust using a tool called sysctl. These parameters control behaviours as fundamental as how the kernel responds to network traffic, how memory is allocated, and how processes communicate with each other. For security, you want to apply a set of known-safe values. For example, you can set a parameter that prevents a process from gaining more privileges than it was originally given — this is called preventing privilege escalation. You can set another parameter that stops a user from seeing processes owned by other users, which prevents one person's container from spying on another's. These parameters live in files under /proc/sys/ and can be changed at runtime.

A critical part of kernel hardening is using Linux Security Modules (LSMs) like AppArmor or SELinux. These are kernel features that enforce security policies on individual programs. Imagine you run a web server container. AppArmor can say: this web server can only read files in /var/www/html, can only open port 80, and can never run other programs. If an attacker compromises the web server and tries to run a shell command, AppArmor blocks it at the kernel level. SELinux works similarly but uses a more complex system of labels and rules. The CKS expects you to know how to enable and configure these tools.

Another major area is filesystem hardening. The host OS has a filesystem that stores all its files. You can set permissions so that only the root user can write to critical directories like /etc or /usr. You can also mount filesystems with special options: for example, mounting a partition as read-only means nothing can be written to it, even by the root user. Kubernetes itself uses container images that are often read-only, but the host's filesystem must also be locked down. You should also ensure that the filesystem itself is encrypted, typically using LUKS (Linux Unified Key Setup), so that even if a hard drive is stolen, the data cannot be read.

Finally, system hardening covers user accounts and authentication. By default, Linux has many system accounts that you never use. You should lock them. You should enforce strong password policies, use key-based SSH authentication instead of passwords, and configure sudo so that only specific users can run privileged commands. The principle of least privilege applies everywhere: every user and every process should get exactly the permissions it needs to do its job, and nothing more. Together, these measures make it vastly harder for an attacker to move from a compromised container to the host operating system or from the host to other machines on the network.

## Real-world context

Imagine you are the new security engineer at a mid-sized e-commerce company. The company runs its entire product catalogue, checkout system, and customer database on a Kubernetes cluster spread across 10 nodes. The nodes are virtual machines running Ubuntu Server. Your boss tells you that a penetration test last month found that an attacker could break out of a container and gain root access to the host node. You have been tasked with hardening every single node.

Your first step is to audit the current configuration. You SSH into one node and run a command like ps aux to see all running processes. You see a dozen services you do not recognise. You check which network ports are open using netstat -tulpn and find that port 25 (used for email) is listening — but the company does not send email from these nodes. You also see that FTP is running, which is a huge risk because FTP sends passwords in plaintext. You document everything and create a plan.

Next, you begin removing unnecessary packages. On each node, you use the package manager apt to remove packages like cups, vsftpd, and postfix. You also purge any unused GUI components because a headless server does not need a desktop environment. After removal, you run a tool called Lynis, which is an open-source security auditing tool, to check for common misconfigurations. It flags that the root account is accessible via SSH. You edit the SSH configuration file /etc/ssh/sshd_config to set PermitRootLogin no and then restart the SSH service.

Now you tackle kernel parameters. You create a file called /etc/sysctl.d/99-kubernetes-security.conf and add parameters that the Kubernetes documentation recommends. For example, you set net.ipv4.conf.all.rp_filter = 1 to enable reverse path filtering, which helps prevent IP spoofing. You set kernel.kptr_restrict = 2 to hide kernel pointers from unprivileged users. You set kernel.dmesg_restrict = 1 so that only root can read kernel log messages. After adding all the parameters, you run sysctl --system to apply them immediately.

Then you configure AppArmor. The company uses an nginx web server container. You create an AppArmor profile that restricts nginx to only reading files in its own webroot, binding to port 80 and 443, and never running executables. You load the profile into the kernel using apparmor_parser. You then modify the Kubernetes Pod specification to include an annotation that tells the container runtime to apply that AppArmor profile. You deploy the change and verify that the profile is loaded using aa-status.

Finally, you enable auditing with auditd. You configure rules that log every time someone tries to change the system time, every failed login attempt, and every modification to critical files in /etc. You also set up log forwarding so that all audit logs go to a central Security Information and Event Management (SIEM) system. When the next penetration test happens, the tester finds that container breakout is now impossible, and the host OS is completely locked down. Your boss approves the changes and you document everything for the compliance team.

## Exam focus

The CKS exam is intensely practical, and the System Hardening section (Objective 4.1) directly tests your ability to secure a Linux host that runs Kubernetes. You will not be asked to write essays — you will be given a terminal on a cluster node and asked to fix misconfigurations or apply specific settings.

Here are the exact concepts the exam tests repeatedly:

- Kernel parameter hardening using sysctl: You must know which parameters are important and how to make them persistent. The exam often gives you a node where kernel parameters are insecure, and you must create a configuration file in /etc/sysctl.d/ and reload it. Common parameters include net.ipv4.ip_forward (set to 0 if not needed), net.bridge.bridge-nf-call-iptables (set to 1 for Kubernetes networking), and kernel.panic (set to a value like 10 to auto-reboot on kernel panic).
- AppArmor profiles: You must be able to create, load, and apply an AppArmor profile to a Pod. The exam will give you a scenario where a container needs restricted access. You must write a profile using the apparmor_parser command, then add the correct annotation to the Pod YAML. The trap is that beginners forget to load the profile before applying the Pod, or they use the wrong syntax for the annotation.
- Filesystem permissions and mount options: You might be asked to mount a filesystem with the noexec option to prevent binaries from being executed on a partition, or to set a directory as read-only (ro). The exam tests your knowledge of /etc/fstab and the mount command. A common trap is forgetting that the noexec option does not apply to scripts executed by /bin/sh — you must also use nodev and nosuid.
- Removing unnecessary packages and services: The exam may present a node with a known insecure service running, like telnet or rsh. You need to stop the service and remove the package. They test whether you know the difference between apt remove (keeps config files) and apt purge (removes everything). They also check if you disable the service from starting at boot using systemctl disable.
- User account hardening: Expect a question where you must lock a system account (usermod -L), delete an unnecessary user (userdel), or change the SSH configuration to disable root login and password authentication. A common trap is editing the SSH config but forgetting to restart the sshd service.

The exam loves to create scenarios where you have a node that is fully patched but still vulnerable because of a misconfigured kernel parameter or an unused service. The correct answer pattern is always to identify the security issue, apply the fix using the correct command or configuration file, and then verify the change. Memorise the specific sysctl parameter names and the AppArmor annotation format (container.apparmor.security.beta.kubernetes.io/containername: localhost/profilename). You will also need to know how to check audit logs using ausearch or aureport. Do not waste time on overly complex attacks — the exam focuses on straightforward, documented best practices.

## Step by step

1. **Inventory Current State** — Log into the node and run commands like ps aux, netstat -tulpn, and systemctl list-units to see all running processes, open ports, and active services. This gives you a complete picture of what is running so you know what to disable or remove.
2. **Remove Unnecessary Packages and Services** — Use the package manager (apt for Debian/Ubuntu, yum for RHEL) to purge packages that are not needed, such as CUPS, FTP, or mail servers. Disable and stop services with systemctl disable and systemctl stop. This shrinks the attack surface by eliminating unused software that could harbour vulnerabilities.
3. **Apply Secure Kernel Parameters** — Create a file like /etc/sysctl.d/99-security.conf and add parameters that prevent privilege escalation, restrict kernel memory access, and improve network security. Then run sysctl --system to apply them. These parameters modify how the kernel behaves at runtime to thwart common exploit techniques.
4. **Configure Mandatory Access Control (MAC)** — Choose AppArmor or SELinux, then create and load a security profile that restricts what a specific process (e.g., nginx) can do. For AppArmor, write a profile file and load it with apparmor_parser. For Kubernetes, annotate the Pod to apply the profile. This prevents a compromised container from damaging the host.
5. **Harden User Accounts and SSH** — Lock unused system accounts with usermod -L, delete unnecessary users with userdel, and edit /etc/ssh/sshd_config to disable root login and password-based authentication. Restart the SSH service. This ensures that only authorised users with strong keys can access the node.
6. **Set Filesystem Permissions and Mount Options** — Edit /etc/fstab to add mount options like noexec, nodev, and nosuid to partitions like /tmp or /var/tmp where attackers might write scripts. Also set correct ownership and permissions on critical directories like /etc/shadow so only root can read them. This stops attackers from executing malicious code or escalating privileges via writable locations.
7. **Enable Auditing and Logging** — Install and configure auditd, add rules to log important events (e.g., changes to /etc/passwd, failed login attempts), and ensure logs are forwarded to a central system. This provides a record of any security incidents and helps with forensic analysis after an attack.

## Comparisons

### sysctl -w (runtime change) vs /etc/sysctl.d/ file (persistent change)

**sysctl -w (runtime change):**
- Applies immediately but only for the current session
- Lost after a system reboot
- Useful for testing a parameter before making it permanent

**/etc/sysctl.d/ file (persistent change):**
- Applies on every boot, survives reboots
- Must be loaded with sysctl --system after creation
- Standard practice for production systems

### AppArmor vs SELinux

**AppArmor:**
- Uses path-based policy rules (e.g., allow read on /etc/passwd)
- Profile syntax is simpler and easier to read
- Commonly used on Debian/Ubuntu distributions

**SELinux:**
- Uses label-based policy rules (e.g., allow httpd_t to read etc_t)
- More complex configuration with booleans and contexts
- Commonly used on RHEL/CentOS distributions

### apt remove vs apt purge

**apt remove:**
- Removes the package binary but leaves configuration files on the system
- Configuration files can still be a security risk or cause confusion
- Faster if you might reinstall the package later

**apt purge:**
- Removes both the package and all its configuration files
- Cleaner outcome for security hardening
- Prevents leftovers from being exploited or interfering

### Disabling root login in SSH vs Disabling password authentication in SSH

**Disabling root login in SSH:**
- Stops direct SSH login as the root user
- Users must still login as a regular user and then use sudo
- Alone, it does not protect against brute-force attacks on other users

**Disabling password authentication in SSH:**
- Stops all password-based authentication, forcing SSH key use only
- Prevents brute-force attacks against any user account
- Combined with root login disabled, provides strong SSH security

## Diagram

_This diagram shows the key layers of host OS and kernel hardening: starting from the Linux kernel with its security modules and sysctl parameters, then up to filesystem permissions, service management, user accounts, and auditing._

```mermaid
flowchart TD
    A[Host Operating System] --> B[Kernel]
    B --> C[Linux Security Modules]
    C --> D[AppArmor or SELinux]
    B --> E[Sysctl Parameters]
    E --> F[Memory Protection]
    E --> G[Network Security]
    E --> H[Privilege Escalation Prevention]
    A --> I[Filesystem Permissions]
    I --> J[Mount Options: noexec, nodev, nosuid]
    A --> K[Service Management]
    K --> L[Remove Unused Packages]
    K --> M[Disable Unnecessary Services]
    A --> N[User Account Hardening]
    N --> O[Lock System Accounts]
    N --> P[Secure SSH Configuration]
    A --> Q[Auditing with auditd]
```

## Common misconceptions

- **Misconception:** I only need to harden the control plane nodes; worker nodes are less important. **Reality:** Every node in a Kubernetes cluster, including worker nodes, must be hardened because a compromised worker node can give an attacker access to the entire cluster network and allow them to pivot to the control plane. (Beginners often think the control plane is the only valuable target, but containers running on worker nodes can be exploited to reach other containers and the host itself.)
- **Misconception:** Setting kernel parameters with sysctl -w makes the change permanent. **Reality:** Using sysctl -w only changes the value until the next reboot. To make changes persistent, you must write them to a file in /etc/sysctl.d/ or /etc/sysctl.conf and run sysctl --system. (The transient nature of runtime changes is counterintuitive to people used to GUI settings that stay saved; they do not realise Linux kernel parameters reset at boot.)
- **Misconception:** AppArmor and SELinux are the same thing and interchangeable. **Reality:** While both are Linux Security Modules, AppArmor uses path-based policies (e.g., allow read on /etc/passwd) and SELinux uses label-based policies (e.g., allow httpd_t to read etc_t files). They have different configuration syntaxes and tools. (The names sound similar and both enforce mandatory access control, so beginners assume they can use whichever with the same commands, which leads to errors on exam tasks.)
- **Misconception:** If I uninstall a service with apt remove, it is completely gone from the system. **Reality:** apt remove leaves configuration files behind. You must use apt purge to remove both the package and its configuration files, otherwise leftover config can still be a risk or cause confusion. (The word 'remove' sounds final, but the default behaviour in Debian-based systems is to preserve config files in case you reinstall the package.)
- **Misconception:** Disabling root login via SSH is enough to secure SSH access. **Reality:** Disabling root login is important, but you also need to disable password authentication entirely and use only SSH keys, set a maximum authentication retries limit, and use AllowUsers or AllowGroups to restrict who can connect. (People focus on the root account but forget that any user with a weak password can still be compromised via SSH brute-force attacks.)
- **Misconception:** Kernel hardening is only about network parameters. **Reality:** Kernel hardening covers many areas including memory protection (kptr_restrict, dmesg_restrict), privilege escalation prevention (kernel.unprivileged_userns_clone), filesystem security (fs.protected_hardlinks, fs.protected_symlinks), and execution control (kernel.exec-shield). (Network parameters are the most obvious and well-documented, so learners often stop there and miss critical non-network settings.)

## Key takeaways

- System hardening reduces the attack surface by removing every unnecessary service, user, and open port from the host operating system.
- Kernel parameters set via sysctl control fundamental security behaviours like IP forwarding, memory pointer hiding, and process privilege escalation prevention.
- AppArmor and SELinux are Linux Security Modules that enforce mandatory access control policies at the kernel level, restricting what a process can do even if it has root privileges.
- To make kernel parameters permanent, write them to a file in /etc/sysctl.d/ and run sysctl --system — never rely on sysctl -w alone.
- The principle of least privilege applies to every user, process, and container: grant only the permissions needed for the specific task.
- Auditing with auditd allows you to log all security-relevant events on the host, which is essential for detecting and investigating breaches.
- Filesystem mount options like noexec, nodev, and nosuid prevent attackers from running malicious binaries or creating device files on specific partitions.
- Always use apt purge instead of apt remove when deleting unnecessary software packages to eliminate their configuration files as well.

## FAQ

**What is the difference between host OS hardening and container hardening?**

Host OS hardening locks down the underlying Linux server itself — its kernel, services, user accounts, and filesystem — while container hardening focuses on the container image and runtime. Both are needed because a weak host can compromise all containers running on it.

**Do I need to know both AppArmor and SELinux for the CKS exam?**

No, the CKS exam typically tests one of them depending on the distribution provided. You should understand both conceptually, but the exam tasks will specify which one is available. AppArmor is more common on Ubuntu-based clusters, while SELinux is common on CentOS or RHEL.

**How do I make sysctl settings survive a reboot?**

You must write the settings to a file in the /etc/sysctl.d/ directory (e.g., /etc/sysctl.d/99-custom.conf) and then run sysctl --system to reload them. Changes made with sysctl -w alone are lost on reboot.

**What does the kernel parameter kernel.kptr_restrict do?**

It controls whether unprivileged users can see kernel memory addresses in /proc files and other interfaces. Setting it to 2 hides those addresses from everyone except root, which makes it harder for an attacker to craft reliable exploit payloads.

**Is it enough to just install security patches on the host OS?**

No, patching only fixes known vulnerabilities in existing software. System hardening is about removing unnecessary software and restricting behaviour so that even unpatched components cannot be exploited effectively. You must do both.

**Can I use the same kernel parameter settings on every node?**

Most security parameters apply universally, but some may interfere with Kubernetes functionality. For example, net.ipv4.ip_forward must be set to 1 for pod-to-pod communication, and net.bridge.bridge-nf-call-iptables must be set to 1 for iptables to work correctly with bridge traffic. Test changes in a staging environment first.

## Check your understanding

1. **Which command do you use to make a sysctl parameter change permanent across reboots?**

   Answer: You write the parameter to a file in /etc/sysctl.d/ (e.g., /etc/sysctl.d/99-security.conf) and then run sysctl --system.

2. **What is the purpose of the AppArmor annotation in a Kubernetes Pod spec?**

   Answer: The annotation tells the container runtime to apply a specific AppArmor profile to a container, enforcing the security policy defined in that profile at the kernel level.

3. **What does the mount option 'noexec' do when applied to a filesystem partition?**

   Answer: The noexec option prevents any binary or script from being executed on that partition, which blocks attackers from running malicious code stored there.

4. **Why should you disable password authentication in SSH on a hardened host?**

   Answer: Disabling password authentication forces the use of SSH keys, which are resistant to brute-force attacks and do not expose passwords to interception or guessing.

5. **What is the difference between apt remove and apt purge in the context of host hardening?**

   Answer: apt remove deletes the package but keeps its configuration files on disk, while apt purge removes both the package and all its configuration files for a cleaner state.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cncf-cks/system-hardening-host-os-security