# Advanced Shell Scripting and Automation

> Chapter 15 of the Courseiva LINUX-PLUS curriculum — https://courseiva.com/learn/linux-plus/advanced-shell-scripting-and-automation

**Official objective:** 5.1 — Write advanced scripts and automate system tasks

## Introduction

Without understanding advanced shell scripting, you would waste hours every day doing the same boring tasks manually — checking disk space, moving log files, restarting services — and you would inevitably make a mistake that takes down a critical application at 2 AM. This chapter teaches you how to write scripts that think for themselves, turning you from a manual labourer into an automation engineer who can manage a hundred servers with a single command, which is exactly what the XK0-006 exam demands you prove you can do.

## The Head Chef's Recipe Binder Analogy

A Head Chef runs a busy commercial kitchen. Every morning, they don't just cook one dish at a time — that would be chaos. Instead, they have a binder of 'master recipes'. Each recipe is not just a list of ingredients; it contains variables (like 'number of portions'), conditional steps (IF the fish is fresh, use the grill; IF it's frozen, use the oven), loops (repeat the plating process for each table), and error handling (if the sauce splits, whisk in cold butter). The Head Chef can take that one recipe for 'Tomato Basil Pasta' and run it for a party of 10, a banquet of 200, or even a single takeaway order. Each time, the recipe adapts automatically because of the instructions written inside it.

This is the essence of advanced shell scripting. The script is your recipe. Instead of manually typing every command to back up a server, check its disk usage, email the report, and then delete old log files each day, you write a shell script (the master recipe) that does all of that automatically. Variables let you change the target server or the email address without editing the script's logic. Conditional statements (if/then/else) check whether yesterday's backup actually succeeded before trying to delete old files — just like the chef checks if the fish is fresh before choosing a cooking method. Loops (for/while) let you repeat the same task for a list of fifty servers, rather than writing the same command fifty times. Error handling (trap commands) catches problems like a full disk and sends an alert, just as the chef catches a broken sauce before it reaches the customer's plate.

Advanced scripting transforms you from a line cook who follows a single printed order into the Head Chef who designs the kitchen's entire workflow. You stop reacting to fires — you build the systems that prevent them, ensuring consistency, speed, and sanity across a hundred servers at once.

## Core explanation

Welcome to the engine room of Linux administration. Advanced shell scripting is the skill that separates someone who knows a few Linux commands from someone who can automate an entire data centre. Let's start with the absolute basics and build up.

First, what is a shell? A shell is a programme that interprets the commands you type into a terminal. The most common shell on Linux is called Bash (Bourne Again SHell). When you write a shell script, you are simply writing a plain text file that contains a sequence of Bash commands. You can run that file, and the shell will execute those commands in order, just as if you had typed them yourself.

But an advanced script is much more than a plain list of commands. It uses several key concepts to make decisions, repeat actions, and handle errors.

**Variables** are containers for storing data. You define a variable like this: `BACKUP_DIR="/var/backups"`. Later in the script, you can refer to it as `$BACKUP_DIR`. This is powerful because you can change the value once at the top of the script, and every command that uses it automatically updates. In the exam, they love to test that variables are case-sensitive and that you must not leave a space around the equals sign.

**Conditionals** (if, then, else, elif, and case statements) allow your script to make decisions based on conditions. For example:

`if [ -f "/etc/passwd" ]; then`
`  echo "File exists"`
`else`
`  echo "File missing"`
`fi`

The `[ ]` brackets are a test command. They check something — in this case, whether the file `/etc/passwd` exists (`-f`). The condition must be true for the `then` block to run. The exam will test your understanding of the test operators like `-eq` (equal), `-ne` (not equal), `-z` (string is empty), and `-d` (directory exists). A common trap is confusing `=` (for string comparison in test brackets) with `-eq` (for integer comparison).

**Loops** let you repeat a set of commands. The two main types are `for` loops and `while` loops.

A `for` loop iterates over a list of items:

`for server in web01 web02 db01; do`
`  ssh $server "df -h"`
`done`

This example checks the disk space on three servers without you having to run the command three times manually. The `while` loop runs as long as a condition is true, which is useful for reading lines from a file, like a list of IP addresses.

**Functions** allow you to group commands under a single name, like a mini-script inside your script. You write:

`function check_disk {`
`  df -h | grep "/dev/sda1"`
`}`

Then you can call `check_disk` anywhere in your script. This keeps your code organised and prevents you from repeating the same block of commands.

**Exit statuses** are the secret language every command uses to tell your script if it succeeded or failed. Every command returns a number when it finishes. A return code of `0` means success. Any non-zero number (1-255) means failure. You can check this with `$?` immediately after the command runs. Your script can then say: 'If the last command returned non-zero, send an email alert and stop, because the backup failed'.

**Exit statuses** are the secret language every command uses to tell your script if it succeeded or failed. Every command returns a number when it finishes. A return code of `0` means success. Any non-zero number (1-255) means failure. You can check this with `$?` immediately after the command runs. Your script can then say: 'If the last command returned non-zero, send an email alert and stop, because the backup failed'.

**Arguments** are values you pass to a script when you run it. When you run `./myscript.sh monday`, the word 'monday' is stored in the special variable `$1`. `$2` holds the second argument, `$@` holds all arguments as a list. This lets your script process different input each time it runs without editing the code.

**Error handling** is what makes a script robust. A trap command can catch signals like Ctrl+C or system shutdowns and run cleanup code (like deleting temporary files) before the script dies. The `set -e` command tells the script to exit immediately if any command fails, rather than blindly continuing and causing more damage.

Why does all this exist? Before scripting, system administrators memorised long command sequences and typed them over and over. They made typos, they forgot steps, and they wasted hours. Scripting replaced that manual labour with precise, repeatable, testable code. Instead of spending ten minutes every day checking log files on ten servers, a script does it in five seconds. Instead of hoping you remember all seven steps of a software deployment, a script enforces them exactly the same way every single time.

On the XK0-006 exam, you will be expected to read a script and predict its output, identify syntax errors, fill in missing lines, and know which control structure (if, for, while, case) fits a particular scenario. You will not be asked to write a long script from scratch, but you must be able to trace through one and understand the logic of every line.

## Real-world context

Let us step into the shoes of a junior Linux administrator at a mid-sized e-commerce company called 'ShopFast'. The company runs its website on fifteen Linux servers: three web servers, two application servers, one database server, and nine development/test servers. The junior admin's job is to keep them running and healthy.

One Monday morning, the senior admin says: 'Every night at midnight, I need you to check the disk usage on every server. If any disk is over 80% full, delete log files older than 7 days from /var/log. But if the disk is over 95% full, also add a second disk and mount it immediately. Then email me a report of what happened.'

Without scripting, the junior admin would need to:
- Manually SSH into each of the fifteen servers.
- Run `df -h` on each.
- Look at the output, calculate percentages in their head.
- If over 80%, run a `find` command to delete old logs.
- If over 95%, also go to the cloud console, provision a disk, attach it, format it, and mount it.
- Then write an email.

This would take at least an hour, and the chance of mistyping a command or forgetting a server is high. With advanced shell scripting, the process becomes:

**Step 1:** The junior admin writes a master script called `nightly_disk_check.sh`. The script contains:
- A list of all fifteen server IP addresses stored as a variable.
- A `for` loop that iterates over each server.
- Inside the loop, a command to SSH in and run `df -h` and capture the output.
- A conditional `if` statement that parses the output and checks disk percentage.
- If over 80%, it runs a `find /var/log -type f -name "*.log" -mtime +7 -delete` command via SSH.
- If over 95%, it calls a separate function `add_disk` that uses cloud API commands to attach a new disk.
- After the loop, it uses `mail` command to send a summary email to the senior admin with a list of which servers needed action.

**Step 2:** The junior admin sets a cron job (a scheduled task) to run this script every night at midnight. To set a cron job, they edit the cron table with `crontab -e` and add the line:

`0 0 * * * /home/admin/scripts/nightly_disk_check.sh`

This tells the system to run that script at minute 0 of hour 0 (midnight) every day of every month.

**Step 3:** The next morning, the senior admin gets a clean email summarising what happened on every server. If a script had an error (e.g., a server was offline), the script logs it and sends a separate alert. The junior admin does not have to log in until there is a real problem.

The critical difference: the script handles fifteen servers identically, never forgets a step, never makes a typo, and completes the whole task in under a minute. The junior admin now has an extra hour every day to work on more important tasks, like security patches or feature updates. They also have a record (the script itself) of exactly what automated steps were taken, which is valuable for audits and for training new team members.

In a real business, this same principle scales to hundreds or thousands of servers. Companies like Netflix and Google use advanced scripting (often in Python or Go as well as Bash) to orchestrate entire data centres. For a Linux+ candidate, demonstrating you understand how to write and troubleshoot a script that uses variables, conditionals, loops, and error handling is the single most practical skill you can show an employer.

## Exam focus

The XK0-006 exam objective 5.1 — 'Write advanced scripts and automate system tasks' — is not about writing a novel-length script from memory. It is about your ability to read, interpret, and complete script logic. Here is exactly what you will see.

**Question Type 1: Syntax and Logic Completion**
You will be shown a partial script with a blank line. The question asks you to choose the correct line from four options to make the script work as described. For example, the script might start a `for` loop but be missing the `done` statement, or it might have a `while` loop without an increment. You must know the exact structure of `for`, `while`, `if`, and `case` statements. The trap here is that the incorrect options will contain plausible but wrong syntax — a missing semicolon, a misplaced `then`, or using `-eq` with a string variable.

**Question Type 2: Output Prediction**
You are given a complete script and asked: 'What will be the output when this script is run?' This tests your ability to trace variable values and condition evaluation. The traps include:
- A variable that is unset or empty (tested with `-z`).
- A conditional that uses `=` instead of `-eq` for integer comparison.
- A loop that accidentally runs zero times because the initial condition is false.
- The difference between `$*` and `$@` when passing arguments.

**Question Type 3: Debugging with `set -x`**
The exam loves to ask what the `set -x` command does. It enables debug mode, printing each command before it executes. You need to know the difference between `set -x` (trace execution) and `set -e` (exit on error). They will show you a trace output and ask which command caused the error.

**Question Type 4: Variable Expansion and Quoting**
A common exam trap involves double quotes versus single quotes. In Bash, variables are expanded inside double quotes but not inside single quotes. A question might show:

echo "The value is $VAR"
echo 'The value is $VAR'

You must know that the first prints the value of VAR, and the second prints the literal string '$VAR'. They also test brace expansion: `${VAR:-default}` which uses a default value if VAR is unset.

**Key Concepts to Memorise**

- `$?` holds the exit status of the last command.
- `$0` is the name of the script.
- `$1`, `$2`, etc. are positional arguments.
- `$#` is the number of arguments.
- `$@` is all arguments as separate words (preserves spacing).
- `$*` is all arguments as a single word.
- `test` command synonyms: `[ ]` and `[[ ]]` (double brackets are safer for string comparisons in Bash).
- Redirection operators: `>` (overwrite stdout), `>>` (append stdout), `2>` (redirect stderr), `&>` (redirect both).
- `&&` (run next command only if previous succeeded) and `||` (run next command only if previous failed).
- `trap` for signal handling (e.g., `trap "rm -f /tmp/tempfile; exit" INT`).

**Trap Patterns**

- They will give you a script that uses `if [ $VAR = "yes" ]` but if VAR is empty, the script will throw an error because the condition becomes `[ = "yes" ]` which is invalid. The fix is to use double quotes: `if [ "$VAR" = "yes" ]`.
- They will show a `for` loop that uses `seq 1 10` but the system might not have the `seq` command (it's not always installed). The more portable method is `for i in {1..10}` (brace expansion).
- They will test whether you know that `while [ $COUNT -le 10 ]` requires incrementing COUNT inside the loop, or it runs forever.

Focus your revision on tracing small scripts by hand. Write a few two-line scripts on paper and predict the output. If you can do that confidently for scripts with two conditionals and one loop, you will score well on this objective.

## Step by step

1. **1. Write the Script Skeleton** — Create a new file with a `.sh` extension (e.g., `backup.sh`). The first line must be a shebang (`#!/bin/bash`) which tells the system which interpreter to use. Without this line, the script may be interpreted by a different shell (like `sh`) which uses slightly different syntax and will cause errors. Then add comments with `#` to explain what each section does — these are ignored by the shell but are vital for anyone reading the script later, including your future self.
2. **2. Define Variables for Configuration** — Declare all configurable values at the top of the script — things like the backup directory, the email address to send alerts to, and the list of servers. Using uppercase variable names (e.g., `BACKUP_DIR`) is a convention that makes them stand out. This is important because it means you can change the behaviour of the entire script by editing just these few lines, without hunting through the code.
3. **3. Add a Function for Reusable Logic** — Write a function (e.g., `send_alert`) that packages a reusable block of commands. If you need to send an email alert from multiple places in the script, you just call `send_alert` instead of writing the same three `mail` commands again. This reduces duplication, makes the script shorter, and means you only need to fix a bug in one place.
4. **4. Implement the Main Logic with Loops and Conditionals** — This is the heart of the script. Use a `for` loop to iterate through your list of servers. Inside the loop, use conditionals (`if/then/else`) to check if each server is reachable (using `ping` or `ssh -o ConnectTimeout=5`), and if it is, run the backup commands. Check the exit status of each command with `$?` and log the result. This step makes the script smart enough to handle different situations without human intervention.
5. **5. Add Error Handling with `trap` and `set -e`** — Insert `set -e` at the top of the script so it stops on the first error. Then add a `trap` command, such as `trap "echo 'Script interrupted at $(date)' >> /var/log/script.log; exit 1" INT TERM`. This ensures that if someone presses Ctrl+C or the system shuts down mid-backup, the script logs the interruption and exits cleanly rather than leaving a partial backup that could cause inconsistencies.
6. **6. Test the Script Thoroughly** — Before running the script on production servers, run it in a test environment. Use `bash -x script.sh` to enable debug mode, which prints each command as it executes. This lets you verify that variables have the expected values and that conditionals take the correct branches. Also test edge cases: what happens if a server is offline? What if a directory is missing? A well-tested script is one that will not surprise you at 2 AM.
7. **7. Make the Script Executable and Schedule It** — Run `chmod +x script.sh` to give the script execute permission. Without this, you cannot run it with `./script.sh`. Then edit the cron table with `crontab -e` and add a line with the schedule (e.g., `0 2 * * * /home/user/backup.sh`). Save the cron table and verify it with `crontab -l`. Now your script runs automatically without any manual effort, which is the entire point of automation.

## Comparisons

### $@ (all arguments as separate words) vs $* (all arguments as one word)

**$@ (all arguments as separate words):**
- Preserves the original boundaries between arguments
- Safely handles arguments containing spaces or tabs
- Is the preferred choice for looping over arguments

**$* (all arguments as one word):**
- Concatenates all arguments into a single string separated by spaces
- Loses the individual argument structure
- Can cause errors if arguments contain spaces and need to be handled separately

### `for` loop (known iterations) vs `while` loop (unknown iterations)

**`for` loop (known iterations):**
- Used when you know the exact list of items to iterate over
- Syntax: `for i in list; do commands; done`
- Good for looping through a fixed set of servers or filenames

**`while` loop (unknown iterations):**
- Used when you need to loop until a condition changes
- Syntax: `while [ condition ]; do commands; done`
- Good for reading lines from a file until EOF or waiting for a process to finish

### `set -e` (exit on error) vs `trap` (signal handling)

**`set -e` (exit on error):**
- Automatically exits the script if any command returns a non-zero exit status
- Applied globally at the top of the script
- Prevents the script from continuing after an unexpected failure

**`trap` (signal handling):**
- Catches specific signals (like SIGINT or EXIT) and runs a custom cleanup routine
- Used to delete temporary files or log messages before exiting
- Does not normally stop the script on every error — only on the signals you specify

### `[ ]` (POSIX test) vs `[[ ]]` (Bash extended test)

**`[ ]` (POSIX test):**
- Part of the POSIX standard, works in all sh-compatible shells
- Requires spaces around the brackets and around operators
- Cannot use pattern matching (globbing) inside the brackets

**`[[ ]]` (Bash extended test):**
- Bash-specific extension, not POSIX compatible
- Allows pattern matching (e.g., `[[ $string == *.txt ]]`)
- Does not need quoting for variables in string comparisons (safer for empty variables)

## Diagram

_This flowchart shows the step-by-step process of building and deploying an advanced shell script from planning to automated execution._

```mermaid
flowchart TD
    A[Write Script with Shebang] --> B[Define Variables]
    B --> C[Add Functions]
    C --> D[Main Logic: Loops and Conditionals]
    D --> E[Check Exit Status with $?]
    E --> F[Handle Errors with trap]
    F --> G[Make Executable chmod +x]
    G --> H[Schedule with Cron]
    H --> I[Script Runs Automatically]
```

## Common misconceptions

- **Misconception:** Shell scripts are just a list of commands typed in order, so writing them is just about knowing the commands. **Reality:** Advanced shell scripting requires understanding control structures (if, for, while), variables, exit statuses, and error handling. Simply listing commands without logic means the script cannot handle errors, repeat tasks, or make decisions — it will break the first time something unexpected happens. (This misconception comes from beginners who have only seen simple one-liner aliases or very short 'hello world' scripts. They haven't encountered a real automated task that needs to check if a file exists before deleting it, or loop through a list of servers.)
- **Misconception:** A script that has no syntax errors and runs correctly once is a perfect script. **Reality:** A robust script must handle edge cases: missing files, servers being offline, empty variables, and unexpected input. It should use error checking (e.g., checking exit status with `$?` and using `set -e`) to stop or alert if something goes wrong. A script that works perfectly in a controlled lab often fails in production because of these edge cases. (New learners often test their script in an ideal environment — the same directory, with all files present — and assume it will always work. They don't naturally think about failure modes until they have experienced a real outage caused by an unhandled error.)
- **Misconception:** The `if` statement in Bash uses the same test syntax regardless of whether you are testing strings or numbers. **Reality:** String comparison inside `[ ]` uses `=` (one equals sign), while numeric comparison uses `-eq`. Confusing these — for example, writing `[ "$COUNT" -eq "yes" ]` — causes a syntax error or silent failure. You must also quote string variables to prevent errors if the variable is empty. (In many programming languages like Python or JavaScript, a single equals sign is an assignment and double equals `==` is comparison. Bash is different: inside test brackets, `=` compares strings, `-eq` compares integers. Beginners naturally reach for `==` because it's more common in other languages.)
- **Misconception:** Using `$*` and `$@` does the same thing — they both give you all the arguments. **Reality:** `$@` preserves the original boundaries between arguments (each argument is a separate word), which is critical when arguments contain spaces. `$*` combines all arguments into a single word. For example, if the script gets arguments 'hello world' and 'test', `$@` gives two separate items, while `$*` gives one item 'hello world test'. (When you quickly glance at documentation, both `$@` and `$*` are described as 'all arguments', so beginners ignore the subtle difference. They only understand the problem after they write a script that processes a list of filenames with spaces in them and suddenly everything breaks.)
- **Misconception:** The `trap` command is just for debugging scripts, like printing messages. **Reality:** `trap` is primarily for cleaning up resources when a script is interrupted. It can be set to remove temporary files, kill child processes, or log errors before the script exits. It is essential for writing scripts that do not leave the system in a broken state after a crash. (The word 'trap' sounds like 'trap' as in 'catch a mistake', so beginners think it's only used for printing error messages. They rarely encounter cleanup scenarios in small scripts, so the real purpose (resource cleanup) feels abstract.)

## Key takeaways

- A shell script is a plain text file containing commands that the Bash shell executes in sequence, but advanced scripts use variables, conditionals, loops, functions, and error handling to make decisions and automate complex tasks.
- Every command in Linux returns an exit status — `0` for success and any non-zero value for failure — which your script can check using `$?` immediately after the command runs.
- The `for` loop iterates over a list of items, the `while` loop runs as long as a condition is true, and the `until` loop runs as long as a condition is false — choose the loop based on whether you know the number of iterations in advance.
- Always quote your variable expansions inside test brackets (`[ "$VAR" = "yes" ]`) to prevent syntax errors if the variable is empty or contains spaces.
- Use `set -e` at the top of a script to make it exit immediately if any command fails, preventing the script from continuing and potentially causing more damage after an error.
- A `trap` command allows your script to catch signals (like Ctrl+C) and run cleanup actions — such as deleting temporary files — before exiting, which is critical for scripts that create resources on the filesystem.
- The difference between `$@` and `$*` is that `$@` preserves argument boundaries (important when arguments contain spaces), while `$*` lumps all arguments into a single string.
- Scheduling a script with cron (using `crontab -e`) is the standard way to automate periodic tasks, and the cron syntax has five fields: minute, hour, day of month, month, day of week.

## FAQ

**What is the difference between `$@` and `$*` in a Bash script?**

`$@` preserves the original boundaries between arguments, treating each argument as a separate word. `$*` combines all arguments into a single string. If your arguments contain spaces, `$@` is the safe choice.

**Why does my script fail with 'command not found' when I run it with `./myscript.sh`?**

The most common cause is a missing shebang line (`#!/bin/bash`) at the very top of the file, or a Windows-style carriage return (`
`) at the end of each line. Run `head -1 myscript.sh | cat -v` to check for `^M` characters, and use `dos2unix` to fix it.

**How do I make a script continue even if one command fails, but still stop on critical errors?**

Use `set -e` at the top to stop on any error, but prefix specific commands you want to allow to fail with `|| true`. For example, `rm -f /var/tmp/tempfile || true` will not stop the script even if the file does not exist.

**What is the `trap` command used for in a shell script?**

`trap` catches signals (like SIGINT for Ctrl+C) or events (like EXIT) and runs a custom command. It is used to clean up temporary files, close file descriptors, or log a message when a script is interrupted unexpectedly.

**How do I pass arguments to a script and use them inside?**

When you run `./myscript.sh arg1 arg2`, the first argument becomes `$1`, the second becomes `$2`, and the total number of arguments is `$#`. Inside the script, you can reference them like any other variable.

**What does the `set -x` command do, and when should I use it?**

`set -x` enables debug mode, which prints each command (after variable expansion) to the terminal before executing it. You should use it when testing a script to see exactly what is being run, which helps you find logic errors or unexpected variable values.

## Check your understanding

1. **What exit status value indicates a command ran successfully in a Bash script?**

   Answer: An exit status of 0 indicates success. Any non-zero value (1-255) indicates an error.

2. **What is the purpose of the shebang line (e.g., `#!/bin/bash`) at the top of a script?**

   Answer: The shebang line tells the operating system which interpreter to use to execute the script. Without it, the system may use an incompatible shell and produce errors.

3. **In a Bash script, what is the difference between `$@` and `$*`?**

   Answer: `$@` treats each argument as a separate word, preserving spaces, while `$*` treats all arguments as a single string. `$@` is safer for arguments containing spaces.

4. **How do you schedule a Bash script called `/home/user/backup.sh` to run at 2 AM every night?**

   Answer: Edit the cron table with `crontab -e` and add the line `0 2 * * * /home/user/backup.sh`. The five fields are minute, hour, day of month, month, day of week.

5. **What does the `trap` command do in a Bash script, and why would you use it?**

   Answer: `trap` catches system signals (like Ctrl+C) or script events (like EXIT) and runs a specified command. You use it to clean up temporary files or log messages when the script is interrupted.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/linux-plus/advanced-shell-scripting-and-automation