# Text Processing and Filtering Tools

> Chapter 4 of the Courseiva LF-LFCS curriculum — https://courseiva.com/learn/lf-lfcs/text-processing-and-filtering-tools

**Official objective:** 1.4 — Use text processing tools to extract and manipulate data

## Introduction

Text processing and filtering tools. They solve the problem of finding exactly what you need inside mountains of messy text files, without opening them manually. For the LFCS exam, these tools are essential because a sysadmin spends half their life reading logs, parsing configuration files, and extracting specific values from command output.

## The Recipe Book Analogy

12 ingredients sit on your kitchen counter: flour, eggs, sugar, butter, milk, vanilla, salt, baking powder, cocoa powder, chocolate chips, nuts, and sprinkles. You have a recipe book with 20 different recipes. Each recipe is a command that tells you exactly which ingredients to pick, in what order, and what to do with them.

In IT, text processing works the same way. Your 'ingredients' are log files, configuration files, or any text data. The 'recipes' are commands like `grep`, `awk`, `sed`, `cut`, `sort`, and `uniq`. Each command does one specific thing: `grep` finds lines containing a word (like picking only the chocolate chips from the bowl), `cut` removes columns (like slicing a carrot into sticks), `sort` arranges everything in order (like organising spices alphabetically), and `uniq` removes duplicates (like throwing away two identical measuring cups).

You never cook by memorising every recipe. Instead, you learn which recipe to use for which result. If you want a chocolate chip cookie, you don't use the soup recipe. Similarly, if you need to find all error messages in a 10,000-line server log, you don't use `cut` — you use `grep`. The power comes from combining recipes: first sort your list, then remove duplicates, then find the most frequent item. That's chaining commands with pipes (`|`), like following a multi-step recipe from one card to the next.

## Core explanation

Text processing tools are small, focused commands that manipulate text in a terminal. A terminal is a text-based interface where you type commands and see results. Instead of using a mouse or clicking buttons, you type instructions. These tools were created in the early days of Unix, when computers had no graphics, only text terminals. They survive because they are fast, scriptable, and powerful.

Let us start with the basic tools you will see on the exam.

`cat` (short for concatenate) prints the entire content of a file to the screen. If you run `cat /var/log/syslog`, you see every line of the system log. This is the simplest tool, but for large files it is overwhelming.

`grep` (Global Regular Expression Print) searches for lines that match a pattern. For example, `grep 'error' /var/log/syslog` shows only lines containing the word 'error'. A pattern can be a plain word, a phrase with spaces in quotes, or a regular expression — a special syntax for matching complex patterns like phone numbers or email addresses. The basic usage is `grep [options] pattern [file...]`. Common options include `-i` to ignore case (so 'Error' matches 'error'), `-v` to show lines that do NOT match, `-c` to count matching lines, and `-r` to search recursively through directories.

`cut` extracts columns or fields from each line of a file. Think of a spreadsheet-like table where columns are separated by a delimiter — often a comma in CSV (Comma-Separated Values) files or a colon in system files like `/etc/passwd`. The command `cut -d ':' -f 1,3 /etc/passwd` says: use colon as delimiter (`-d ':'`), and print the first and third fields (`-f 1,3`). This would show you usernames and their user IDs.

`sort` arranges lines in a specified order. By default it sorts alphabetically. `sort -n` sorts numerically. `sort -r` reverses the order. `sort -k 2` sorts by the second field (column). This is useful before piping output to `uniq`, because `uniq` only detects adjacent duplicates — so you must sort first to group identical lines together.

`uniq` (short for unique) removes consecutive duplicate lines. It is almost always used after `sort`. The command `sort file.txt | uniq` gives you a list with each line appearing only once. `uniq -c` prepends a count of how many times each line appeared. `uniq -d` shows only the lines that are duplicated.

`wc` stands for word count. `wc -l` counts lines, `wc -w` counts words, `wc -c` counts bytes (roughly characters). Common usage: `wc -l file.txt` tells you how many lines are in the file.

`head` and `tail` show the beginning or end of a file. `head -n 5` shows the first 5 lines. `tail -n 10` shows the last 10 lines. `tail -f` follows a file in real time, displaying new lines as they are written — this is the standard way to watch log files live during troubleshooting.

`tr` translates or deletes characters. `tr 'a-z' 'A-Z'` converts all lowercase letters to uppercase. `tr -d ' '` deletes all spaces. It reads from standard input (keyboard or pipe) and writes to standard output.

`tee` reads from standard input and writes to both standard output and a file. It is like a T-pipe in plumbing: the data goes to two places at once. `echo 'hello' | tee file.txt` prints 'hello' on the screen and saves it to `file.txt`.

Now, the real power comes from combining these tools with pipes. A pipe, represented by the vertical bar `|`, sends the output of one command as input to the next command. For example:

`cat /var/log/syslog | grep 'error' | grep -v 'database' | wc -l`

This reads the syslog file, keeps only lines containing 'error', removes lines that also mention 'database', and then counts how many lines remain. This is a pipeline.

Another essential concept is regular expressions (regex). A regular expression is a pattern that describes a set of strings. For example, the regex `^[A-Z].*` matches any line that starts with a capital letter. The caret `^` anchors to the start of a line, `[A-Z]` matches any single uppercase letter, and `.*` matches any characters after it. `grep` uses regex by default, though you can use `-E` for extended regex which supports more patterns like `+` (one or more) and `?` (zero or one).

These tools replace manual searching through files with a text editor. Imagine a 500,000-line log file from a web server. Manually scrolling to find requests that returned a 404 error is impossible. With `grep '404' access.log`, the answer appears instantly. This is why sysadmins cherish the command line.

## Real-world context

Imagine you are the sole sysadmin at a mid-sized e-commerce company. It is 2 AM. A monitoring alert says the website is returning 500 Internal Server Errors for 1% of users. The developer team is asleep. You must pinpoint the cause from log files.

First, you connect to the server via SSH (Secure Shell — an encrypted remote connection). You navigate to the application log directory: `cd /var/log/myapp`. The application writes logs to a file called `app.log`. This file is rotated every hour, so there is `app.log.1`, `app.log.2.gz` (compressed), etc.

You start with `tail -n 200 app.log` to see the most recent entries. You spot lines with '500' but they are mixed with '200' (success) and '404' (not found). You need to isolate only the errors.

You run `grep '500' app.log` and get 1,200 lines. Too many to read. You refine: `grep '500' app.log | grep -v 'healthcheck'` — this removes lines from automated health checks that always trigger harmless 500s. Down to 800 lines.

You suspect a specific API endpoint `/checkout` is failing. You search: `grep '/checkout' app.log | grep '500'`. Now 150 lines. You want to see the timestamps and error messages:

`grep '/checkout' app.log | grep '500' | cut -d ' ' -f 1,4,8-`

Assuming the log format is: timestamp, level, endpoint, status, message. This shows you the first field (timestamp), the fourth field (endpoint), and the eighth field onwards (message). The `cut` command here is splitting each line by spaces (`-d ' '`).

You notice all errors happen when a parameter `user_id` equals 'guest'. You hypothesise the checkout logic fails for guest users. You check if this is a pattern: `grep '/checkout' app.log | grep '500' | grep 'guest' | wc -l` returns 148, meaning 148 of the 150 errors involve guest users. You have your culprit.

You save the evidence: `grep '/checkout' app.log | grep '500' | grep 'guest' > /tmp/error_evidence.txt`. The `>` symbol redirects output to a file. For a permanent record, you might also use `tee`: `grep '/checkout' app.log | grep '500' | grep 'guest' | tee /tmp/evidence.log | wc -l`. This saves the matching lines to a file and prints the count to your screen.

Now you need to check if any other endpoints have similar patterns. You run `grep '500' app.log | cut -d ' ' -f 4 | sort | uniq -c | sort -rn`. This pipeline: finds all 500s, extracts the endpoint field, sorts the endpoints alphabetically, counts unique occurrences with `uniq -c`, then sorts the results in reverse numeric order (`-rn`). The output shows you which endpoint has the most 500 errors, ranked from highest to lowest.

You see `/checkout` at the top with 150 errors, then `/login` with 12 errors, then `/home` with 2. The pattern is clear. You write a summary in a ticket for the developers, attach the evidence file, and leave a note: 'Guest user checkout logic appears broken. Priority: high. Log snippet attached.'

This workflow saved you 40 minutes of manual scrolling. The tools let you filter millions of lines in seconds.

## Exam focus

The LFCS exam tests Objective 1.4 by giving you a command-line scenario and asking which tool or pipeline produces a specific result. You will see questions like: 'Given a file with comma-separated values, which command extracts the second and fourth columns?' or 'Which option to `grep` matches whole words only?'

Here are the exact concepts the exam targets:

`grep` options are heavily tested. You must know:
- `-i` for case-insensitive search
- `-v` for inverted match (show lines that do NOT match)
- `-c` for counting matches
- `-w` for whole word match (so `grep -w 'cat'` matches 'cat' but not 'catalog')
- `-r` or `-R` for recursive search through directories
- `-l` to list only filenames with matches, not matching lines
- `-E` for extended regular expressions
- `-P` for Perl-compatible regex (though this is less tested)

`cut` is tested with the `-d` and `-f` options. You must remember that `cut` requires a delimiter flag if fields are not separated by tabs. A common trap: `cut -f 2` without `-d` works only if the delimiter is a tab character, not a space or comma.

`sort` options to memorise:
- `-n` for numeric sort (treats '10' as greater than '2')
- `-r` for reverse order
- `-k` to specify which field to sort by
- `-t` to specify a field separator (like `-t ':'`)
- `-u` to output only unique lines (essentially sort then uniq in one step)

`uniq` options:
- `-c` to prefix each line with a count
- `-d` to show only duplicated lines
- `-u` to show only unique lines
- `-i` to ignore case when comparing lines

`wc` options: `-l` (lines), `-w` (words), `-c` (bytes/characters). The exam loves asking for line count alone.

`head` and `tail`: remember that `head -n 5` shows first 5 lines, but `head -5` also works. `tail -f` follows a file — used for monitoring logs in real time. `tail -n +5` shows lines starting from line 5 onwards.

`tr` is less common but appears: `tr '[:upper:]' '[:lower:]'` converts case. `tr -d` deletes characters. `tr -s` squeezes repeated characters into one (e.g., removing extra spaces).

Redirection operators are tested side-by-side with these tools:
- `>` replaces a file
- `>>` appends to a file
- `|` pipes output to another command
- `2>&1` redirects standard error to standard output
- `&>` redirects both stdout and stderr

Exam traps:
- They may give a command like `cut -d: -f1 /etc/passwd | sort` and ask what it does. The correct answer: prints usernames in alphabetical order.
- They may replace `sort` with `uniq` without a preceding sort, and the answer is that `uniq` only removes consecutive duplicates, so it fails to remove all duplicates.
- They may use `grep -v error` and ask what it does: it shows lines that do NOT contain 'error'.
- They may ask 'which command counts the number of unique error types in a log?' The pipeline is `grep 'error' logfile | sort | uniq -c | wc -l` — but careful: `uniq -c` counts occurrences per type, `wc -l` counts types.
- They may test `tee` by asking which command saves output to a file while still displaying it on screen. Many beginners pick `>` or `>>` but those do not show on screen. The answer is `tee`.

Key definitions to memorise:
- Standard input (stdin): input stream, usually from keyboard
- Standard output (stdout): output stream, usually the screen
- Standard error (stderr): error stream, also the screen by default
- Pipe: connects stdout of one command to stdin of another
- Regular expression: a pattern that describes text to match
- Delimiter: a character that separates fields in a line

## Step by step

1. **Identify the Data Source** — Decide which file or command output contains the text you need to process. For example, a log file like `/var/log/syslog` or the output of `ls -l`. You cannot filter what you have not located. Use `cat` to preview a small portion if unsure.
2. **Choose the Primary Filter Tool** — Pick the right tool for the first operation. If you need lines containing a keyword, use `grep 'keyword'`. If you need specific columns, use `cut -d' ' -f2`. If you need to see just the first few lines, use `head`. Choosing the wrong tool first wastes time and produces meaningless results.
3. **Apply the Filter with Options** — Add options to refine the tool's behaviour. For `grep`, use `-i` for case-insensitive, `-v` to exclude, `-w` for whole words. For `cut`, you must specify the delimiter with `-d` and the field numbers with `-f`. Options turn a blunt tool into a precise scalpel.
4. **Chain Commands with Pipes** — Use the pipe `|` to send the output of one command into the next. Example: `grep '500' log.txt | cut -d ' ' -f 4 | sort | uniq -c`. Each pipe refines the data further. A pipeline is more powerful than any single command because it combines strengths.
5. **Save or Display the Final Result** — Decide what to do with the filtered output. Use `>` to save to a file, `>>` to append to an existing file, or let it print to the screen. Use `tee` if you want both. This step matters because without it, the filtered data disappears from the terminal after the command finishes.
6. **Verify the Output** — Check a sample of the output to confirm your pipeline produced the expected result. Use `head` or `wc -l` to see line counts. A single mistake in a delimiter or field number can produce empty or wrong output. Verification prevents disaster from a bad filter.

## Comparisons

### grep vs find

**grep:**
- Searches inside files for text patterns
- Outputs lines from files that match the pattern
- Used with regular expressions for complex searches

**find:**
- Searches for files and directories by name or attribute
- Outputs file paths, not content inside files
- Uses criteria like -name, -type, -size, not regex by default

### sort | uniq vs sort -u

**sort | uniq:**
- A two-command pipeline: first sort, then uniq
- Allows extra options on uniq like -c for counts
- More flexible for intermediate steps

**sort -u:**
- A single command with the -u flag on sort
- Does not support counting or showing only duplicates
- Equivalent to sort then uniq but with no extra uniq features

### > (redirection) vs | (pipe)

**> (redirection):**
- Sends output to a file
- Overwrites or appends to a file on disk
- Does not send output to any command

**| (pipe):**
- Sends output to another command as input
- Does not create or modify any file
- Connects commands into a pipeline

### wc -l vs cat -n

**wc -l:**
- Counts the total number of lines in a file or input
- Outputs only a number, not the actual lines
- Used in scripts to check line counts

**cat -n:**
- Numbers every line of a file
- Outputs the entire file with line numbers prepended
- Used for reading a file with line references

## Diagram

_A flowchart showing how common text processing tools filter data from a source file or command output, connected via pipes, leading to a final filtered result._

```mermaid
flowchart TD
    A[Text File or Command Output] --> B[grep: Find matching lines]
    A --> C[cut: Extract columns]
    A --> D[sort: Arrange lines]
    B --> E[Pipe |]
    C --> E
    D --> E
    E --> F[uniq: Remove duplicates]
    E --> G[wc: Count lines/words]
    E --> H[head/tail: Show top/bottom]
    F --> I[Final Filtered Output]
    G --> I
    H --> I
```

## Common misconceptions

- **Misconception:** I can use `uniq` by itself on any file and it will remove all duplicate lines. **Reality:** `uniq` only removes consecutive duplicate lines. You must first sort the file with `sort` to group identical lines together before piping to `uniq`. If duplicates are scattered across the file, `uniq` will miss them. (The name 'uniq' sounds like it makes the entire file unique, but the tool was designed to work on sorted input for efficiency. Beginners assume it works like a global deduplicator.)
- **Misconception:** `grep -v` shows the lines containing the pattern but with some visual indicator. **Reality:** `grep -v` inverts the match — it shows all lines that do NOT contain the pattern. It does not highlight or mark anything. It is the opposite of a regular search. (The letter 'v' is naturally associated with 'visual' or 'view'. Beginners think it shows the matches visually, not realising it stands for 'invert match'.)
- **Misconception:** `cut -f 2` will always extract the second column regardless of the delimiter. **Reality:** `cut -f 2` only works if the delimiter is a tab character by default. For comma-separated or colon-separated files, you must also specify `-d ','` or `-d ':'`. Without `-d`, `cut` assumes tabs. (Many beginners have never worked with tab-separated files. They assume `cut` automatically detects the delimiter, but it does not. The exam frequently tests this exact oversight.)
- **Misconception:** The pipe symbol `|` saves output to a file so I can look at it later. **Reality:** The pipe symbol `|` sends output directly to another command as input. It does not save anything to a file. To save output, you use redirection `>` or `>>`, or the `tee` command which both saves and displays. (The visual of a pipe 'carrying' data away makes beginners think it is being stored somewhere. They confuse piping with file redirection because both move data, but to different destinations.)
- **Misconception:** `sort -n` sorts alphabetically but with numbers treated as characters. **Reality:** `sort -n` sorts by numerical value, not by character encoding. So '10' comes after '2', which is correct numerically. Without `-n`, `sort` treats numbers as text, so '10' comes before '2' because the character '1' is sorted before '2'. (In everyday life, alphabetising lists that include numbers is rare. Beginners expect '10' to come after '2' intuitively, but default `sort` follows ASCII order, placing '10' before '2'. The flag `-n` fixes this counterintuitive behaviour.)

## Key takeaways

- `grep` searches for patterns in text; `grep -i` ignores case, `grep -v` inverts the match, and `grep -w` matches whole words only.
- `cut -d ':' -f 1,3` extracts the first and third fields from a colon-separated file; always specify the delimiter when it is not a tab.
- `sort` must precede `uniq` because `uniq` only removes consecutive duplicates; chain them as `sort file | uniq`.
- `sort -n` sorts numerically (treats 10 as greater than 2) while plain `sort` sorts alphabetically by character order (treats 10 as less than 2).
- The pipe `|` connects commands by sending the output of one command as input to the next; it does not save data to a file.
- `tail -f` displays new lines as they are written to a file, making it indispensable for monitoring live logs during troubleshooting.
- `tee` splits output to both the screen and a file simultaneously, useful for logging while observing results in real time.
- Regular expressions use special characters like `^` (start of line), `$` (end of line), and `.*` (any characters) for advanced pattern matching with `grep`.

## FAQ

**What is the difference between `grep` and `find`?**

`grep` searches inside files for matching text. `find` searches for files and directories by name, type, or date. Use `grep` when you need to find content within files; use `find` when you need to locate files themselves.

**Why does `uniq` not remove all duplicate lines from my file?**

`uniq` only removes consecutive duplicate lines. If duplicate lines are scattered throughout the file, they will not be detected. You must run `sort file | uniq` to group identical lines together first.

**How do I count how many times each error appears in a log file?**

Use `grep 'error' logfile | sort | uniq -c`. The `sort` groups identical error messages, then `uniq -c` counts how many of each error message there are.

**What does `tee` do and when should I use it?**

`tee` reads from standard input and writes to both standard output (your screen) and one or more files. Use it when you want to see command output in real time while simultaneously saving it to a file for later review.

**How do I extract the third column from a CSV file?**

Use `cut -d ',' -f 3 filename.csv`. The `-d ','` flag tells `cut` that fields are separated by commas. Without it, `cut` assumes tab separators and will not work correctly.

**What does `grep -v 'error'` show me?**

It shows all lines that do NOT contain the word 'error'. The `-v` flag inverts the match. This is useful for filtering out noise, like removing debug lines from a log file.

**How do I watch a log file update in real time?**

Use `tail -f /path/to/logfile`. The `-f` flag means 'follow', and the command displays new lines as they are appended to the file. Press Ctrl+C to stop following.

## Check your understanding

1. **A file has lines with colon-separated fields: username:uid:gid:home. Which command extracts the username and home directory?**

   Answer: `cut -d ':' -f 1,4 filename` extracts the first field (username) and the fourth field (home directory) using colon as the delimiter.

2. **What does the command `grep -c 'error' /var/log/syslog` output?**

   Answer: It outputs the number of lines in /var/log/syslog that contain the word 'error'. The `-c` flag stands for count.

3. **You run `uniq file.txt` but duplicates remain. Why?**

   Answer: `uniq` only removes consecutive duplicate lines. You must sort the file first with `sort file.txt | uniq` to group identical lines together.

4. **What is the purpose of the `tee` command?**

   Answer: `tee` reads from standard input and writes simultaneously to standard output (the screen) and to one or more files. It is used to both display and save command output.

5. **Explain the difference between `sort` and `sort -n` when applied to a list containing the numbers 2, 10, and 1.**

   Answer: `sort` sorts alphabetically, producing 1, 10, 2 (because '1' and '10' both start with '1'). `sort -n` sorts numerically, producing 1, 2, 10 (correct numerical order).

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/lf-lfcs/text-processing-and-filtering-tools