Text Processing and Filtering Tools

Text processing tools are small, focused commands that manipulate text in a terminal. A terminal is a text-based interface where you type commands and see results. Instead of using a mouse or clicking buttons, you type instructions. These tools were created in the early days of Unix, when computers had no graphics, only text terminals. They survive because they are fast, scriptable, and powerful.

Let us start with the basic tools you will see on the exam.

cat (short for concatenate) prints the entire content of a file to the screen. If you run cat /var/log/syslog, you see every line of the system log. This is the simplest tool, but for large files it is overwhelming.

grep (Global Regular Expression Print) searches for lines that match a pattern. For example, grep 'error' /var/log/syslog shows only lines containing the word 'error'. A pattern can be a plain word, a phrase with spaces in quotes, or a regular expression — a special syntax for matching complex patterns like phone numbers or email addresses. The basic usage is grep [options] pattern [file...]. Common options include -i to ignore case (so 'Error' matches 'error'), -v to show lines that do NOT match, -c to count matching lines, and -r to search recursively through directories.

cut extracts columns or fields from each line of a file. Think of a spreadsheet-like table where columns are separated by a delimiter — often a comma in CSV (Comma-Separated Values) files or a colon in system files like /etc/passwd. The command cut -d ':' -f 1,3 /etc/passwd says: use colon as delimiter (-d ':'), and print the first and third fields (-f 1,3). This would show you usernames and their user IDs.

sort arranges lines in a specified order. By default it sorts alphabetically. sort -n sorts numerically. sort -r reverses the order. sort -k 2 sorts by the second field (column). This is useful before piping output to uniq, because uniq only detects adjacent duplicates — so you must sort first to group identical lines together.

uniq (short for unique) removes consecutive duplicate lines. It is almost always used after sort. The command sort file.txt | uniq gives you a list with each line appearing only once. uniq -c prepends a count of how many times each line appeared. uniq -d shows only the lines that are duplicated.

wc stands for word count. wc -l counts lines, wc -w counts words, wc -c counts bytes (roughly characters). Common usage: wc -l file.txt tells you how many lines are in the file.

head and tail show the beginning or end of a file. head -n 5 shows the first 5 lines. tail -n 10 shows the last 10 lines. tail -f follows a file in real time, displaying new lines as they are written — this is the standard way to watch log files live during troubleshooting.

tr translates or deletes characters. tr 'a-z' 'A-Z' converts all lowercase letters to uppercase. tr -d ' ' deletes all spaces. It reads from standard input (keyboard or pipe) and writes to standard output.

tee reads from standard input and writes to both standard output and a file. It is like a T-pipe in plumbing: the data goes to two places at once. echo 'hello' | tee file.txt prints 'hello' on the screen and saves it to file.txt.

Now, the real power comes from combining these tools with pipes. A pipe, represented by the vertical bar |, sends the output of one command as input to the next command. For example:

cat /var/log/syslog | grep 'error' | grep -v 'database' | wc -l

This reads the syslog file, keeps only lines containing 'error', removes lines that also mention 'database', and then counts how many lines remain. This is a pipeline.

Another essential concept is regular expressions (regex). A regular expression is a pattern that describes a set of strings. For example, the regex ^[A-Z].* matches any line that starts with a capital letter. The caret ^ anchors to the start of a line, [A-Z] matches any single uppercase letter, and .* matches any characters after it. grep uses regex by default, though you can use -E for extended regex which supports more patterns like + (one or more) and ? (zero or one).

These tools replace manual searching through files with a text editor. Imagine a 500,000-line log file from a web server. Manually scrolling to find requests that returned a 404 error is impossible. With grep '404' access.log, the answer appears instantly. This is why sysadmins cherish the command line.

Imagine you are the sole sysadmin at a mid-sized e-commerce company. It is 2 AM. A monitoring alert says the website is returning 500 Internal Server Errors for 1% of users. The developer team is asleep. You must pinpoint the cause from log files.

First, you connect to the server via SSH (Secure Shell — an encrypted remote connection). You navigate to the application log directory: cd /var/log/myapp. The application writes logs to a file called app.log. This file is rotated every hour, so there is app.log.1, app.log.2.gz (compressed), etc.

You start with tail -n 200 app.log to see the most recent entries. You spot lines with '500' but they are mixed with '200' (success) and '404' (not found). You need to isolate only the errors.

You run grep '500' app.log and get 1,200 lines. Too many to read. You refine: grep '500' app.log | grep -v 'healthcheck' — this removes lines from automated health checks that always trigger harmless 500s. Down to 800 lines.

You suspect a specific API endpoint /checkout is failing. You search: grep '/checkout' app.log | grep '500'. Now 150 lines. You want to see the timestamps and error messages:

grep '/checkout' app.log | grep '500' | cut -d ' ' -f 1,4,8-

Assuming the log format is: timestamp, level, endpoint, status, message. This shows you the first field (timestamp), the fourth field (endpoint), and the eighth field onwards (message). The cut command here is splitting each line by spaces (-d ' ').

You notice all errors happen when a parameter user_id equals 'guest'. You hypothesise the checkout logic fails for guest users. You check if this is a pattern: grep '/checkout' app.log | grep '500' | grep 'guest' | wc -l returns 148, meaning 148 of the 150 errors involve guest users. You have your culprit.

You save the evidence: grep '/checkout' app.log | grep '500' | grep 'guest' > /tmp/error_evidence.txt. The > symbol redirects output to a file. For a permanent record, you might also use tee: grep '/checkout' app.log | grep '500' | grep 'guest' | tee /tmp/evidence.log | wc -l. This saves the matching lines to a file and prints the count to your screen.

Now you need to check if any other endpoints have similar patterns. You run grep '500' app.log | cut -d ' ' -f 4 | sort | uniq -c | sort -rn. This pipeline: finds all 500s, extracts the endpoint field, sorts the endpoints alphabetically, counts unique occurrences with uniq -c, then sorts the results in reverse numeric order (-rn). The output shows you which endpoint has the most 500 errors, ranked from highest to lowest.

You see /checkout at the top with 150 errors, then /login with 12 errors, then /home with 2. The pattern is clear. You write a summary in a ticket for the developers, attach the evidence file, and leave a note: 'Guest user checkout logic appears broken. Priority: high. Log snippet attached.'

This workflow saved you 40 minutes of manual scrolling. The tools let you filter millions of lines in seconds.