# Working with the File System and Directories

> Chapter 10 of the Courseiva PYTHON-PCAP curriculum — https://courseiva.com/learn/python-pcap/working-with-file-systems

**Official objective:** 3.4 — Use os and pathlib modules to navigate, create, and inspect files and directories.

## Introduction

Without knowing how to navigate and inspect files and directories in Python, you'll constantly hit a wall: your script crashes because it can't find a file, or it saves your new data to the wrong place, overwriting something important. For the PCAP-31-03 exam, you need to be comfortable using the os and pathlib modules to travel through folders, create new ones, check what's inside them, and verify whether a path points to a file or a directory.

## The Filing Cabinet Analogy

Your computer's file system works exactly like a giant filing cabinet you might find in an old office. Let's say you have one drawer that's labelled '2024 Projects'. Inside that drawer, you've got hanging folders for 'Client Reports', 'Internal Notes', and 'Budget Spreadsheets'. Inside the 'Client Reports' folder, there are individual manila folders — one for each client. Each manila folder holds actual papers: a letter, a printed spreadsheet, a signed contract.

When you open the top drawer, you're looking at the root of that drawer's universe. To get to the signed contract for 'Acme Corp', you move through a path: Drawer 1 -> Client Reports -> Acme Corp -> Signed Contract. In your computer, that path looks like: C:\2024 Projects\Client Reports\Acme Corp\signed_contract.pdf (on Windows) or /home/you/2024_Projects/Client_Reports/Acme_Corp/signed_contract.pdf (on Mac/Linux).

Now, Python gives you tools — the os module and the pathlib module — that let you do everything a filing clerk can do. The os module is like knowing the exact drawer number and folder label to grab something. The pathlib module is like having a smart assistant who understands that 'the signed contract in the Acme Corp folder' means the same thing, no matter which drawer system you're using. You can create new folders (hanging folders), rename files (replacing a paper with an updated version), check if a file exists (peeking into a folder to see if the contract is still there), and list all the files in a directory (pulling out every paper from a folder to see what you have). Without these tools, you'd have to manually memorise every single location — which is impossible when your filing cabinet has thousands of folders.

## Core explanation

When you write a Python program that needs to read a configuration file, save user data, or process a set of images, you need to tell Python exactly where those files live on your computer's hard drive. This is the file system — the structured way your operating system organises files and folders (which are also called directories).

Think of the file system as a tree. The very top is called the root directory. On Windows, that's something like C:\. On macOS or Linux, it's just /. Underneath root, you have subdirectories (folders), and inside those, you have files. To point to one specific file, you use a path.

A path is simply the address of a file or folder. An absolute path starts from the root and gives the full address, like /home/anna/projects/data.csv. A relative path starts from wherever your Python program is currently running (its current working directory), like projects/data.csv.

Python provides two main modules to work with paths and the file system: os and pathlib. The os module (short for operating system) has been around for a long time. Its os.path submodule contains functions to manipulate paths. For example, os.path.join('home', 'anna', 'projects') will build the correct path string for your operating system. os.path.exists('data.csv') returns True or False. os.path.isfile('data.csv') checks if the path is a file, and os.path.isdir('projects') checks if it's a directory. To get a list of everything in a directory, you use os.listdir('.').

The pathlib module is newer and more object-oriented. Instead of dealing with strings, you create Path objects. For example, Path('data.csv').exists() does the same thing as os.path.exists('data.csv'). The big advantage is that you can use methods directly on the object, and it handles both Windows backslashes and Unix forward slashes automatically.


Here are the key operations you need for the exam:

- Navigating: os.getcwd() tells you your current working directory. os.chdir('/new/path') changes it. With pathlib, Path.cwd() gives you the current directory as a Path object.

- Creating: os.mkdir('new_folder') creates a single new directory. os.makedirs('parent/child/grandchild') creates all directories in that path if they don't exist. With pathlib, you use Path('new_folder').mkdir() and Path('parent/child/grandchild').mkdir(parents=True).

- Inspecting: os.path.exists(path), os.path.isfile(path), os.path.isdir(path). With pathlib, Path(path).exists(), Path(path).is_file(), Path(path).is_dir().

- Listing: os.listdir(path) returns a list of names of files and folders in that directory. pathlib offers Path.iterdir() which gives you an iterator of Path objects.

- Renaming and deleting: os.rename(old, new) renames or moves a file. os.remove(filepath) deletes a file. os.rmdir(dirpath) deletes an empty directory. pathlib equivalents are Path.rename(target), Path.unlink(), and Path.rmdir().

Both modules do similar things, but pathlib is generally preferred in modern Python because it's cleaner and less prone to bugs from string manipulation. However, the exam tests both, so you need to recognise the syntax for each.

Why does all this matter? Because without it, you can't write a script that processes yesterday's log files, checks if a backup exists before overwriting it, or creates a folder structure for each new user. File system operations are the backbone of almost every practical Python program that deals with persistent data.

## Real-world context

Imagine you work for a small marketing agency that runs weekly email campaigns. Every Monday, a script needs to generate personalised HTML email files based on customer data. Here's what an IT professional actually does with file system operations in Python:

First, the script checks if the output directory for this week's campaign exists. If it's week 44 of 2024, the script should save everything to /campaigns/2024/week44/. Using pathlib: if not Path('/campaigns/2024/week44/').exists(): Path('/campaigns/2024/week44/').mkdir(parents=True). This prevents errors when trying to write files into a non-existent folder.

Next, the script reads a CSV file of customer information from a centralised data directory. It uses os.path.join to build the path: data_file = os.path.join(data_dir, 'customers.csv'). This is safe because it respects the operating system's path separator.

Then, for each customer, the script needs to write a personalised HTML file. It creates the file path: output_path = Path(f'/campaigns/2024/week44/{customer_id}.html'). It writes the HTML content to that path. If a customer has previously opted out (a status flag in the CSV), the script should skip that customer and check if an old file for that customer exists to delete it. That's os.path.exists and os.remove in action.

After generating all files, the script needs to do a final sanity check — count how many HTML files were created and compare it to the number of customers processed. It uses os.listdir(output_dir) to get all file names, then filters for .html files using str.endswith() or Path.suffix.

Finally, the script archives the previous week's campaign folder to save space. It renames the folder from /campaigns/2024/week43 to /campaigns/2024/archive/week43. That's os.rename or Path.rename.


In a more complex scenario, the agency might have a shared network drive. The script must use os.path.isdir to check if a mounted network path is available before trying to read files. If the network is down, the script logs an error and exits gracefully instead of crashing with a FileNotFoundError.

Another real-world task: a system administrator writes a Python script to clean up old log files. The script walks through the /var/log directory tree, checks each file's modification time using os.path.getmtime, and deletes any .log file older than 30 days. This uses os.listdir, os.path.join, os.path.isfile, and os.remove in a loop. Without these file system tools, the admin would have to manually log into every server and delete files by hand — a waste of hours every week.

## Exam focus

The PCAP-31-03 exam tests your ability to use both the os and pathlib modules for basic file system operations. Expect around 2-4 questions on this objective. They will not ask you to memorise every single function in both modules, but they will test your understanding of the common ones and the key differences between the two approaches.


Here are the specific concepts they love to test:

- Recognising which function belongs to which module. For example, they might show you code like: import os; os.path.exists('test.txt') and ask what it does. A trap is that os.path.exists is in the os.path submodule, not os.exists(). Similarly, Path('test.txt').exists() belongs to pathlib.

- The difference between absolute and relative paths. They might give you a code snippet that changes the current working directory with os.chdir, then uses a relative path, and ask you to predict the final path.

- Creating directories with mkdir vs makedirs. They love to test that os.mkdir can only create a single directory and fails if parent directories don't exist, while os.makedirs creates intermediate directories too. With pathlib, Path.mkdir needs parents=True to act like makedirs.

- The os.walk function — a powerful tool that recursively goes through every subdirectory. They may ask you to identify what a function like os.walk returns. It yields a tuple of (root_dir, list_of_subdirs, list_of_files).


Common traps and patterns:

- Trap: Forgetting that os.remove() cannot delete a directory (use os.rmdir instead), and os.rmdir cannot delete a non-empty directory. With pathlib, Path.unlink() is for files, Path.rmdir() for empty directories.

- Trap: Path manipulation on different operating systems. os.path.join('folder', 'subfolder') works cross-platform, but manually writing 'folder/' + 'subfolder' does not.

- Trap: The pathlib Path object can be used with forward slashes even on Windows because Python converts them internally. So Path('data/file.txt') is fine.


Key definitions to memorise for the exam:

- Current working directory (cwd): the directory your script is running from.

- Absolute path: starts from the root (e.g., /home/user/file.txt or C:\Users\user\file.txt).

- Relative path: starts from the current working directory (e.g., ../data/file.txt).

- os.path.exists(path): returns True if the path exists.

- os.path.isfile(path) and os.path.isdir(path): check type.

- os.mkdir(path) vs os.makedirs(path): single vs recursive directory creation.

- pathlib Path object: Path(path) creates an object with .exists(), .is_file(), .is_dir(), .mkdir(), .rename(), .unlink(), .rmdir() methods.

Expect multiple-choice questions where you must choose which code snippet correctly creates a directory structure, or which method checks if a path is a file. Some questions will present a scenario and ask you to identify the best module to use — pathlib is almost always the 'modern' answer.

## Step by step

1. **Import the required module** — You need to import the module before using it. For os, write 'import os' or 'from os import path' for path functions. For pathlib, write 'from pathlib import Path'. This gives you access to all the file system functions.
2. **Define a path** — Create a path variable representing the file or directory you want to work with. With os, you typically use a string like 'data_folder/report.csv'. With pathlib, you create a Path object: 'p = Path('data_folder/report.csv')'. This path can be absolute or relative.
3. **Check if the path exists and what it is** — Before trying to read or write, verify the path exists using os.path.exists() or Path.exists(). Then check if it's a file or directory with os.path.isfile()/os.path.isdir() or Path.is_file()/Path.is_dir(). This prevents your script from crashing with a FileNotFoundError.
4. **Create directories if needed** — If your path points to a location that doesn't exist, create it. For a single new directory, use os.mkdir('new_folder') or Path('new_folder').mkdir(). For nested directories like 'parent/child/grandchild', use os.makedirs('parent/child/grandchild') or Path('parent/child/grandchild').mkdir(parents=True).
5. **List or traverse the directory contents** — To see what files and folders are in a directory, use os.listdir('path') which gives a simple list of names. For a recursive traversal of all subdirectories, use os.walk('path') which yields tuples of (root, dirs, files). With pathlib, use Path('path').iterdir() to iterate over entries as Path objects.
6. **Perform file operations (rename, delete, move)** — Once you've found the files you need, you can rename them with os.rename(old, new) or Path.rename(target). Delete a file with os.remove(filepath) or Path.unlink(). Delete an empty directory with os.rmdir(dirpath) or Path.rmdir(). For non-empty directories, use shutil.rmtree('path').

## Comparisons

### os.path functions vs pathlib Path methods

**os.path functions:**
- Takes and returns plain strings for paths
- Older, more verbose syntax
- Requires importing os and os.path submodule

**pathlib Path methods:**
- Uses Path objects with method chaining
- Newer, more Pythonic and readable syntax
- Single import: from pathlib import Path

### os.mkdir() vs os.makedirs()

**os.mkdir():**
- Can only create a single directory
- Fails if parent directories are missing
- Simple use case: os.mkdir('new_folder')

**os.makedirs():**
- Creates all intermediate directories in the path
- Does not fail if parents are missing — it creates them
- Useful for nested paths: os.makedirs('parent/child')

### os.remove() vs os.rmdir()

**os.remove():**
- Deletes a single file only
- Raises an error if used on a directory
- Commonly used for cleaning up individual files

**os.rmdir():**
- Deletes a single empty directory only
- Raises an error if the directory contains files
- Must manually empty directory first or use shutil.rmtree

### Absolute path vs Relative path

**Absolute path:**
- Starts from the root of the file system (e.g., / or C:\)
- Always points to the exact same location regardless of cwd
- Safer for scripts that must run from any directory

**Relative path:**
- Starts from the current working directory (e.g., ./data or data)
- Changes meaning if the cwd changes
- More portable but requires cwd to be correct

## Diagram

_Flowchart showing the main file system operations in Python using os and pathlib modules._

```mermaid
flowchart TD
    A[Python Script] --> B{What do you need?}
    B --> C[Navigate]
    B --> D[Create]
    B --> E[Inspect]
    B --> F[List]
    C --> C1[os.getcwd / os.chdir]
    C --> C2[Path.cwd]
    D --> D1[os.mkdir / os.makedirs]
    D --> D2[Path.mkdir]
    E --> E1[os.path.exists / isfile / isdir]
    E --> E2[Path.exists / is_file / is_dir]
    F --> F1[os.listdir / os.walk]
    F --> F2[Path.iterdir]
```

## Common misconceptions

- **Misconception:** The os module and pathlib module are completely interchangeable and do exactly the same things in the same way. **Reality:** While both modules handle file system operations, os uses string-based paths and function calls, while pathlib uses object-oriented Path objects. pathlib is generally cleaner and more Pythonic, but os is still widely used in older codebases. (Beginners see that both modules can check if a file exists and assume they're identical, not realising the different programming styles and subtleties in how they handle paths.)
- **Misconception:** os.remove() can delete both files and directories. **Reality:** os.remove() only deletes files. To delete a directory (an empty one), you must use os.rmdir(). If the directory contains files, you need shutil.rmtree() or manually empty it first. (The word 'remove' sounds general, so beginners assume it works for anything. The operating system treats files and directories differently, and Python reflects that.)
- **Misconception:** You can use os.mkdir() to create nested directories like os.makedirs(). **Reality:** os.mkdir() only creates a single directory and raises a FileNotFoundError if any parent in the path doesn't exist. os.makedirs() creates all intermediate directories. In pathlib, you need to pass parents=True to Path.mkdir() to achieve the same effect. (Beginner tutorials often show mkdir for a single folder. When they try to create a nested path, they expect it to 'just work', not realising the distinction.)
- **Misconception:** Paths in Python are always strings, so you can just concatenate them like 'folder/' + 'file.txt'. **Reality:** While that works on some systems, it's not cross-platform. Windows uses backslashes, so you'd get 'folder\file.txt'. Always use os.path.join() or pathlib Path / operator for safe path building. (Most beginners start on Windows or Mac, and simple concatenation seems to work. When they share code or work on a different OS, it breaks. They don't know about path separators.)
- **Misconception:** The current working directory never changes within a running Python script unless you explicitly change it. **Reality:** That's true, but many beginners forget that operations like file open() with a relative path depend on the cwd. If another part of the code calls os.chdir(), earlier assumptions about relative paths become invalid. Also, the cwd might be different when running from an IDE vs the terminal. (They treat the cwd as a fixed constant, not a variable that can change during execution. They don't debug by printing os.getcwd() to confirm where files are being read from.)

## Key takeaways

- The os module provides functions like os.path.exists(), os.path.isfile(), and os.mkdir() for string-based file system operations.
- The pathlib module provides object-oriented Path objects with methods like Path.exists(), Path.is_file(), and Path.mkdir(parents=True).
- os.makedirs() creates all intermediate directories in a path; os.mkdir() creates only the final directory and fails if parents are missing.
- Use os.path.join() or the pathlib Path / operator to build cross-platform paths safely instead of string concatenation.
- os.listdir() returns a list of filenames in a directory; os.walk() recursively traverses the entire directory tree.
- A relative path depends on the current working directory (os.getcwd()), which can change during script execution.
- os.remove() deletes files only; use os.rmdir() for empty directories or shutil.rmtree() for non-empty ones.

## FAQ

**What is the difference between os.path.exists and os.path.isfile?**

os.path.exists returns True if the path exists as either a file or a directory. os.path.isfile returns True only if the path exists and is a regular file (not a directory). So if a path points to a folder, exists returns True but isfile returns False.

**Why does os.mkdir fail with 'No such file or directory' when I try to create a nested folder?**

os.mkdir only creates a single directory. If any parent directory in the path doesn't exist, it fails. Use os.makedirs instead, which creates all missing intermediate directories automatically.

**Should I use os or pathlib in my Python project?**

pathlib is generally preferred in modern Python code (3.6+) because it's cleaner, more Pythonic, and automatically handles cross-platform path separators. However, the PCAP exam tests both, and you may encounter os in older codebases.

**How do I get the current working directory in Python?**

Use os.getcwd() (get current working directory) to return a string. With pathlib, use Path.cwd() to get a Path object representing the current directory.

**What does os.path.join do and why should I use it?**

os.path.join takes multiple path segments and joins them into a single path using the correct separator for your operating system. For example, os.path.join('folder', 'subfolder', 'file.txt') returns 'folder/subfolder/file.txt' on Linux/Mac and 'folder\\subfolder\\file.txt' on Windows. This prevents bugs when your code runs on different systems.

**How do I delete a directory that still has files inside it?**

You cannot delete a non-empty directory with os.rmdir (it only works on empty directories). Use shutil.rmtree('path_to_directory') which recursively deletes the directory and all its contents. Be very careful with this command — it permanently deletes everything with no undo.

## Check your understanding

1. **Which os function would you use to create the directory /home/user/projects/data/reports if none of those directories exist yet?**

   Answer: os.makedirs('/home/user/projects/data/reports') because it creates all missing intermediate directories, unlike os.mkdir which would fail.

2. **What is the difference between os.path.isfile(path) and os.path.isdir(path)?**

   Answer: os.path.isfile returns True if the path points to a regular file; os.path.isdir returns True if it points to a directory. They are mutually exclusive for any given path.

3. **Using pathlib, how do you check if 'config.yaml' exists in the current directory?**

   Answer: Path('config.yaml').exists() returns True if the file exists, False otherwise.

4. **Write the pathlib code to create a nested directory structure 'logs/2024/october' without raising an error if intermediate directories are missing.**

   Answer: Path('logs/2024/october').mkdir(parents=True, exist_ok=True) creates all directories in the path and does not raise an error if they already exist.

5. **What does os.listdir('/home/user/docs') return?**

   Answer: It returns a list of strings containing the names of all files and subdirectories inside '/home/user/docs', but not the full paths.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/python-pcap/working-with-file-systems