Auditing and Governance with AWS CloudTrail and Config

Let's start with the core problem. You have an AWS account. People and services inside that account are constantly creating, modifying, and deleting resources like virtual servers (EC2 instances), databases, and storage buckets. As a DevOps engineer, you are responsible for the security and compliance of this environment. But you cannot watch every action live. You need a system that records everything and checks everything automatically. This is where CloudTrail and Config come in.

CloudTrail is AWS's service for auditing. Think of it as a security camera that records every single API call made to your AWS account. An API call is just a command issued to AWS to do something — like 'launch an EC2 instance', 'create an S3 bucket', or 'change a security group rule'. CloudTrail captures who made the call, when it was made, what the call was, where it came from (the IP address), and what the result was. This record is stored as a 'trail', which is a log file that can be saved to an S3 bucket (a storage container) for long-term storage and analysis.

Every action is recorded: actions via the AWS Management Console (the web interface), via the AWS CLI (command line interface), via SDKs (software development kits), and via other AWS services acting on your behalf. This means you have an immutable, tamper-proof record of everything. If a developer accidentally deletes a production database, CloudTrail will show exactly who issued the delete command and when. It cannot prevent the action, but it provides the evidence for incident response.

CloudTrail has two main modes: management events and data events. Management events are actions on resources like launching an EC2 instance or creating a VPC. These are recorded by default and are free. Data events are actions inside those resources, like reading an object from an S3 bucket or invoking a Lambda function. These are more costly and are not enabled by default. For the DOP-C02 exam, you must know the difference: management events are always on, data events must be specifically enabled. Multi-region trails are another key concept. A multi-region trail records events from all AWS regions into a single S3 bucket, giving you a single pane of glass for your entire global infrastructure.

AWS Config is a different service with a different purpose. While CloudTrail records actions (the 'who did what'), Config records the state of your resources (the 'what's the current configuration'). Config is a continuous compliance monitoring and configuration tracking service. It takes a snapshot of your resources at regular intervals and compares them to the rules you define. These rules are called 'Config rules'. You can use managed rules provided by AWS (like 'ec2-instances-against-security-group' or 's3-bucket-public-read-prohibited') or create your own custom rules using AWS Lambda (a serverless function).

For example, you can create a Config rule that says 'all S3 buckets must be private'. Config will then evaluate all your S3 buckets every time they change, or at a specified frequency, and report each bucket as 'compliant' or 'non-compliant'. If someone later makes an S3 bucket public, Config will detect that change and flag it as non-compliant. Config can also trigger automatic remediation actions, like making the bucket private again or sending a notification to a security team. Config keeps a historical record of your resource configurations. You can see what an EC2's security group looked like last week compared to today. This is extremely useful for troubleshooting and compliance auditing.

The relationship between CloudTrail and Config is complementary. When Config detects a non-compliant resource, you can use CloudTrail to find out who made the change that caused the non-compliance. CloudTrail gives you the 'who, what, when', and Config gives you the 'what's the state now' and 'what was the state then'. Together, they provide a complete picture of your account's security and operational health.

GuardDuty is the third service in this triad. GuardDuty is a threat detection service. It uses machine learning and integrated threat intelligence to monitor your AWS environment for suspicious activity. It analyses CloudTrail logs, VPC Flow Logs (network traffic logs), and DNS logs to identify patterns like unusual API calls, crypto mining activity, or compromised EC2 instances communicating with known malicious IP addresses. GuardDuty is not covered in as much depth as CloudTrail and Config in this chapter, but it is a critical part of the auditing and governance puzzle. It is the 'guard dog' that barks when something looks wrong, based on the data collected by CloudTrail.

For the exam, you need to understand the security benefits of each service. CloudTrail provides accountability and forensic analysis. Config provides compliance and configuration history. GuardDuty provides intelligent threat detection. They are often used together in a 'detect and respond' architecture. You also need to know about CloudTrail log file integrity validation, which is a feature that uses digital signatures to ensure the log files have not been tampered with after they were created. Config aggregators allow you to view compliance data from multiple accounts and regions in a single place, which is critical for organisations managing many AWS accounts.

Imagine you work as a DevOps engineer for a company called 'FinTech Corp', which handles financial data. The company has strict compliance requirements from auditors who want proof that no unauthorised person can access customer data. You have two AWS accounts: one for production and one for development.

Step one: You enable CloudTrail for both accounts, and crucially, you create a multi-region trail. This records every management API call from every region into a single S3 bucket named 'company-cloudtrail-logs'. You also enable log file integrity validation on this bucket, so auditors can verify the logs have not been altered. You then set up a CloudTrail dashboard in CloudWatch Logs (a monitoring service) to alert the security team whenever someone deletes a CloudTrail trail. An attacker might try to delete the trail to hide their actions.

Step two: You enable AWS Config in both accounts. You start by applying AWS managed rules: 's3-bucket-public-read-prohibited', 'ec2-instances-in-vpc', and 'iam-user-mfa-enabled'. These run automatically and check your resources every 24 hours. You also create a custom rule to check that all EC2 instances have a specific security group tag, because the finance team needs to track which application each instance belongs to.

Three months later, an auditor asks for a report of all security group changes over the last month. You go into AWS Config's timeline view for security groups and see exactly when each rule was added or removed. For each change, you click a button in the Config console that says 'Related CloudTrail events', which shows you exactly which IAM user made the change. The auditor is satisfied.

Later, an engineer accidentally makes an S3 bucket public. Within minutes, Config detects the change and marks it as non-compliant. An automated remediation action (a Lambda function) is triggered, making the bucket private again. CloudTrail then logs the original action (the engineer making it public) and the remediation action (the Lambda function making it private). You use CloudTrail to find the engineer and provide retraining. Without CloudTrail and Config, you would have no idea who made the change or that the bucket was ever public.

Or consider a security incident. GuardDuty detects unusual traffic from an EC2 instance to a known malicious IP address. It generates a finding. You go to CloudTrail to see what API calls that instance's role has made recently. You find that it has been making calls to create IAM users — a clear sign of compromise. You then use CloudTrail to trace back to the source of the compromise: the instance was created by a developer who left two years ago, and their access keys were still active. You can then disable those keys and patch the security hole.