# Data Governance, Logging, and Compliance with AWS CloudTrail and Config

> Chapter 12 of the Courseiva AWS-DATA-ENGINEER-ASSOCIATE curriculum — https://courseiva.com/learn/aws-data-engineer-associate/data-governance-compliance

**Official objective:** 3.3 — Implement data governance and audit controls

## Introduction

How do you prove to an auditor that no one secretly accessed your customer database six months ago? This is the core problem that data governance, logging, and compliance solve for anyone preparing for the DEA-C01 exam. Without these controls, your organisation cannot know who did what, when, or whether your cloud setup follows security rules — which means you fail audits and risk data breaches.

## The Flat Share Security Camera and Rulebook Analogy

You move into a shared flat with three other people. First, everyone agrees on one basic rule: the front door must always be locked, and you need a key to get in. This rule is your governance policy. Because you all own valuable things, you decide to install a small camera in the hallway that records every time someone enters or leaves. This is your CloudTrail — it logs every action at the front door (your AWS account), without recording what happens inside anyone's private room. The video footage is kept for ninety days in case you need to check who came home at 3am on Tuesday, which is your logging retention.

Now, the second problem: you need to make sure everyone actually follows the shared rules, like no leaving rubbish in the hall or propping the fire door open. For that, you hire a quiet inspector who checks the hallway every hour. The inspector has a list of rules — for example, 'the fire door must close automatically'. If the inspector finds the door wedged open, he writes a report and sends an alert to the whole flat. This is your AWS Config: it continuously checks your cloud resources against a set of rules and tells you when something is out of compliance. Without the camera and the inspector, you would have no idea if someone broke the rules or if someone came in who shouldn't have. Data governance, logging, and compliance are exactly that combination: a camera that records everything and an inspector who checks the rules, giving you confidence that your cloud environment is secure and well-behaved.

## Core explanation

When you build applications in Amazon Web Services (AWS), you are essentially renting computers, storage, and databases that live in Amazon's data centres. You do not have physical access to these machines, so you need a different way to see what is happening inside your account. This is where logging and monitoring tools come in. Two of the most important services for data governance are AWS CloudTrail and AWS Config. They solve two different but related problems: who did what, and is everything set up correctly?

AWS CloudTrail is the service that records every action taken in your AWS account. Imagine it as a CCTV camera for your cloud. Every time someone uses the AWS Management Console (the web interface), the Command Line Interface (a text-based tool), or an API call (a programmatic request), CloudTrail captures that event. It records who made the request, what time it happened, what service was accessed, and what the response was. For example, if a junior developer deletes an Amazon S3 bucket (a storage container) by mistake, CloudTrail will have a log entry showing the developer's user name, the exact time of deletion, and the IP address they used. This is essential for security investigations and for proving compliance with regulations like GDPR or HIPAA.

CloudTrail can deliver these logs to an Amazon S3 bucket for long-term storage. By default, logs are kept for ninety days in the CloudTrail event history, but you can configure a trail to send them to S3 indefinitely. You can also send them to Amazon CloudWatch Logs for real-time monitoring and alerting. A key concept is that CloudTrail is recording management events by default — actions like creating or deleting resources. You can also enable data events, which record actions within a resource, like reading an object inside an S3 bucket or invoking a Lambda function. Data events are more detailed but generate a much higher volume of logs, which costs more money.

AWS Config is a different service that focuses on the state of your resources. It is not about who did what, but about what your resources look like right now, and whether they match the rules you have defined. Think of Config as a compliance inspector. You define rules — for example, 'all S3 buckets must be encrypted', or 'all security groups must not allow SSH access from 0.0.0.0/0' (which means anyone on the internet can connect). AWS Config continuously checks your resources against these rules. If a bucket is created without encryption, Config flags it as non-compliant, and you can set up automatic remediation, like forcing encryption to be applied.

The two services work together beautifully. CloudTrail tells you who turned off encryption on a bucket at 2pm on Tuesday. Config tells you that the bucket is now non-compliant because encryption is off. Together, they give you both the history of changes and the current state of compliance. This is the foundation of data governance: knowing that your data is stored correctly, that only authorised people can access it, and that you can prove this to auditors.

Before these services existed, organisations had to build their own logging and compliance checking systems. They would write custom scripts that ran every hour to check resource configurations, and they had to rely on application logs to understand who made changes. This was slow, error-prone, and did not scale. With CloudTrail and Config, AWS does most of the heavy lifting, and you just need to configure the services correctly and react to the alerts.

A common beginner mistake is confusing the two services. Many people think CloudTrail checks compliance and Config records events, but it is the opposite. CloudTrail is the historian, Config is the inspector. Memorise that difference and you will avoid many exam traps.

## Real-world context

Let us walk through a realistic scenario at a company called MediData, which handles patient health records and must comply with HIPAA (a strict US healthcare privacy law). MediData uses AWS to store its data. The company's compliance officer needs to answer a question from an auditor: 'Can you prove that no unauthorised person accessed the patient database in the last quarter?'

First, the cloud architect ensures that CloudTrail is enabled across all AWS regions. This creates a trail that logs every management event — every time someone creates, modifies, or deletes a resource. The logs are delivered to a dedicated S3 bucket that has its own access controls and encryption. The bucket is set to keep logs for seven years, as required by HIPAA. The architect also enables data events on the S3 bucket containing the patient database, so every time a file is read or downloaded, it is recorded.

Next, the architect sets up AWS Config with a set of managed rules. One rule checks that all S3 buckets have server-side encryption enabled. Another rule checks that the patient database is not publicly accessible. A third rule verifies that all security groups only allow traffic on port 443 (HTTPS). Every time a resource changes, Config evaluates it against these rules within minutes.

When the auditor asks for proof, the compliance officer goes to CloudTrail and runs a query for the past ninety days. She filters by the name of the patient database and sees every single access attempt, including the user, time, and IP address. There is one anomaly: a user called 'dev-alice' accessed the database at 3am on a Saturday. The officer checks with Alice, who forgot she was running a backup script. The officer documents this in the audit report. This satisfies the auditor.

Simultaneously, Config reports that for the entire quarter, the patient database bucket was always compliant with the encryption rule and the public access rule. The auditor can see a compliance score and a timeline of changes. This means MediData passes the audit without needing to manually collect evidence from dozens of different systems.

In a real IT job, a data engineer does not just set these services up and forget them. They:

- Configure CloudTrail trails across all regions and accounts using AWS Organizations
- Set up S3 lifecycle policies to move old logs to cheaper storage (like Glacier) after ninety days
- Create CloudWatch alarms that trigger when Config finds a non-compliant resource
- Write AWS Lambda functions that automatically fix common compliance violations, like removing public access from a bucket
- Use AWS Control Tower to enforce governance rules across multiple accounts automatically
- Regularly review CloudTrail logs using Amazon Athena (a query service) to look for suspicious patterns

This is the daily life of ensuring data governance in the cloud: knowing what happened, checking that rules are followed, and being able to prove both to anyone who asks.

## Exam focus

The DEA-C01 exam tests your understanding of CloudTrail and Config in several specific ways. First, know the difference between CloudTrail and Config inside out. The exam loves to present a scenario and ask which service would solve the problem. If the question says 'who deleted a resource', the answer is CloudTrail. If it says 'check if a resource meets a security standard', the answer is Config. They will also test you on what CloudTrail records by default (management events, not data events) and how to enable data events (which cost extra).

Expect questions about CloudTrail log file integrity. CloudTrail can sign its log files using a digest file, which lets you verify that the logs have not been tampered with. The exam asks what mechanism does this — the answer is SHA-256 hashing with a digital signature. Another common topic is CloudTrail vs. CloudWatch. CloudTrail records API activity; CloudWatch records application performance metrics like CPU usage and also stores log files from applications. Do not mix them up.

For AWS Config, the exam tests your understanding of Config rules. There are managed rules (pre-built by AWS) and custom rules (written by you using AWS Lambda). The exam will ask when to use each one. They also test Config's ability to evaluate resources on a schedule or when a configuration change happens. A common trap is that Config does not prevent changes — it only detects and reports them. Remediation can be automated, but that is an optional extra step using AWS Systems Manager.

Key concepts that appear on the exam:
- CloudTrail trails can be applied to a single region or all regions
- You can aggregate logs from multiple accounts using an S3 bucket in a central account
- CloudTrail Insights helps detect unusual activity by analysing normal patterns and alerting on deviations
- Config supports multi-account, multi-region data aggregation using an aggregator
- Both services integrate with AWS Organizations for centralised management
- Config's configuration recorder must be turned on before it can track resources

The exam will also test your knowledge of compliance frameworks. You might be asked how to prepare for HIPAA, SOC 2, or PCI DSS audits. The answer usually involves enabling CloudTrail, Config, and using AWS Artifact (a service that provides compliance reports). Remember that CloudTrail logging is a requirement for many compliance certifications.

Another trap: the difference between CloudTrail event history (which shows the last ninety days) and CloudTrail trails (which deliver logs to S3 for long-term storage). If an auditor asks for logs from six months ago, you need a trail that was set up before that time. The event history alone will not be enough.

Finally, expect questions about log encryption. CloudTrail can encrypt logs using SSE-S3 (server-side encryption with Amazon S3) or SSE-KMS (encryption with AWS Key Management Service). Config can also encrypt its results. The exam will ask which encryption method provides the most control — the answer is SSE-KMS because you manage the keys. Know these details and you will pass the governance questions with confidence.

## Step by step

1. **Enable CloudTrail across all regions** — Create a new CloudTrail trail and choose 'Apply to all regions'. This ensures every API call in every AWS region is recorded. Without this step, if someone creates a resource in a region where CloudTrail is not enabled, that action is lost forever.
2. **Configure the S3 bucket for log storage** — CloudTrail will deliver logs to an S3 bucket you specify. Enable bucket encryption (using SSE-S3 or SSE-KMS) and enable log file validation. Set a lifecycle policy to move old logs to Glacier storage after 90 days to save money, or delete them after your compliance retention period.
3. **Enable data events for critical resources** — Decide which resources need data-level logging, such as S3 buckets containing sensitive data. Go to the CloudTrail trail settings and select 'Data events', then choose the specific S3 buckets or Lambda functions. Data events generate more logs, so only enable them for resources that require audit-level detail.
4. **Set up AWS Config with managed rules** — Turn on AWS Config and choose a set of managed rules relevant to your compliance needs, such as 'S3 bucket server-side encryption enabled' and 'EC2 instances should not use public IPs'. Config will start evaluating your existing and new resources against these rules automatically.
5. **Create an SNS topic for compliance alerts** — When Config finds a non-compliant resource, it can send a notification to an SNS (Simple Notification Service) topic. Subscribe your team's email addresses or a Slack channel to this topic. This turns compliance checking from a manual chore into an automated alerting system.
6. **Review CloudTrail logs using Amazon Athena** — Create an Athena table that maps to your CloudTrail S3 bucket. This lets you run SQL queries against your logs — for example, 'SELECT user, eventtime FROM cloudtrail_logs WHERE eventname = 'DeleteBucket'.' This is how professionals analyse millions of log entries without downloading them.

## Comparisons

### AWS CloudTrail vs AWS Config

**AWS CloudTrail:**
- Records discrete events (API calls) as they happen
- Focuses on who did what and when
- Primary use: auditing and security investigation

**AWS Config:**
- Records the state of resources at a point in time
- Focuses on what resources look like and if they meet rules
- Primary use: compliance checking and resource inventory

### CloudTrail Event History vs CloudTrail Trail (S3)

**CloudTrail Event History:**
- Free, kept for 90 days only
- Cannot customise which events are logged
- Good for quick lookups of recent activity

**CloudTrail Trail (S3):**
- Paid (S3 storage costs), kept indefinitely
- Can filter by data events and management events
- Good for long-term retention and compliance audits

### CloudTrail vs CloudWatch Logs

**CloudTrail:**
- Logs AWS API calls from users and services
- Cannot store application log files
- Enabled at the account level, not the application level

**CloudWatch Logs:**
- Logs application output (stdout, stderr, custom log files)
- Can store any text-based log data from EC2 or Lambda
- Enabled at the instance or function level

### Managed Config Rule vs Custom Config Rule

**Managed Config Rule:**
- Pre-built by AWS, no coding required
- Covers common compliance checks like encryption or public access
- Limited to the checks AWS provides

**Custom Config Rule:**
- You write the logic in AWS Lambda
- Covers any compliance check you can imagine
- Requires development effort and Lambda runtime costs

### SSE-S3 Encryption vs SSE-KMS Encryption

**SSE-S3 Encryption:**
- AWS manages the encryption keys entirely
- No customer control over key rotation or access
- Slightly lower cost, simpler setup

**SSE-KMS Encryption:**
- You manage the keys using AWS KMS
- Full control over who can decrypt the logs
- Required for certain compliance standards like PCI DSS

## Diagram

_How CloudTrail records user actions and Config evaluates resource compliance, with logs going to S3 for analysis and alerts going to the team._

```mermaid
flowchart TD
    U[User] -->|Makes API call| CT[AWS CloudTrail]
    U -->|Creates resource| R[AWS Resource]
    R -->|Configuration change| CG[AWS Config]
    CT -->|Logs event| S3[S3 Bucket]
    S3 -->|Delivered to| AT[Amazon Athena for queries]
    CG -->|Evaluates against rules| CR[Config Rule]
    CR -->|Non-compliant?| SNS[SNS Notification]
    SNS -->|Alert| TE[Team Email]
    CG -->|Compliance snapshot| D Compliance Dashboard
```

## Common misconceptions

- **Misconception:** CloudTrail logs everything by default, including data events like reading files from S3. **Reality:** CloudTrail only logs management events by default. Data events must be explicitly enabled and incur additional charges. (Beginners assume 'record everything' means exactly that, but AWS charges based on volume, so they turn off data events by default to control costs.)
- **Misconception:** AWS Config prevents you from creating non-compliant resources. **Reality:** AWS Config only evaluates and reports compliance. It does not block resource creation unless you add automatic remediation with AWS Lambda or Systems Manager. (The name 'Config rule' sounds like a policy that enforces behaviour, but it is really just an auditing and detection tool, not a prevention tool.)
- **Misconception:** Once you enable CloudTrail, logs are stored forever for free. **Reality:** CloudTrail stores logs in S3, and you pay for S3 storage. The default event history only keeps logs for ninety days at no extra charge. (People confuse the free default event history with the paid trail that sends logs to S3, and they forget that storing data long-term costs money.)
- **Misconception:** CloudTrail and CloudWatch Logs are the same service and do the same thing. **Reality:** CloudTrail records API activity (who did what). CloudWatch Logs stores application and system logs (what an application outputs). They serve different purposes. (Both involve logging, and both can send logs to S3, so beginners group them together, but the source of the logs is completely different.)
- **Misconception:** You only need CloudTrail in one region to cover your entire AWS account. **Reality:** CloudTrail is regional. To get logs for all regions, you must create a trail that applies to all regions, or create separate trails per region. (AWS resources are regional, and beginners often think an AWS account is a single global space, not understanding that each region is isolated.)

## Key takeaways

- AWS CloudTrail records every API call in your account, functioning as a complete audit log of who did what and when.
- AWS Config continuously evaluates your resource configurations against rules and reports which resources are compliant or non-compliant.
- CloudTrail records management events by default; you must explicitly enable data events for detailed logging inside resources like S3 buckets.
- AWS Config does not prevent non-compliant changes; it only detects and reports them, and remediation requires separate automation.
- CloudTrail logs are stored in S3 for long-term retention, while the free event history only covers the last 90 days.
- For compliance audits like HIPAA, you must enable CloudTrail in all regions and use AWS Config to demonstrate continuous compliance.
- You can verify CloudTrail log file integrity using digest files that contain SHA-256 hashes and digital signatures.
- Both CloudTrail and Config can be centrally managed across multiple AWS accounts using AWS Organizations.

## FAQ

**Does CloudTrail record everything I do in the AWS Console automatically?**

Yes, for management events. Every time you click a button in the Console, CloudTrail records it. Data events, like downloading a file from S3, are not recorded unless you enable them.

**How long are CloudTrail logs kept by default?**

CloudTrail event history is kept for 90 days at no extra cost. To keep logs longer, you must create a trail that delivers logs to an S3 bucket, and you pay for S3 storage.

**Can AWS Config automatically fix a non-compliant resource?**

Config itself does not fix anything, but you can create automatic remediation using AWS Systems Manager Automation documents or AWS Lambda functions that run when Config detects a violation.

**What is the difference between CloudTrail and CloudWatch Logs?**

CloudTrail records AWS API activity (who created a resource). CloudWatch Logs stores logs from your applications (for example, web server logs or database error logs). They collect different types of data.

**Do I need CloudTrail in every AWS region?**

Yes, if you want complete coverage. You can create one trail that applies to all regions, which saves you from setting up a trail in each region individually.

**Is AWS Config free to use?**

No. AWS Config charges per rule per region, and also charges for configuration item records. The first 100,000 configuration items per account per region are free, but beyond that there are costs.

## Check your understanding

1. **Which AWS service records every API call made in your account and is used for security auditing?**

   Answer: AWS CloudTrail. It logs all management events by default and can be configured to log data events.

2. **Which service would you use to check if all S3 buckets in your account have encryption enabled?**

   Answer: AWS Config. You would create a Config rule that checks for the 's3-bucket-server-side-encryption-enabled' managed rule.

3. **How long is the CloudTrail event history available for free?**

   Answer: 90 days. After that, you must use a CloudTrail trail that delivers logs to an S3 bucket for long-term storage.

4. **What is the difference between a CloudTrail management event and a data event?**

   Answer: Management events record actions on AWS resources (like creating or deleting an S3 bucket). Data events record actions within a resource (like reading an object in an S3 bucket).

5. **Can AWS Config automatically prevent someone from creating a non-compliant resource?**

   Answer: No. Config only detects and reports non-compliance. To prevent creation, you would need to use AWS Service Control Policies (SCPs) or a custom solution.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/aws-data-engineer-associate/data-governance-compliance