AWS CloudWatch and CloudTrail

Amazon CloudWatch is a monitoring and observability service that provides data and actionable insights for AWS resources, applications, and on-premises servers. It answers questions like: Is my EC2 instance CPU utilization too high? Is my application responding slowly? Are my RDS connections maxing out? CloudWatch collects metrics (time-ordered data points), logs, and events, and allows you to set alarms and automate responses.

AWS CloudTrail is a governance, compliance, and audit service that records every API call made in your AWS account. It answers questions like: Who launched an EC2 instance last night? Did someone delete an S3 bucket? Which IAM user made a change to a security group? CloudTrail provides a history of AWS API calls for your account, including calls made via the AWS Management Console, SDKs, CLI, and AWS services.

CloudWatch operates on three core concepts: metrics, alarms, and logs.

Metrics are the fundamental building blocks. A metric is a time-ordered set of data points representing a variable you want to monitor. For example, the CPUUtilization metric for an EC2 instance is published every minute by default. Metrics are stored for 15 months, with granularity decreasing over time: 1-second resolution for the first 3 hours, 1-minute for 15 days, 5-minute for 63 days, and 1-hour for 455 days (15 months). You can also publish custom metrics (e.g., application-level metrics like number of orders processed) using the PutMetricData API.

Alarms watch a single metric or an expression based on multiple metrics. You set a threshold and an action. For example, if CPUUtilization > 80% for 5 consecutive minutes, send an SNS notification to your operations team. Alarms have three states: OK, ALARM, and INSUFFICIENT_DATA. You can also use composite alarms to combine multiple alarms into a single alert.

Logs – CloudWatch Logs lets you ingest, store, and analyze log files from AWS resources and applications. You can create log groups and log streams. For example, EC2 instances can send system logs to CloudWatch Logs via the CloudWatch agent. You can then search logs, create metric filters to count occurrences of specific terms (e.g., "ERROR"), and set alarms on those metrics. Logs are retained indefinitely by default, but you can set retention policies (1 day to 10 years).

CloudWatch Dashboards allow you to create customizable views of metrics and alarms. You can share dashboards across accounts or with the public (with caution).

CloudWatch Events (now part of Amazon EventBridge) – CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. For example, an EC2 instance state change to "running" triggers an event. You can set rules to route events to targets like Lambda functions, SNS topics, or SQS queues. EventBridge is the newer, more feature-rich version.

CloudWatch Agent – For monitoring EC2 instances and on-premises servers, you install the CloudWatch agent. It collects metrics (like memory utilization, disk space) and logs that are not available by default. The agent can also collect metrics from Windows or Linux systems.

CloudTrail records every API call as an event. There are two types of trails: a trail that applies to all regions (default) or a single region. When you enable CloudTrail, it automatically creates a trail that logs events for all regions. Events are delivered to an S3 bucket of your choice. You can also optionally deliver to CloudWatch Logs for real-time monitoring.

Management Events – These are operations performed on AWS resources, such as creating an EC2 instance (RunInstances), modifying a security group (AuthorizeSecurityGroupIngress), or deleting an S3 bucket (DeleteBucket). Management events are recorded by default. You can separate them into Read events (e.g., DescribeInstances) and Write events (e.g., RunInstances).

Data Events – These are operations performed on or within resources, such as S3 object-level API calls (GetObject, PutObject), Lambda function invocations, and DynamoDB item operations. Data events are not recorded by default because they can generate a high volume of logs. You must explicitly enable them.

CloudTrail Insights – An optional feature that uses machine learning to identify unusual API activity in your account, such as a spike in terminations or a user making API calls from an unusual location. Insights are recorded as events in a separate log file.

Event History – By default, CloudTrail provides a viewable, searchable, downloadable record of the last 90 days of management events in the CloudTrail console. This is free and always enabled. For longer retention, you must create a trail that delivers to S3.

Log File Integrity Validation – CloudTrail can sign log files using SHA-256 hashing and digital signatures to ensure they have not been tampered with. This is critical for compliance (e.g., PCI DSS, SOC).

In an on-premises data center, monitoring often involves a combination of SNMP traps, syslog servers, and custom scripts. CloudWatch provides a fully managed, scalable alternative with no servers to maintain. Similarly, auditing on-premises might involve reviewing server logs or database logs. CloudTrail provides a centralized, tamper-evident audit trail for all AWS API calls, which is much harder to achieve manually.

Competing cloud monitoring services include Azure Monitor and Google Cloud Operations Suite. However, for AWS-centric environments, CloudWatch and CloudTrail are tightly integrated with all AWS services, making them the natural choice.

They often work together: you can send CloudTrail logs to CloudWatch Logs to set alarms on specific API calls (e.g., alarm when an IAM user creates a new access key).

Scenario 1: E-commerce Platform Performance Monitoring

An e-commerce company runs a web application on EC2 instances behind an Application Load Balancer (ALB). They use CloudWatch to monitor key metrics: ALB request count, latency (p99), EC2 CPU utilization, and RDS connection count. They set CloudWatch Alarms to notify the DevOps team when latency exceeds 2 seconds or CPU is above 80% for 5 minutes. When an alarm triggers, an SNS notification sends an email to the on-call engineer, who can then investigate and scale out the Auto Scaling group. Additionally, they use CloudWatch Logs to collect application logs from EC2 instances. They create a metric filter to count "5xx" errors and set an alarm for high error rates. This setup ensures they can detect and respond to performance issues quickly. Cost: CloudWatch metrics and alarms are low cost; CloudWatch Logs ingestion and storage costs can be significant if logs are verbose. They mitigate this by setting log retention to 30 days and using metric filters to extract only what they need.

Scenario 2: Security Incident Investigation with CloudTrail

A financial services company needs to comply with SOC 2 and must audit all changes to their AWS environment. They enable CloudTrail in all regions with log file validation enabled. They also enable data events for S3 buckets containing sensitive customer data. When a security analyst notices an unauthorized S3 bucket deletion, they go to CloudTrail Event History and filter by 'DeleteBucket'. They find the event, which shows the IAM user who called the API, the source IP address, and the time. They discover that the user's credentials were compromised. They then use CloudTrail Insights to check for other unusual activity. The trail logs are stored in an S3 bucket with a lifecycle policy that moves older logs to Glacier after 90 days for long-term retention. They also send CloudTrail logs to CloudWatch Logs to set an alarm that triggers when a security group is modified (a critical control). This setup helps them detect and investigate security incidents quickly.

Scenario 3: Misconfiguration Leading to Oversight

A startup enables CloudTrail but forgets to enable data events for S3. Later, they suspect a data breach but cannot find any API calls for GetObject or PutObject in CloudTrail because those data events were not logged. They mistakenly think no access occurred. In reality, the logs are missing critical evidence. Also, they set CloudWatch alarms only on EC2 CPU, but not on memory or disk. When an instance runs out of memory, the application crashes, but no alarm triggers because memory metrics are not sent by default (they require the CloudWatch agent). They learn the hard way that monitoring must be comprehensive. This scenario highlights the importance of understanding default vs. optional features.