Centralized Logging, VPC Flow Logs, and Log Aggregation

Let us start with the core idea: cloud logging is the act of recording events that happen inside your AWS environment. Think of it like a security camera for your virtual network. Every time an EC2 instance — an Amazon Elastic Compute Cloud instance, which is a virtual server — sends or receives a packet of data, a record can be created. That record is a log entry. AWS offers several ways to capture these entries, and the most important one for the SCS-C02 exam is called VPC Flow Logs.

VPC stands for Virtual Private Cloud. It is your own isolated section of the AWS cloud where you launch your resources. A VPC Flow Log captures metadata about the network traffic flowing in and out of your VPC. Metadata means it records who talked to whom, the port number used, the protocol (TCP or UDP), and the action taken (ACCEPT or REJECT). It does not record the actual content of the communication — just the envelope information. This is crucial for security because if an attacker tries to probe your network, a rejected connection will appear in the flow log. You can enable VPC Flow Logs at three levels: the VPC itself, a specific subnet, or a specific network interface attached to an EC2 instance. Each level gives you more granularity.

Now, where do those logs go? AWS lets you publish VPC Flow Logs to three destinations:

Centralised logging means you choose one of these destinations — usually CloudWatch Logs or S3 — and send all your VPC Flow Logs, plus other logs like AWS CloudTrail (API activity logs) and AWS Config (resource configuration change logs), to that single location. You then use a tool like Amazon OpenSearch Service (a managed version of the Elasticsearch search engine) or Amazon Athena (an interactive query service) to search and analyse the logs. Without centralisation, you would have to log into each EC2 instance separately, check its own local log files, and manually correlate timestamps — a nightmare during an incident response.

The concept of log aggregation is the process of collecting logs from multiple sources into a single repository. AWS services like Amazon CloudWatch Logs and Amazon S3 are the aggregation points. You can also use third-party solutions like Splunk or the ELK Stack (Elasticsearch, Logstash, Kibana) if you deploy them on EC2. The exam expects you to know when to use each destination based on cost, query speed, and retention requirements.

Why does centralised logging matter for security? Imagine a scenario where an employee mistakenly exposes an S3 bucket to the public internet. A VPC Flow Log would show traffic from unknown IP addresses reaching that bucket. CloudTrail would show who made the bucket policy change. By bringing both logs into one dashboard, a security analyst can quickly correlate: "Someone changed the bucket policy at 2:00 PM, and by 2:05 PM, an attacker from IP 203.0.113.5 downloaded 500 MB of data." Without centralisation, the analyst would have to dig through separate services, wasting precious minutes.

In summary, centralised logging with VPC Flow Logs gives you visibility into your network traffic, helps you detect anomalies, and provides the evidence needed for forensic investigations. For the SCS-C02 exam, you need to know how to enable VPC Flow Logs, where to publish them, and how to query them using services like Amazon Athena.

Meet Priya, the newly hired cloud security engineer at FinSecure, a mid-sized financial services company. FinSecure runs a three-tier web application on AWS: a load balancer in a public subnet, application servers in private subnets, and a database in a private subnet. Priya's first task is to investigate a suspicious spike in outbound traffic reported by the finance team. She notices that a single EC2 instance has been sending 10 GB of data per hour to an IP address in a foreign country—an abnormal pattern.

Priya starts by enabling VPC Flow Logs at the subnet level for the private subnets. She configures the logs to be published to Amazon S3 because the finance team requires logs to be retained for seven years for compliance. She then uses Amazon Athena to query the logs directly from S3. She writes a simple SQL query like: SELECT * FROM vpc_flow_logs WHERE src_ip = '10.0.1.25' AND bytes > 1000000. The query returns that the source EC2 instance is communicating over port 443 (HTTPS) to a known malicious IP address listed in threat intelligence feeds.

Now Priya needs to dig deeper. She enables AWS CloudTrail to capture API calls made by any IAM user or role in the account. She finds that a developer's access keys were used to modify the security group rules for that instance, allowing outbound traffic to any IP address. Without centralised logging, Priya would have to separately check CloudTrail logs, VPC Flow Logs, and Amazon GuardDuty findings. Instead, she uses Amazon CloudWatch Logs Insights to aggregate logs from multiple sources. She sets up a dashboard that shows VPC Flow Logs, CloudTrail events, and GuardDuty alerts side by side.

The step-by-step actions Priya took:

After rectifying the issue—rotating the compromised access keys and restricting security group rules—Priya creates a playbook for future incidents. The playbook includes steps to enable VPC Flow Logs on all new VPCs, aggregate logs into a central Amazon S3 bucket, and use Amazon Macie to scan S3 objects for sensitive data that might have been exfiltrated. FinSecure now has a centralised logging pipeline that reduces incident detection time from days to minutes.