EC2 Enhanced Monitoring and CloudWatch Agent

EC2 Enhanced Monitoring (also called Detailed Monitoring) is a feature that enables EC2 instances to send metrics to CloudWatch at a 1-minute frequency instead of the default 5-minute frequency. However, Enhanced Monitoring only covers hypervisor-level metrics such as CPU, network, and disk I/O (the same metrics available at 5-minute intervals, but more granular). It does NOT provide OS-level metrics like memory usage, disk space, or process-level information. To get those, you need the CloudWatch Agent.

The CloudWatch Agent is a software package that you install on an EC2 instance (or on-premises server) to collect system-level metrics and custom logs. It replaces the older CloudWatch Logs Agent and the deprecated EC2Config service (for Windows). The agent can collect:

The CloudWatch Agent runs as a background service on the instance. It uses a JSON configuration file (usually located at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json on Linux or C:\ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.json on Windows) to determine what to collect and where to send it. The agent collects metrics at a configurable interval (default 60 seconds) and sends them to CloudWatch via the CloudWatch API. It can also send logs to CloudWatch Logs.

The agent uses the AWS SDK to authenticate. It requires IAM permissions to put metrics and logs. The recommended approach is to attach an IAM role to the EC2 instance with the CloudWatchAgentServerPolicy managed policy. For on-premises servers, you create an IAM user with programmatic access and configure the agent with those credentials.

Installation (Linux - Amazon Linux 2/Ubuntu/CentOS):

sudo yum install amazon-cloudwatch-agent -y   # Amazon Linux 2
sudo apt-get install amazon-cloudwatch-agent  # Ubuntu

Configuration Wizard:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

This interactive wizard creates the configuration file. You can also manually edit the JSON file.

Start the Agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

The -m ec2 indicates the source of credentials (EC2 instance role). Use -m onPrem for on-premises.

Verify Agent Status:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

Check Logs:

sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Windows Installation: Download the agent from AWS or use SSM. Run the MSI. Then use the wizard or manually configure.

Below is a sample configuration file that collects CPU, memory, disk, and network metrics every 60 seconds, and sends logs from /var/log/syslog:

{
  "agent": {
    "metrics_collection_interval": 60,
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  },
  "metrics": {
    "namespace": "CWAgent",
    "metrics_collected": {
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
        "metrics_collection_interval": 60
      },
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": ["used_percent"],
        "resources": ["*"],
        "metrics_collection_interval": 60
      },
      "net": {
        "measurement": ["bytes_sent", "bytes_recv"],
        "metrics_collection_interval": 60
      }
    },
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/syslog",
            "log_group_name": "syslog",
            "log_stream_name": "{instance_id}",
            "timestamp_format": "%b %d %H:%M:%S"
          }
        ]
      }
    }
  }
}

| Metric Category | Default EC2 Metrics (5-min) | Enhanced Monitoring (1-min) | CloudWatch Agent (OS-level) | |-----------------|-----------------------------|-----------------------------|-----------------------------| | CPU | CPUUtilization | CPUUtilization (1-min) | cpu_usage_idle, user, system, etc. | | Network | NetworkIn, NetworkOut | Same (1-min) | bytes_sent, bytes_recv, packets, etc. | | Disk I/O | DiskReadOps, DiskWriteOps | Same (1-min) | disk_io (per device) | | Memory | Not available | Not available | mem_used_percent, mem_available, etc. | | Disk Space | Not available | Not available | disk_used_percent, disk_free, etc. | | Swap | Not available | Not available | swap_used_percent | | Processes | Not available | Not available | procstat (if configured) | | Logs | Not available | Not available | Log file collection |

The CloudWatch Agent supports publishing metrics at sub-minute resolution (down to 1 second). To enable high-resolution metrics, set metrics_collection_interval to 1 in the configuration file. When you do this, the agent automatically calls PutMetricData with StorageResolution=1. Note that high-resolution metrics are stored for only 3 hours by default, but you can extend retention by defining a metric filter or using the API. Also, high-resolution metrics cost more.

The instance role must have at least the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:DescribeLogGroups",
        "logs:CreateLogStream",
        "logs:CreateLogGroup"
      ],
      "Resource": "*"
    }
  ]
}

The managed policy CloudWatchAgentServerPolicy provides these permissions.

This content directly maps to SOA-C02 Domain 1: Monitoring, Objective 1.1: Implement monitoring and reporting. Specifically, you need to know:

In a real-world enterprise, you might manage hundreds of EC2 instances running a critical web application. Without the CloudWatch Agent, you have no visibility into memory or disk usage. One common scenario is a memory leak in the application that causes instances to become unresponsive. Without agent-collected memory metrics, you would only see CPU utilization rise, which is a secondary effect. By installing the agent and setting an alarm on mem_used_percent > 90%, you can proactively detect the leak before it causes downtime. In production, you would deploy the agent via Systems Manager State Manager to ensure all instances have the same configuration. You would also collect logs from the application's log files to CloudWatch Logs for centralized troubleshooting. Another scenario is monitoring disk space on data nodes. If a server runs out of disk, it can fail. The agent collects disk_used_percent per device, allowing you to set an alarm at 80% to trigger an automatic scaling or cleanup process. A common misconfiguration is forgetting to append dimensions like InstanceId or InstanceType, making it hard to identify which instance the metric belongs to. Also, if you set the collection interval too low (e.g., 1 second) without understanding the cost implications, your monthly bill can spike. In large fleets, you should use the CWAgent namespace and organize metrics with tags. Finally, when troubleshooting, always check the agent log file at /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log for errors like permission denied or invalid JSON.