CloudWatch Logs Agent and Unified Agent

Amazon CloudWatch Agents are software components that you install on EC2 instances or on-premises servers to collect logs and metrics and send them to AWS. The CloudWatch Logs Agent (often called the legacy agent) was the first iteration, designed solely to forward log files to CloudWatch Logs. The Unified CloudWatch Agent is the newer, recommended agent that can collect both logs and system metrics (CPU, memory, disk, network) and send them to CloudWatch Logs and CloudWatch Metrics.

The legacy agent was limited. It could only push log data. As AWS evolved, customers needed a single agent to collect both logs and metrics for comprehensive monitoring. The Unified Agent was released in 2018 and is now the standard. AWS recommends using the Unified Agent for all new deployments. The legacy agent is still supported but not actively developed.

The legacy agent is a Python-based daemon that runs on Linux instances. It monitors specified log files and sends new log events to a CloudWatch Logs log stream within a log group. Configuration is done via a JSON file at /etc/awslogs/awscli.conf and a separate config file at /etc/awslogs/config/. - It uses the AWS CLI credentials from the instance metadata or a credentials file. - It reads log files line by line and batches events before sending. - Default batch size is 10,000 events or 1 MB, whichever is smaller. - The agent uses a state file (/var/lib/awslogs/agent-state) to track which lines have been sent, ensuring no duplicates. - It only supports Linux instances. - It cannot collect custom metrics or system metrics.

The Unified Agent is also Python-based but more modular. It can collect: - Logs: From files, and can filter, transform, and route them. - Metrics: CPU, memory, disk, network, swap, and custom metrics from collectd or statsd. - Procstat metrics: Process-level metrics like CPU and memory per process.

Configuration is done via a single JSON or TOML file, typically at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json. The agent supports both Linux and Windows Server.

Internal Mechanism: - The agent runs as a service: amazon-cloudwatch-agent. - It reads the configuration file and starts collectors. - For logs, it uses file watchers (inotify on Linux) to detect new lines. - For metrics, it uses system calls (e.g., /proc/stat on Linux, performance counters on Windows). - Data is buffered and sent in batches to CloudWatch. - The agent can also send metrics to CloudWatch using the PutMetricData API with a resolution of either standard (60 seconds) or high-resolution (1 second) if configured.

Legacy Agent: - State file: /var/lib/awslogs/agent-state - Log group default: /var/log/messages - Batch size: 10,000 lines or 1 MB - Credentials: Uses default AWS credential chain (instance profile > environment variables > ~/.aws/credentials).

Unified Agent: - Configuration file: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json - State file: /opt/aws/amazon-cloudwatch-agent/logs/state - Metrics collection interval: default 60 seconds, configurable down to 1 second. - Log stream name: can be auto-generated using {instance_id}, {hostname}, or custom. - Log group name: configurable per log file. - Supported platforms: Amazon Linux, Ubuntu, CentOS, Red Hat, Windows Server 2012 R2 and later. - The agent can also be configured via AWS Systems Manager Parameter Store or AWS Secrets Manager.

Installing the Unified Agent:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
unzip AmazonCloudWatchAgent.zip
sudo ./install.sh

Configuration Wizard:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

This interactive wizard produces a JSON config file.

Starting the Agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Verification:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

Check CloudWatch Logs for log streams. Check CloudWatch Metrics for custom metrics.

Logs Example Config:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/syslog",
            "log_group_name": "system-logs",
            "log_stream_name": "{instance_id}",
            "timezone": "UTC"
          }
        ]
      }
    }
  }
}

Metrics Example Config:

{
  "metrics": {
    "namespace": "CustomMetrics",
    "metrics_collected": {
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user"],
        "metrics_collection_interval": 60
      },
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      }
    }
  }
}

| Feature | Legacy Agent | Unified Agent | |---------|--------------|---------------| | Log collection | Yes | Yes | | Custom metrics | No | Yes | | Procstat metrics | No | Yes | | Windows support | No | Yes | | Configuration format | Two files (INI+JSON) | Single JSON/TOML | | Configuration wizard | No | Yes | | SSM integration | No | Yes | | High-resolution metrics | No | Yes | | Filtering/transformation | No | Yes (log filters) |

A financial services company must retain application logs for seven years to meet regulatory requirements. They have hundreds of EC2 instances running Linux and Windows. They use the Unified CloudWatch Agent to collect logs from all instances into a central CloudWatch Logs group per application. Logs are then exported to an S3 bucket using an export task (or via Kinesis Firehose) for long-term storage. The Unified Agent is chosen over the legacy agent because it supports Windows servers and can also collect system metrics for operational monitoring. In production, they configure the agent to use a log group per application and a log stream per instance. They set the log retention policy on the log group to 7 years. They also use CloudWatch Logs Insights to search for specific error patterns across all instances.

A SaaS provider runs a web application on an Auto Scaling group. They need to scale based on memory utilization, but the standard EC2 metrics do not include memory. They deploy the Unified CloudWatch Agent on each EC2 instance to publish memory metrics (mem_used_percent) to CloudWatch. They create a CloudWatch Alarm on the average memory utilization across the Auto Scaling group and use that to trigger scaling policies. The Unified Agent is configured with a metrics collection interval of 60 seconds. They also collect disk and network metrics for troubleshooting. In this scenario, the legacy agent would be insufficient because it cannot publish memory metrics.