AKS Monitoring: Container Insights and Prometheus

Container Insights is a feature of Azure Monitor that collects performance metrics, inventory data, and logs from containerized workloads running on AKS, AKS-engine, and Azure Container Instances. It solves the problem of 'black box' containers: without it, you would have to SSH into nodes, run kubectl top or docker stats, and manually correlate data across clusters. Container Insights automates this by deploying a containerized Log Analytics agent (the omsagent) as a DaemonSet on each node. The agent collects:

The data is stored in a Log Analytics workspace, where you can query it using Kusto Query Language (KQL), create alerts, and visualize with workbooks. Container Insights is enabled per cluster and requires the Microsoft.OperationsManagement and Microsoft.OperationalInsights resource providers to be registered.

When you enable Container Insights on an AKS cluster, the following happens:

2. Metrics Collection: The agent scrapes metrics from: - cAdvisor: A built-in Kubernetes component that exposes container resource usage via the kubelet API. The agent queries /stats/summary on the kubelet every 60 seconds by default. - Kubelet API: For node-level metrics like node_cpu_usage, node_memory_working_set. - Kubernetes API: For inventory data (pods, deployments, namespaces) every 60 seconds.

4. Data Flow: The agent sends data to the Log Analytics workspace over HTTPS on port 443. The data is stored in custom log tables: - Perf: Performance metrics (CPU, memory, disk, network). - ContainerInventory: Container lifecycle events. - ContainerLog: Container stdout/stderr logs. - KubePodInventory: Pod metadata. - InsightsMetrics: Additional metrics like node filesystem usage.

Key Timers and Defaults:

Prometheus is an open-source monitoring and alerting toolkit originally built by SoundCloud. It is now a CNCF-graduated project and the de facto standard for Kubernetes monitoring. Prometheus excels at collecting high-dimensional, time-series data from targets that expose metrics via HTTP endpoints (e.g., /metrics). It is pull-based: the Prometheus server scrapes metrics from targets at configurable intervals. For AKS, you can deploy Prometheus yourself (using the Prometheus Operator or Helm chart) or use Azure Monitor's managed Prometheus service (part of Azure Monitor managed service for Prometheus).

The exam focuses on the integration of Prometheus with Azure Monitor via Container Insights. You can enable Prometheus metrics scraping in Container Insights by modifying the agent ConfigMap to add scrape targets. This allows you to collect Prometheus-formatted metrics from your applications and send them to the Log Analytics workspace, where they are stored in the InsightsMetrics table.

When you configure Container Insights to scrape Prometheus metrics:

InsightsMetrics
| where Namespace == "prometheus"
| where Name == "container_cpu_usage_seconds_total"
| summarize avg(Val) by bin(TimeGenerated, 1m), Computer

Key Prometheus Concepts: - Metric types: Counter (cumulative), Gauge (current value), Histogram (buckets), Summary (quantiles). - Labels: Key-value pairs that add dimensions to metrics (e.g., pod, namespace, method). - Scrape interval: Default 60 seconds, configurable per target. - Scrape timeout: Default 10 seconds.

To enable Prometheus scraping, you must create or edit the ConfigMap container-azm-ms-agentconfig with the following structure:

apiVersion: v1
kind: ConfigMap
metadata:
  name: container-azm-ms-agentconfig
  namespace: kube-system
data:
  prometheus-data-collection-settings: |-
    [cluster_level_targets]
    # Targets that are accessible from any node
    targets = ["https://kubernetes.default.svc:443/metrics"]
    interval = "60s"

    [node_level_targets]
    # Targets per node
    targets = ["http://<node-ip>:9100/metrics"]
    interval = "30s"

You can also use a ConfigMap from a URL or apply it directly via kubectl apply -f. After applying, check the agent logs for errors.

Container Insights is a managed, out-of-the-box monitoring solution that requires no configuration to start collecting basic metrics. Prometheus is a more powerful but complex tool that requires setup of exporters and scrape targets. The exam tests when to use each:

To verify Container Insights is working:

kubectl get ds omsagent --namespace=kube-system
# Should show desired = ready = number of nodes

Check agent logs:

kubectl logs -n kube-system -l app=omsagent

Check if metrics are flowing to Log Analytics: run a query in the Log Analytics workspace:

Perf
| where TimeGenerated > ago(5m)
| count

For Prometheus scraping, check the ConfigMap:

kubectl get configmap container-azm-ms-agentconfig -o yaml -n kube-system

If Prometheus metrics are not appearing, check the agent logs for scrape errors:

kubectl logs -n kube-system -l app=omsagent --tail=50 | grep "prometheus"

Container Insights data flows into Azure Monitor, which means you can:

The exam may ask about alert rules: you must specify the 'Resource' as the Log Analytics workspace, not the AKS cluster directly, for log alerts based on Container Insights data.

Container Insights costs are based on: - Log Analytics data ingestion: The amount of data sent to the workspace (GB/month). Container logs and performance metrics count towards this. - Data retention: Beyond 31 days (for Perf and ContainerLog) incurs additional cost. - Prometheus scraping: No extra cost for the agent, but the additional metrics increase ingestion volume. - Live Data: Free (uses existing Kubernetes API bandwidth).

Default data retention is 30 days for the workspace. You can change it per table (e.g., retain Perf for 90 days, ContainerLog for 30 days).

The omsagent requires: - Privileged container access: To read Docker socket and cgroups. - Kubernetes RBAC: A ClusterRole cluster-admin (or custom) to read Kubernetes API objects. - Log Analytics workspace key: Stored as a Kubernetes secret in the cluster.

On the exam, remember that Container Insights cannot be enabled on a cluster that is private (no outbound internet access) unless you use a Log Analytics gateway or enable forced tunneling with a firewall. The agent must be able to reach the Log Analytics endpoint (e.g., <workspace-id>.ods.opinsights.azure.com).

A large e-commerce company runs 50 microservices on a 10-node AKS cluster. They need to monitor both infrastructure health and business metrics (e.g., checkout latency, order rate). They enable Container Insights for basic CPU/memory alerts and to collect container logs. They then configure Prometheus scraping to collect custom metrics from each microservice (e.g., http_requests_total, checkout_duration_seconds). They use the InsightsMetrics table to create a Grafana dashboard showing real-time order rates and error rates. The challenge: high cardinality metrics (e.g., per-URL request counts) can explode ingestion costs. They solve this by using Prometheus relabeling to drop unnecessary labels and by setting a metric collection interval of 60 seconds instead of 15.

A bank runs a PCI-DSS compliant AKS cluster. They cannot use the default Container Insights agent because it requires privileged access. They deploy the agent with a custom ConfigMap that limits log collection to specific namespaces and disables the collection of sensitive data. They also use Azure Policy to enforce that only approved container images are used. Prometheus is used to monitor node-level metrics (disk I/O, network) via node exporters, but application metrics are collected via a sidecar exporter that sends data to a separate Prometheus server outside the cluster to meet audit requirements. The key lesson: in regulated environments, you must review the agent's data collection scope and ensure no sensitive data leaves the cluster.

A SaaS provider manages 20 AKS clusters across different regions. They send all Container Insights data to a single Log Analytics workspace (cross-region) to have a unified view. They use Azure Workbooks to create a 'global health' dashboard that shows the status of all clusters. The pitfall: if the workspace is in a different region, data transfer costs apply. Also, if one cluster's agent has network issues, it can affect the entire workspace's data completeness. They mitigate this by setting up health alerts per cluster and using a separate workspace for critical production clusters. The exam may test that a Log Analytics workspace can receive data from multiple clusters, but you cannot have a single cluster send data to multiple workspaces.