Elastic Stack (ELK) for Log Analysis

The Elastic Stack (formerly known as ELK Stack) is a collection of open-source tools designed for ingesting, storing, searching, and visualizing log data. It was created to address the challenge of centralized log management in distributed systems. Before tools like ELK, administrators had to SSH into each server and grep through log files individually—a slow, error-prone process. The Elastic Stack provides a unified platform where logs from hundreds or thousands of sources can be collected, normalized, and made searchable in near real-time.

The core components are: - Elasticsearch: A distributed, RESTful search and analytics engine based on Apache Lucene. It stores data as JSON documents in indices and provides powerful full-text search capabilities. - Logstash: A server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and sends it to a stash like Elasticsearch. - Kibana: A visualization layer that works on top of Elasticsearch, providing dashboards, charts, and a query interface.

The stack originally was ELK (Elasticsearch, Logstash, Kibana). With the addition of Beats (lightweight data shippers), it was rebranded as the Elastic Stack. On the exam, you may still see the term "ELK" used interchangeably.

The Elastic Stack operates in a pipeline fashion:

2. Data Processing (Logstash): Logstash uses a pipeline with three stages: inputs, filters, and outputs. - Inputs: Define where data comes from (e.g., beats, syslog, TCP/UDP). - Filters: Process and transform data. Common filters include: - grok: Parses unstructured log data into structured fields using predefined patterns. - mutate: Performs general field transformations like renaming, removing, or converting data types. - date: Parses timestamps to use for event ordering. - geoip: Adds geographic location data from IP addresses. - Outputs: Define where processed data goes (e.g., Elasticsearch, file, stdout).

Filebeat configuration (/etc/filebeat/filebeat.yml):

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

Logstash pipeline (/etc/logstash/conf.d/logstash.conf):

input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "apache-logs-%{+YYYY.MM.dd}"
  }
}

Verify cluster health:

curl -X GET "localhost:9200/_cluster/health?pretty"

List indices:

curl -X GET "localhost:9200/_cat/indices?v"

Search a specific index:

curl -X GET "localhost:9200/filebeat-*/_search?q=status:500"

Kibana Dev Tools: Use the Console tab to run Elasticsearch queries directly.

The Elastic Stack often integrates with other security tools: - SIEM: Elastic Security (formerly SIEM) app in Kibana provides pre-built dashboards and detection rules for security events. It can ingest alerts from Suricata, Zeek, and other NIDS. - Threat Intelligence: Elasticsearch can store threat intelligence feeds (e.g., known malicious IPs, domains) and enrich logs via Logstash filters or Enrichment policies. - Cloud Services: Filebeat can collect logs from AWS CloudTrail, Azure Audit Logs, and GCP Audit Logs via modules. Logstash can pull from S3 buckets using the S3 input plugin. - Containers: Filebeat and Metricbeat can run as sidecar containers or DaemonSets in Kubernetes to collect logs and metrics from pods and nodes.

A large e-commerce company runs 500 Linux servers across multiple data centers. Each server generates syslog, application logs, and security logs. The security team needs to search for anomalies (e.g., failed SSH attempts, SQL injection attempts) across all servers in real time. They deploy Filebeat on each server, sending logs to a cluster of three Logstash nodes (load-balanced via a TCP load balancer). Logstash parses the logs using grok patterns and enriches them with GeoIP data. The output goes to a six-node Elasticsearch cluster with 12TB of SSD storage. Kibana dashboards show real-time failed login attempts by source country, top targeted ports, and error rates. The team uses Index Lifecycle Management to rotate indices every 30 days or 50GB, keeping hot data on SSDs and moving older data to warm nodes with spinning disks. Common issues: Logstash becomes a bottleneck if pipelines are too complex; they mitigate by using multiple pipeline workers and tuning the batch size (e.g., 125 events per batch).

A financial institution uses the Elastic Stack as its SIEM. They ingest logs from firewalls, IDS/IPS, endpoints (via Winlogbeat and Auditbeat), and cloud services (CloudTrail via Filebeat module). The SOC analysts use Kibana's Security app to investigate alerts. Pre-built detection rules (e.g., "Suspicious Process Execution") trigger alerts when certain patterns are matched. When an alert fires, analysts use the Timeline feature to pivot from the alert to related events—for example, from a suspicious IP to all connections from that IP in the last 24 hours. They can also use machine learning jobs to detect anomalies in user behavior. Performance scales by adding more Elasticsearch nodes and using index aliases for time-based data. Misconfiguration example: if ILM is not set up, indices can grow to hundreds of gigabytes, causing slow queries and disk pressure.

A SaaS provider runs microservices on Kubernetes. Each pod writes logs to stdout/stderr, which are collected by a DaemonSet running Filebeat. Filebeat sends logs to a Logstash service that enriches them with Kubernetes metadata (pod name, namespace, labels) using the Kubernetes filter plugin. The logs are stored in Elasticsearch and visualized in Kibana. Developers use Kibana to debug application errors, while the security team monitors for suspicious activity like unexpected network connections. Challenges: log volume can spike during incidents; they use Logstash's persistent queues to buffer data and prevent loss. They also configure Elasticsearch's disk-based shard allocation to prevent nodes from running out of disk space.