In the age of micro‑services and sprawling infrastructure, system logs have become the lifeblood of troubleshooting and performance tuning. The ELK stack—Elasticsearch, Logstash, and Kibana—offers a battle‑tested framework to ingest, parse, and visualize those logs in real time. This guide walks you through the full pipeline, from installing the stack to creating powerful dashboards that let you spot anomalies before they turn into outages.
What You’ll Need
- A Linux server (Ubuntu 22.04 or CentOS 9) with root or sudo access
- Java 17 installed (required by Elasticsearch)
- Enough RAM (at least 8 GB for a small‑to‑medium deployment)
- Internet connectivity to download packages
- Basic familiarity with shell commands and YAML syntax
Step 1: Install and Configure Elasticsearch
Elasticsearch is the heart of the ELK stack. It stores the indexed logs and serves them to Kibana for visualization. Start by adding the official repository and installing the package.
sudo apt-get update
sudo apt-get install -y openjdk-17-jdk
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/oss-8.x/apt stable main" > /etc/apt/sources.list.d/elastic-8.x.list'
sudo apt-get update
sudo apt-get install -y elasticsearch-oss
Once installed, tweak /etc/elasticsearch/elasticsearch.yml to suit a production environment. At a minimum, set the node name, cluster name, and network host.
cluster.name: elk-demo
node.name: elk-node-1
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: false
Enable and start the service:
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
Verify that the node is up:
curl http://localhost:9200/_cluster/health?pretty
Step 2: Install and Configure Logstash
Logstash acts as the ingestion layer. It pulls raw logs, applies parsing rules, and forwards structured events to Elasticsearch. Install it from the same repo:
sudo apt-get install -y logstash-oss
Create a pipeline configuration file at /etc/logstash/conf.d/system_logs.conf that reads from syslog, parses with grok, and tags the output.
input {
file {
path => "/var/log/syslog"
start_position => "beginning"
sincedb_path => "/var/lib/logstash/sincedb"
}
}
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{SYSLOGHOST:host} %{DATA:program}[%{NUMBER:pid}]: %{GREEDYDATA:msg}" }
}
date {
match => ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
mutate {
add_tag => ["syslog"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "syslog-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
Start Logstash and watch the console for any parsing errors:
sudo systemctl enable logstash
sudo systemctl start logstash
Step 3: Deploy Filebeat for Log Shipping
While Logstash can read files directly, Filebeat is lightweight and ideal for shipping logs from multiple hosts to a central Logstash instance. Install Filebeat:
sudo apt-get install -y filebeat-oss
Configure Filebeat to forward logs to Logstash. Edit /etc/filebeat/filebeat.yml and set the output:
filebeat.inputs:
- type: log
paths:
- /var/log/*.log
output.logstash:
hosts: ["localhost:5044"]
Enable the built‑in Logstash module for syslog and start the service:
sudo filebeat modules enable system
sudo filebeat setup
sudo systemctl enable filebeat
sudo systemctl start filebeat
On the Logstash side, add a new input listening on the Beats port:
input {
beats {
port => 5044
}
}
Reload Logstash to apply the change.
Step 4: Create Index Patterns in Kibana
Now that data is flowing into Elasticsearch, open Kibana by navigating to http://localhost:5601. In the Management → Index Patterns section, create a new pattern that matches syslog-*. Set @timestamp as the time field. This tells Kibana how to slice your data over time.
Once the pattern is created, explore the Discover tab. You should see parsed fields like host, program, and msg automatically extracted.
Step 5: Visualize and Query Logs in Kibana
With data indexed, build visualizations:
- Open Visualize Library → Create new visualization.
- Choose Bar chart and set the Y‑axis to Count of documents.
- Set the X‑axis to Terms on the
program.keywordfield. - Apply a filter for the last 24 hours.
- Save and add it to a new dashboard.
Repeat for other dimensions (e.g., host.keyword, msg.keyword). Combine multiple visualizations into a single dashboard for a holistic view of your system’s health.
Step 6: Set Up Alerts with Watcher (Optional)
For proactive monitoring, use Kibana’s built‑in Watcher (requires the X-Pack license, but the OSS version offers basic alerting). Define a simple watch that triggers when the error rate exceeds 5% in a minute:
PUT _watcher/watch/error_rate
{
"trigger": {
"schedule": { "interval": "1m" }
},
"input": {
"search": {
"request": {
"indices": ["syslog-*"],
"body": {
"query": { "match": { "level": "ERROR" } }
}
}
}
},
"condition": { "compare": { "ctx.payload.hits.total": { "gt": 5 } } },
"actions": {
"email_admin": {
"email": {
"to": "admin@example.com",
"subject": "High error rate detected",
"body": "{{ctx.payload.hits.total}} errors in the last minute."
}
}
}
}
Enable the watch and you’ll receive an email whenever the threshold is breached.
Common Mistakes to Avoid
1. Over‑Indexing: Creating an index for every log file can quickly exhaust disk space. Use index templates and lifecycle policies to rollover and delete old indices.
2. Skipping Field Normalization: Forgetting to set keyword subfields on string fields limits your ability to filter and aggregate. Define them in your index template.
3. Ignoring Security: Running Elasticsearch on an open network port without authentication exposes sensitive data. Enable TLS and user authentication in production.
4. Misconfiguring Log Rotation: If syslog rotates too quickly, Logstash may miss new files. Ensure file input’s ignore_older and sincedb_path are set appropriately.
Tips and Tricks
Use Templates Early: Define an _template that sets field mappings and analyzers before any data lands. This prevents mapping conflicts later.
Leverage Metricbeat: Complement Filebeat with Metricbeat to ingest host metrics (CPU, memory, disk). Correlate metrics with log events for richer insights.
Batch Grok Patterns: Instead of writing a new grok rule for every log type, create reusable patterns in /etc/logstash/patterns and reference them.
Monitor Elasticsearch Health: Use curl http://localhost:9200/_cat/indices?v to spot shards stuck in UNASSIGNED state.
Frequently Asked Questions
What version of Elasticsearch should I use?
Stick with the latest stable release that matches your OS. For most production setups, Elasticsearch 8.x is recommended due to its performance and security improvements.
Can I run the ELK stack on a single VM?
Yes, for development or small environments a single VM can host all three components. However, for high availability and scalability, separate nodes for Elasticsearch, Logstash, and Kibana are best practice.
How do I troubleshoot a missing field in Kibana?
Check the Logstash pipeline logs for parsing errors. Ensure the grok pattern matches the log line exactly. If a field is missing, add a mutate filter to rename or default it.
Conclusion
By now you should have a fully functional ELK stack ingesting, indexing, and visualizing system logs. The power of this setup lies in its extensibility: add new pipelines for application logs, enrich data with GeoIP or user agent parsing, and scale horizontally as traffic grows. Remember that the key to mastery is iterative tuning—regularly review your index templates, monitor cluster health, and adjust alert thresholds based on real‑world usage. Happy logging!
Photo by Luke Chesser on Unsplash
