Kafka Monitoring Stack with Prometheus and Grafana

A comprehensive monitoring solution for Apache Kafka using Prometheus for metrics collection and Grafana for visualization. This repository provides a complete Docker-based setup for monitoring Kafka clusters with pre-configured dashboards and alerting rules.

🏗️ Architecture

This monitoring stack consists of:

• Apache Kafka - Message broker with JMX metrics enabled
• Zookeeper - Kafka's coordination service
• Prometheus - Time-series database for metrics collection
• Grafana - Visualization and dashboard platform
• JMX Prometheus Agent - Exposes Kafka JMX metrics to Prometheus

📋 Prerequisites

• Docker and Docker Compose
• Ports 3000, 9090, 9092, 2181 available

🚀 Quick Start

1. Clone the repository

   git clone <repository-url>
   cd kafka-promethues-grafana

2. Start the monitoring stack

   docker-compose up -d

3. Access the services

   • Grafana Dashboard: http://localhost:3000 (admin/admin)
   • Prometheus: http://localhost:9090
   • Kafka Broker: localhost:9092
   • Zookeeper: localhost:2181

📊 Features

Pre-configured Dashboards

• Kafka Overview Dashboard (kafka_dashboard.json)

  • Cluster health metrics
  • Topic and partition statistics
  • Consumer lag monitoring
  • Broker performance metrics

• Advanced Kafka Dashboard (grafana-kafka-dashboard-advanced.json)

  • Detailed broker metrics
  • Network and request statistics
  • Log and controller metrics
  • Custom visualizations

Metrics Collection

The stack collects comprehensive Kafka metrics including:

• Cluster Metrics: Broker count, controller status, cluster size
• Topic Metrics: Message rates, partition counts, replication status
• Consumer Metrics: Consumer lag, fetch rates, commit rates
• Network Metrics: Request rates, response times, connection counts
• JVM Metrics: Memory usage, GC statistics, thread counts

🔧 Configuration

Prometheus Configuration

The prometheus.yml file is configured to:

• Scrape Kafka metrics from port 7071
• Use a 15-second scrape interval
• Collect metrics with proper labeling

JMX Agent Configuration

The prom-jmx-agent-config.yml file defines:

• Metric naming conventions
• Label extraction patterns
• Metric type mappings (Counter, Gauge, etc.)

Docker Configuration

The Dockerfile extends the official Kafka image to:

• Add the JMX Prometheus agent
• Configure JMX metrics exposure
• Set up proper port mappings

📈 Monitoring Key Metrics

Critical Metrics to Watch

1. Consumer Lag (kafka_consumer_lag)

   • Indicates if consumers are keeping up with producers
   • High lag may indicate performance issues

2. Broker Health (kafka_server_broker_state)

   • Monitors broker availability and status
   • Critical for cluster stability

3. Request Rates (kafka_network_request_total)

   • Tracks incoming request volume
   • Helps identify traffic patterns

4. JVM Metrics (jvm_*)

   • Memory usage and garbage collection
   • Essential for performance tuning

🚨 Alerting

While this setup includes comprehensive monitoring, you may want to add alerting rules for:

• High consumer lag
• Broker unavailability
• High memory usage
• Network errors
• Topic replication issues

🛠️ Customization

Adding Custom Metrics

1. Modify prom-jmx-agent-config.yml to include additional JMX metrics
2. Update prometheus.yml if adding new scrape targets
3. Restart the stack: docker-compose restart

Dashboard Customization

1. Import the provided dashboard JSON files into Grafana
2. Modify panels, queries, and visualizations as needed
3. Export updated dashboards and replace the JSON files

📁 Project Structure

kafka-promethues-grafana/
├── docker-compose.yml              # Service orchestration
├── Dockerfile                      # Custom Kafka image with JMX agent
├── prometheus.yml                  # Prometheus configuration
├── prom-jmx-agent-config.yml      # JMX metrics mapping
├── kafka_dashboard.json           # Basic Kafka dashboard
├── grafana-kafka-dashboard-advanced.json  # Advanced dashboard
└── README.md                      # This file

🔍 Troubleshooting

Common Issues

1. Services not starting

   • Check if required ports are available
   • Ensure Docker has sufficient resources

2. No metrics in Grafana

   • Verify Prometheus is scraping Kafka metrics
   • Check JMX agent configuration

3. Dashboard import errors

   • Ensure Grafana version compatibility
   • Check JSON file format

Useful Commands

# Check service status
docker-compose ps

# View logs
docker-compose logs kafka
docker-compose logs prometheus
docker-compose logs grafana

# Restart specific service
docker-compose restart kafka

# Clean up
docker-compose down -v

📚 Additional Resources

• Kafka Monitoring Best Practices
• Prometheus JMX Exporter
• Grafana Dashboard Documentation

---

Note: This repository was created as part of the blog article Kafka Monitoring Using Prometheus. For detailed explanations and best practices, refer to the original article.