Grafana, Prometheus, Loki, Promtail, Alloy, Tempo and Exporters

• configuration

Metric categories

• System-level Performance Metrics
• Application-level Performance Metrics
• Server Availability and Uptime
• Business-level Metrics

Methods

USE	RED	Google SRE
Utilization (Usage)	Rate (Number of requests)	Latency (Time to service the request)
Saturation (Load)	Error	Traffic
Error	Duration (Time to finish request)	Error
		Saturation (How full is service)

Notes

• Prometheus and Alloy both can scrape metrics from endpoints and remote write to Mimir

Installation

Install minio for storing logs and metrics

helm upgrade -i operator minio-operator/operator \
  --namespace minio-operator \
  --create-namespace \
  --version 7.1.1

helm upgrade -i tenant minio-operator/tenant \
  --namespace minio-operator \
  --create-namespace \
  --version 7.1.1 \
  -f ../minio/k8s/helm/tenant-values.yaml

kube-prometheus-stack

Setup custom and external metrics API server

Loki

Exporters

Promtail

Fluent-Bit

Alloy

Tempo

Mimir

Pyroscope

Grafana AI (Grot)

k8s-monitoring

The Kubernetes Monitoring Helm chart is a comprehensive Helm chart provided by Grafana that deploys a complete monitoring solution for Kubernetes clusters. It is designed to simplify the collection, processing, and forwarding of telemetry data—including metrics, logs, traces, and profiles—from both your cluster and the applications running within it.

Yes, the Kubernetes Monitoring Helm chart primarily deploys and manages the configuration of Grafana Alloy and related monitoring components within your Kubernetes cluster. It automates the creation and management of Alloy objects based on the features you enable in your values.yaml file. The chart also installs and configures supporting components such as Node Exporter, kube-state-metrics, and others, but it does not install storage backends like Loki or Prometheus. Instead, it forwards collected telemetry data to the destinations you specify.

Generate helm or terraform code from grafana cloud. See Guide

# need for opencost
kubectl create ns observability
kubectl -n observability create secret generic grafana-cloud-metrics-grafana-k8s-monitoring \
  --from-literal=password='glc_xxx=' --dry-run=client -o yaml | kubectl apply -f -

kubectl -n observability create secret generic alloy-logs-remote-cfg-grafana-k8s-monitoring \
  --from-literal=password='glc_xxx=' --dry-run=client -o yaml | kubectl apply -f -

helm upgrade -i k8s-monitoring grafana/k8s-monitoring \
  -n observability --create-namespace \
  --version 3.1.2 \
  -f k8s/helm/k8s-monitoring.yaml