Grafana Wiki

Thus far this page was auto-generated with Cursor, will put my git info in here soon.

A working reference for Grafana and the surrounding observability stack: what each piece stores, how data gets there, and how you look at it.

What Grafana is

Grafana is a visualization and alerting front end. It does not have to own your data. It queries data sources (Prometheus, Loki, Tempo, SQL, cloud APIs, and many others), draws dashboards, and fires alerts.

Grafana Labs also builds the storage and collection pieces most teams pair with it. Together they are often called the LGTM (or LGTM+) stack.

LGTM+ stack

Four telemetry databases plus Grafana as the UI. You can run them yourself or use Grafana Cloud, where they are hosted for you.

  • Loki — logs. Prometheus-style labels, not a full-text search cluster. Query language: LogQL.
  • Grafana (the app) — dashboards, Explore, alerting, correlations across metrics, logs, traces, and profiles.
  • Tempo — distributed traces. Cheap to store at volume because it is optimized for trace IDs and TraceQL, not a metrics index of every span. Query language: TraceQL.
  • Mimir — long-term, highly available Prometheus-compatible metrics. Query language: PromQL.
  • Pyroscope — continuous profiling (CPU, memory, and related signals) so you can see why a service is hot, not only that it is.

Mental shortcut: metrics tell you something is wrong, logs tell you what happened, traces tell you where it went, profiles tell you why it was slow.

Getting data in: Alloy

Grafana Alloy is the current collector. It is an OpenTelemetry Collector distribution with native Prometheus pipelines and first-class Loki / Tempo / Pyroscope support.

One Alloy process can scrape or receive metrics, logs, traces, and profiles, then fan them out. You wire components in a config file: receivers, processors, exporters. Typical destinations:

  • Metrics → Mimir or Prometheus via remote_write
  • Logs → Loki push API
  • Traces → Tempo over OTLP
  • Profiles → Pyroscope

Older names you will still see in the wild: Grafana Agent (Alloy's predecessor) and Promtail (Loki-only log shipper). New work should start on Alloy.

Grafana itself

Data sources

A data source is a connection plus query dialect. Dashboards are just panels that each run queries against one or more sources. You can mix Prometheus, Loki, SQL, and cloud plugins on one dashboard.

Dashboards and Explore

  • Dashboards — saved, shareable layouts. Variables let one board cover many hosts, jobs, or environments.
  • Explore — ad-hoc query UI. Use it to hunt, then save a working query as a panel.
  • Correlations / Exemplars — jump from a metric spike to related logs or a trace ID.

Alerting

Grafana Alerting evaluates rules against data sources on a schedule. Firing alerts route through notification policies to contact points (email, Slack, PagerDuty, webhooks). Keep alert queries as simple as the dashboard queries they came from; complexity belongs in recording rules or the backend, not in a 40-line panel expression.

Query cheatsheet

  • PromQL — rate(http_requests_total[5m]), histogram_quantile(0.99, ...). Instant vectors vs range vectors matter.
  • LogQL — label selector first, then pipeline: {app="api"} |= "error" | json | status >= 500.
  • TraceQL — find spans by service, duration, status, or attributes, then open the trace.

How to think about a new environment

  1. Pick backends (self-hosted LGTM, Grafana Cloud, or “Grafana + existing Prometheus”).
  2. Put Alloy (or an equivalent collector) next to the workloads.
  3. Confirm each signal arrives: a node metric, a log line, a test trace.
  4. Build one service dashboard that links metrics → logs → traces.
  5. Add a handful of alerts on symptoms (error rate, latency, saturation), not every metric.

Related project notes on this site: Oxide Wiki.