sre-dashboards – DevOps & Infrastructure skill for AI agents

SRE Dashboards

Build dashboards that help teams detect, triage, and prevent reliability incidents.

When to Use This Skill

Use this skill when:

• Defining service-level dashboards for production systems
• Tracking SLO health and error-budget burn
• Creating incident command-center views
• Standardizing dashboard patterns across teams

Prerequisites

• Metrics pipeline (Prometheus, OpenTelemetry, or vendor equivalent)
• Logs/traces linked to services and environments
• Agreed service taxonomy (team, service, tier, environment)

Dashboard Architecture

Structure dashboards in layers:

1. Executive Reliability View: SLO attainment, incident counts, MTTR trends.

2. Service Health View: RED/USE metrics, dependency health, release markers.

3. Deep-Dive View: Per-endpoint latency, resource saturation, error categories.

Keep each view answer-oriented:

• *Are customers impacted?*
• *What changed?*
• *Where is the bottleneck?*

Core SRE Panels

Golden Signals

• Latency: p50/p95/p99 request duration by endpoint
• Traffic: request throughput and queue depth
• Errors: 5xx rate, failed jobs, timeout ratio
• Saturation: CPU, memory, disk I/O, thread/connection pool exhaustion

SLO Panels

• Current SLI value (rolling windows: 5m, 1h, 24h, 30d)
• Error-budget remaining (%)
• Burn-rate panels (fast and slow windows)
• Multi-window burn alert status

Change Correlation

• Deployment markers and config-change annotations
• Feature flag state overlays
• Upstream/downstream dependency error rates

Example PromQL Snippets

# API error rate (%)
100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# p95 latency by route
histogram_quantile(0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

# 5-minute burn rate for an illustrative 99.9% availability SLO
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
) / (1 - 0.999)

Burn rate divides the bad-event ratio by the allowed bad-event ratio (1 - SLO), not by another window’s observed errors. Compute the 1h window separately and combine short/long threshold conditions for an alert. Scope both numerator and denominator to the same service/environment; distinguish missing/no-traffic data from healthy zero errors. See [Google SRE alerting guidance](https://sre.google/workbook/alerting-on-slos/).

Operational Guidelines

• Use consistent color semantics (green=healthy, yellow=degrading, red=breach)
• Label units explicitly (ms, req/s, %, cores)
• Default time windows to incident-friendly ranges (15m, 1h, 6h, 24h)
• Minimize panel count per dashboard to reduce cognitive load
• Add runbook links directly in panel descriptions

Troubleshooting

Panel appears flat or empty

• Verify label cardinality and filters (service, env, region)
• Confirm scrape/ingest latency is within expected range
• Check metric rename regressions after instrumentation updates

High cardinality slows dashboards

• Aggregate by stable dimensions (service, route_group) instead of raw IDs
• Use recording rules for expensive percentile and ratio queries
• Split deep-dive dashboards from NOC summary dashboards

Related Skills

• prometheus-grafana (prometheus-grafana) - Dashboard implementation and PromQL
• opentelemetry (opentelemetry) - Standardized telemetry instrumentation
• alerting-oncall (alerting-oncall) - Reliability alert routing and escalation
• agent-observability (agent-observability) - AI workload reliability telemetry

Limitations

• Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
• Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.

Example

git status && git diff --stat
kubectl diff -f manifest.yaml

> Adapted from [BagelHole/DevOps-Security-Agent-Skills](https://github.com/BagelHole/DevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.