Dashboards, the very thing that has to hint you what is broken becomes the thing most likely to break.
"Dashboard-as-Code" has been a broken promise for years. Teams export dashboards into giant JSON blobs or YAML files, only to commit them to Git, and call it "Infrastructure as Code". But these files are difficult to review in a pull request, they can break silently when a field name changes, and they offer little validation before deployment.
What if your observability dashboards could be treated the same way as your application code?
This talk covers Perses, a CNCF Sandbox project for vendor neutral dashboards, and its Kubernetes Operator. Together they bring Kubernetes-native Dashboard-as-Code to the cloud-native ecosystem providing the option to author dashboards with the Perses Go SDK and deploy them via GitOps.
This talk will focus on Prometheus metrics dashboards (Prometheus plugin).
The Problem
Consider a typical observability workflow today:
A team member builds a dashboard in a UI, exports it as a 1000 line JSON or YAML file, and commits it.
Another team member edits a Prometheus query as a raw string. The change is hard to review in a giant JSON/YAML diff, and nothing in the usual export and commit workflow validates the dashboard definition before deploy.
The dashboard gets deployed and panels break due to a malformed query or a wrong field in the JSON/YAML.
The Perses Approach
Perses provides an open dashboard and datasource specification aimed at interoperability across tools. The Perses Operator brings dashboard and datasource management to Kubernetes with four Custom Resource Definitions (CRDs):
Perses: Declares a Perses server instance. The operator manages the Deployment or StatefulSet, Service, ConfigMap, and optional TLS, storage, and health probes.
PersesDashboard: Defines a dashboard as a Kubernetes resource. The Kubernetes namespace maps to a Perses project.
PersesDatasource: Declares a project scoped datasource (e.g., a Prometheus endpoint) synced to matching Perses instances.
PersesGlobalDatasource: A cluster scoped datasource shared across all projects.
You do not have to hand write these manifests. With the Perses Go SDK and promql-builder, dashboards are authored in Go and can be generated as PersesDashboard YAML for GitOps. promql-builder constructs PromQL as an AST instead of fragile raw strings, with built-in validation. The YAML is the delivery format; the source of truth is code.
Example: A generated PersesDashboard CR (simplified):
apiVersion: perses.dev/v1alpha2
kind: PersesDashboard
metadata:
name: node-exporter-nodes
namespace: perses-dev
spec:
config:
display:
name: Node Exporter / Nodes
duration: 1h
layouts:
- kind: Grid
spec:
display:
title: CPU
items:
- x: 0
y: 0
width: 12
height: 8
content:
$ref: "#/spec/panels/0_0"
panels:
"0_0":
kind: Panel
spec:
display:
name: CPU Usage
description: CPU busy ratio (1 − idle)
plugin:
kind: TimeSeriesChart
spec:
yAxis:
format:
unit: percent-decimal
queries:
- kind: TimeSeriesQuery
spec:
plugin:
kind: PrometheusTimeSeriesQuery
spec:
query: |
1 - avg by (instance) (
rate(
node_cpu_seconds_total{instance=~"$instance",job="node-exporter",mode="idle"}[$__rate_interval]
)
)
seriesNameFormat: "{{instance}}"
The operator watches for changes, syncs valid dashboards to the matching Perses instance via the Perses API, and reports status on the Custom Resource. No manual imports and no fragile copy paste scripts.
What I'll Cover
From Go code to PersesDashboard CRs: author dashboards with the Perses Go SDK and promql-builder, validate PromQL at build / CI time, and generate PersesDashboard Custom Resources instead of maintaining giant exported JSON or YAML.
Compose and extend: import reusable panels from community-mixins as a Go module, customize label matchers for your environment, and add your own panels alongside them.
GitOps workflow: PR review, render manifests, deploy via Argo CD; the operator reconciles PersesDashboard CRs and they appear in the Perses UI.
