Observability

The api and worker each expose an ops listener (default 0.0.0.0:9090, configured by METRICS_ADDR, set-but-empty disables) serving:

Endpoint	Purpose
GET /metrics	Prometheus scrape endpoint
GET /healthz	Liveness — 200 while the process runs, no dependency checks
GET /readyz	Readiness — Postgres ping, Redis ping, migrations current

The ops port is not authenticated by design: metrics expose operational detail (cluster names, error rates), so bind it to an internal interface / network and scrape it there. Never publish it through your reverse proxy. In the dev compose stack the api maps to host 9090 and the worker to 9091.

The api additionally keeps /healthz and /readyz on its public API port (part of the OpenAPI contract); both surfaces run the same checks.

Readiness semantics

/readyz returns 503 {"status":"not_ready","dependency":"..."} naming the first failing dependency: postgres, redis, or migrations.

The migrations gate compares the schema version in goose_db_version against the newest migration embedded in the binary at build time: a database behind the binary is not ready (apply migrations first — they are never applied automatically); a database ahead of the binary is accepted (rolling upgrade). A never-migrated database is not ready.

Prometheus scrape configuration

scrape_configs:
  - job_name: planeon-api
    static_configs:
      - targets: ["your-host:9090"]
  - job_name: planeon-worker
    static_configs:
      - targets: ["your-host:9091"]

Metric reference

Cardinality is bounded by design: labels are job kind, cluster name, session state, status, outcome and HTTP method — never vmid, job id or request path.

Metric	Type	Labels	Meaning
planeon_jobs_depth	gauge	kind, status, cluster	Jobs queued/running right now (exported by api and worker — aggregate exporters with max by; cluster is empty for jobs not tied to one)
planeon_jobs_oldest_queued_age_seconds	gauge	kind, cluster	Age of the oldest queued job; 0 when the queue is empty
planeon_jobs_processed_total	counter	kind, outcome	Runner outcomes: succeeded, retried, failed (terminal), released (graceful-shutdown requeue), lease_lost (result discarded after reclaim)
planeon_job_duration_seconds	histogram	kind	Handler execution time
planeon_job_lease_reclaims_total	counter	kind	Expired worker leases reclaimed (job requeued or failed after a worker crash)
planeon_reconcile_duration_seconds	histogram	—	One reconcile cycle across active pools
planeon_reconcile_cycles_total	counter	outcome	Reconcile cycles: ok / error
planeon_pve_requests_total	counter	cluster, method, outcome	Proxmox API calls: success / error / timeout
planeon_pve_request_duration_seconds	histogram	cluster	Proxmox API round-trip latency
planeon_sessions	gauge	state, cluster	Sessions by lifecycle state and PVE cluster
planeon_db_pool_*	gauge/counter	—	pgx pool: conns (max/total/idle/acquired/constructing), acquire counters and cumulative wait
go_*, process_*	—	—	Standard runtime collectors

Queue and session gauges are computed from Postgres at scrape time by both apps on purpose: if the worker is down, the growing queue is still visible through the api. Point dashboards at one job or aggregate with max by (...).

Grafana dashboard

Download planeon-overview.json (served by this docs site), import it into Grafana, and select your Prometheus datasource. Panels are grouped into Availability, Jobs, Reconcile, PVE, Sessions and DB pool rows, with units and thresholds matching the alerting suggestions below. Template variables at the top filter every panel: $job / $instance select the scrape target (api vs worker — both export the DB-derived gauges), $cluster filters by PVE cluster name, and $tenant is a forward-compatibility placeholder that stays on "All" until metrics carry a tenant label.

Alerting suggestions

No alerting ships with the platform — bring your own stack. Suggested rules:

groups:
  - name: planeon
    rules:
      - alert: PlaneonReadyzDown
        expr: up == 0
        for: 3m
        annotations:
          summary: "Planeon {{ $labels.job }} scrape target down"
      - alert: PlaneonQueueBacklogAge
        expr: max(planeon_jobs_oldest_queued_age_seconds) > 300
        for: 10m
        annotations:
          summary: "Jobs waiting more than 5 minutes — worker down or stuck?"
      - alert: PlaneonJobFailures
        expr: sum(rate(planeon_jobs_processed_total{outcome="failed"}[15m])) > 0
        for: 15m
        annotations:
          summary: "Jobs failing terminally"
      - alert: PlaneonLeaseReclaims
        expr: sum(increase(planeon_job_lease_reclaims_total[15m])) > 0
        annotations:
          summary: "A worker crashed mid-job or stalled past its lease — check worker restarts/logs"
      - alert: PlaneonPVEErrors
        expr: |
          sum by (cluster) (rate(planeon_pve_requests_total{outcome!="success"}[10m]))
            / sum by (cluster) (rate(planeon_pve_requests_total[10m])) > 0.2
        for: 10m
        annotations:
          summary: "More than 20% of PVE API calls failing on {{ $labels.cluster }}"
      - alert: PlaneonDBPoolSaturated
        expr: planeon_db_pool_acquired_conns >= planeon_db_pool_max_conns
        for: 5m
        annotations:
          summary: "pgx pool exhausted on {{ $labels.instance }}"