Observability Backends

An observability backend is a metrics store you run — VictoriaMetrics, Grafana Mimir or Prometheus — connected to ZopDay as an org-level integration. Attach it to a VM deployment space and that space’s VM ships host and container metrics to it every 30 seconds. The service Observability tab then reads those numbers back from your store, so the tab answers in well under a second and shows usage history instead of a single point-in-time reading.

Without a backend attached, or while the backend is unreachable, the Observability tab works exactly as before: it reads the VM directly over the cloud provider’s exec channel and shows the Checked on the VM pill.

Add a backend

Settings → Integrations → Observability → Add backend. Integrations are shared by every project in the organisation, so any VM space can attach the same backend.

The paths are the stores’ own. See the Mimir HTTP API reference for /api/v1/push and /prometheus/api/v1/query_range, the Prometheus HTTP API for /api/v1/write (needs --web.enable-remote-write-receiver) and /api/v1/query_range, and the VictoriaMetrics URL examples for both.

Click Test connection before saving. The probe writes one sample to the write URL and then reads it back from the read URL:

• The write must pass; Save stays disabled until it does.
• A read that answers correctly but does not yet show the sample is reported as a note, not a failure — most stores hide the newest 30 seconds or so.
• A read that is rejected (wrong path, wrong credential) is reported as a warning so you can fix it before attaching.

Editing a saved backend re-runs the same probe, so a rotated credential is verified before it replaces the stored one.

curl -X POST https://api.zop.dev/orgs/<orgID>/integrations/test \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-victoria",
    "provider": "victoriametrics",
    "category": "observability",
    "config": "{\"signal\":\"metrics\",\"writeUrl\":\"https://metrics.example.com/api/v1/write\",\"readUrl\":\"https://metrics.example.com/api/v1/query_range\"}",
    "credential": "{\"authType\":\"basic\",\"username\":\"vmadmin\",\"password\":\"…\"}"
  }'

{
  "data": {
    "writeOk": true,
    "readOk": true,
    "readNote": "read endpoint verified; the probe sample was not visible yet (ingestion delay)"
  }
}

POST /integrations, PUT /integrations/{id} and DELETE /integrations/{id} work as for every other integration. config carries signal (always metrics today), writeUrl, readUrl and optional tenant; credential carries authType (basic, bearer or header) with username/password, token, or headerName/headerValue. The credential is never returned by any read.

Attach it to a VM space

Attaching is what installs the agent. Two entry points, same result:

• Infrastructure → your VM space → Settings → Metrics backend. Pick the backend and Save. The confirmation says exactly what will happen on the VM.
• Infrastructure → your VM space → Components → Observability agent → Install. Pick the backend in the dialog and Install.

Either way ZopDay records the attach on the space and runs the observability-agent component on the VM. The Components card shows the install in progress and then Installed · enabled; the first samples reach your store within about a minute.

• Switch to another backend by picking it and saving: the agent is re-applied with the new settings and metrics resume within a minute.
• Detach by choosing No backend: the agent is removed from the VM and the tab goes back to reading the VM directly.
• A backend that any space still uses cannot be deleted; the Disconnect dialog names the spaces to detach first.

What runs on the VM

The agent is a single pinned OpenTelemetry Collector container (otel/opentelemetry-collector-contrib), capped at 0.25 CPU and 512 MiB. Measured footprint is 120 to 200 MiB for a VM with 20 to 30 live containers. It reads the Docker socket for per-container stats and the host for CPU, memory, disk, filesystem, load and network, every 30 seconds, and pushes them with Prometheus remote write. Its configuration and a root-only environment file with the credential are written to /etc/zop-observability/ on the VM. Docker must already be present, which is true for any VM space that has deployed a service.

What the Observability tab shows

Open a VM service → Observability. With a backend attached and the agent shipping, the header shows a Metrics backend pill and a green Live · Ns ago pill that says how old the newest sample is.

• Status strip — status, replicas ready and restarts. A container that exited cleanly (exit code 0 or 143, not out-of-memory) rolls up to Scaled down; a crash stays Degraded; a VM that cannot be reached stays Unreachable.
• Usage — CPU, Memory, Network and Block I/O over 15m / 1h / 6h / 24h. The CPU chart draws the container’s --cpus cap and the Memory chart its memory limit as dashed ceilings. Gaps in collection are drawn as gaps, not bridged. A scaled-down service shows dashes and the time of its last sample.
• Replicas — one card per container: uptime, restarts, PIDs and exit code, plus CPU, memory, network and block I/O when there are no charts.

If the agent stops reporting for more than 90 seconds — the VM rebooted, the store is down, the credential was rotated without updating the integration — the tab falls back to reading the VM directly and the pill switches to Checked on the VM. It returns to the backend on its own within a minute of samples resuming.

Metrics and labels

Everything the agent ships is standard OpenTelemetry hostmetrics and docker_stats data, so you can build your own Grafana dashboards on it. The names below are what lands in a Prometheus-compatible store after the exporter’s unit suffixes.

Every series carries these labels

Container series also carry container_name (zopday-<service>-<id8>), service_id (the ZopDay service id) and container_runtime.

Host series

system_memory_usage_bytes is also the liveness signal: the tab treats the agent as up while a host sample is newer than 90 seconds.

Application metrics per service

The agent can also collect a service’s own metrics. Set configs.metrics on the deployment and redeploy:

{ "metrics": { "mode": "scrape", "port": 8000, "path": "/metrics" } }

• scrape — the collector discovers the container and scrapes http://<container>:<port><path> every 30 seconds. Every scraped series carries the same org_id, space_id, vm_uid, container_name and service_id labels, plus an up series that reads 1 while the endpoint answers and 0 when it does not.
• otlp — the container is given the collector’s OTLP endpoint on the Docker bridge (http://172.17.0.1:4318, gRPC on 4317); push from an OpenTelemetry SDK.
• off, or no metrics key — nothing is collected. mode is required whenever the key is present.

Application metrics are shipped as-is, with no name filter; the per-service switch is the gate.

From the MCP server

The same flow is available to an agent through the ZopNight MCP server. Four tools cover it, in the order you would use them:

Reading back takes two tools:

• get_live_metrics with scope: "vm" (and optionally container) returns the live container rows the Observability tab’s strip shows.
• get_vm_usage_history returns one container’s CPU, memory, network and block I/O series over 15m, 1h, 6h or 24h from the backend.

Both address the VM by resource_id. Start from get_service_infrastructure: it names the pool a VM service runs on, its target, its infrastructureOrgId, and the pool’s machines under vms. Which id to pass depends on who owns the pool:

• Your own pool. Pass the VM’s id from the ZopDay inventory, the value list_resources reports as id for the machine of that name.
• A shared pool (target: "zopcloud"). Pass the machine’s instanceId from vms, the provider’s id such as sep-vm-0. A machine listed with a name but no instanceId has no key that fits the route and cannot be read this way yet. The VM belongs to the platform org, so also pass infrastructureOrgId as resource_org_id on both reads while keeping org_id as your own org. Without it the reads see no containers.

Container names follow zopday-<service>-<first 8 characters of the service id>. A 404 from get_vm_usage_history means one of three things, in the order to check them: the VM id is wrong or not visible to your org, the container is not one of your org’s deployments on that VM, or the pool has no metrics backend attached yet.

Troubleshooting