# Observability Backends

> Ship host and container metrics from your VM deployment spaces to your own VictoriaMetrics, Grafana Mimir or Prometheus, and read them back on the service Observability tab.

Source: https://zop.dev/developer-docs/integrations/observability

---

An **observability backend** is a metrics store you run — VictoriaMetrics, Grafana Mimir or Prometheus — connected to ZopDay as an org-level integration. Attach it to a VM deployment space and that space's VM ships host and container metrics to it every 30 seconds. The service **Observability** tab then reads those numbers back from your store, so the tab answers in well under a second and shows usage history instead of a single point-in-time reading.

**Info**

ZopDay keeps only the backend's write URL, read URL, optional tenant and an encrypted credential. Every sample lives in your store, under your retention, queryable with your own Grafana.

Without a backend attached, or while the backend is unreachable, the Observability tab works exactly as before: it reads the VM directly over the cloud provider's exec channel and shows the **Checked on the VM** pill.

## Add a backend

**Settings → Integrations → Observability → Add backend.** Integrations are shared by every project in the organisation, so any VM space can attach the same backend.

| Field | What to enter |
|---|---|
| Name | A name your team recognises, e.g. `prod-victoria`. |
| Provider | VictoriaMetrics, Grafana Mimir or Prometheus. |
| Write URL | Where the VM pushes metrics (Prometheus remote write). `https://<host>/api/v1/write` for VictoriaMetrics and Prometheus, `https://<host>/api/v1/push` for Mimir. **https only.** |
| Read URL | The `query_range` endpoint ZopDay queries for the tab. `https://<host>/api/v1/query_range`, or `https://<host>/prometheus/api/v1/query_range` for Mimir. Must be reachable from ZopDay. **https only.** |
| Tenant | Optional. Sent as `X-Scope-OrgID` on every write and read. **Required for Mimir.** |
| Authentication | **Username** (basic auth), **Bearer token**, or a **Custom header** name and value. |

The paths are the stores' own. See the [Mimir HTTP API reference](https://grafana.com/docs/mimir/latest/references/http-api/) for `/api/v1/push` and `/prometheus/api/v1/query_range`, the [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) for `/api/v1/write` (needs `--web.enable-remote-write-receiver`) and `/api/v1/query_range`, and the [VictoriaMetrics URL examples](https://docs.victoriametrics.com/victoriametrics/url-examples/) for both.

Click **Test connection** before saving. The probe writes one sample to the write URL and then reads it back from the read URL:

- The **write must pass**; Save stays disabled until it does.
- A read that answers correctly but does not yet show the sample is reported as a note, not a failure — most stores hide the newest 30 seconds or so.
- A read that is rejected (wrong path, wrong credential) is reported as a warning so you can fix it before attaching.

Editing a saved backend re-runs the same probe, so a rotated credential is verified before it replaces the stored one.

**Warning**

The write and read URLs must resolve to public addresses. Hosts that resolve to private ranges, plain `http://`, and redirects are all rejected — the same rule the collector on the VM follows.

### POST /integrations/test

Run the write/read connection probe against a backend without saving it. Same body as POST /integrations with category=observability. (config)

```bash title="Request"
curl -X POST https://api.zop.dev/orgs/<orgID>/integrations/test \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-victoria",
    "provider": "victoriametrics",
    "category": "observability",
    "config": "{\"signal\":\"metrics\",\"writeUrl\":\"https://metrics.example.com/api/v1/write\",\"readUrl\":\"https://metrics.example.com/api/v1/query_range\"}",
    "credential": "{\"authType\":\"basic\",\"username\":\"vmadmin\",\"password\":\"…\"}"
  }'
```

```json title="Response"
{
  "data": {
    "writeOk": true,
    "readOk": true,
    "readNote": "read endpoint verified; the probe sample was not visible yet (ingestion delay)"
  }
}
```

`POST /integrations`, `PUT /integrations/{id}` and `DELETE /integrations/{id}` work as for every other [integration](https://zop.dev/docs/integrations). `config` carries `signal` (always `metrics` today), `writeUrl`, `readUrl` and optional `tenant`; `credential` carries `authType` (`basic`, `bearer` or `header`) with `username`/`password`, `token`, or `headerName`/`headerValue`. The credential is never returned by any read.

## Attach it to a VM space

Attaching is what installs the agent. Two entry points, same result:

- **Infrastructure → your VM space → Settings → Metrics backend.** Pick the backend and **Save**. The confirmation says exactly what will happen on the VM.
- **Infrastructure → your VM space → Components → Observability agent → Install.** Pick the backend in the dialog and **Install**.

Either way ZopDay records the attach on the space and runs the **observability-agent** component on the VM. The Components card shows the install in progress and then **Installed · enabled**; the first samples reach your store within about a minute.

- **Switch** to another backend by picking it and saving: the agent is re-applied with the new settings and metrics resume within a minute.
- **Detach** by choosing **No backend**: the agent is removed from the VM and the tab goes back to reading the VM directly.
- A backend that any space still uses **cannot be deleted**; the Disconnect dialog names the spaces to detach first.

### What runs on the VM

The agent is a single pinned OpenTelemetry Collector container (`otel/opentelemetry-collector-contrib`), capped at **0.25 CPU and 512 MiB**. Measured footprint is 120 to 200 MiB for a VM with 20 to 30 live containers. It reads the Docker socket for per-container stats and the host for CPU, memory, disk, filesystem, load and network, every 30 seconds, and pushes them with Prometheus remote write. Its configuration and a root-only environment file with the credential are written to `/etc/zop-observability/` on the VM. Docker must already be present, which is true for any VM space that has deployed a service.

### PATCH /spaces/{spaceID}

Attach, switch or detach: body { config: { observability: { metrics:
' | null } } }. The id must be an observability integration of this org for the metrics signal. Follow with the component install below; the UI does both on Save. (config)" />

### POST /components

Apply the agent: body { spaceId, targetType:

## What the Observability tab shows

Open a VM service → **Observability**. With a backend attached and the agent shipping, the header shows a **Metrics backend** pill and a green **Live · Ns ago** pill that says how old the newest sample is.

- **Status strip** — status, replicas ready and restarts. A container that exited cleanly (exit code 0 or 143, not out-of-memory) rolls up to **Scaled down**; a crash stays **Degraded**; a VM that cannot be reached stays **Unreachable**.
- **Usage** — CPU, Memory, Network and Block I/O over **15m / 1h / 6h / 24h**. The CPU chart draws the container's `--cpus` cap and the Memory chart its memory limit as dashed ceilings. Gaps in collection are drawn as gaps, not bridged. A scaled-down service shows dashes and the time of its last sample.
- **Replicas** — one card per container: uptime, restarts, PIDs and exit code, plus CPU, memory, network and block I/O when there are no charts.

If the agent stops reporting for more than 90 seconds — the VM rebooted, the store is down, the credential was rotated without updating the integration — the tab falls back to reading the VM directly and the pill switches to **Checked on the VM**. It returns to the backend on its own within a minute of samples resuming.

**Info**

Docker's stats stream carries no exit code, so the replica card shows a dash for it while the read comes from your store. It is filled in when the tab reads the VM directly.

### GET /resources/{resourceID}/metrics/live

Current state of the VM
 narrows to one service. source is 'backend' or 'exec'. (aggregator)" />

### GET /resources/{resourceID}/metrics/range
, range=15m|1h|6h|24h. About 120 points per series; counters are returned as bytes per second. 404 when the space has no backend. (aggregator)" />

## Metrics and labels

Everything the agent ships is standard OpenTelemetry `hostmetrics` and `docker_stats` data, so you can build your own Grafana dashboards on it. The names below are what lands in a Prometheus-compatible store after the exporter's unit suffixes.

**Every series carries these labels**

| Label | Value |
|---|---|
| `org_id` | Your ZopDay organisation id |
| `space_id` | The deployment space id |
| `vm_uid` | The cloud provider's VM id, e.g. `projects/<project>/zones/<zone>/instances/<name>` on GCP |
| `cloud` | `aws`, `gcp` or `azure` |

**Container series** also carry `container_name` (`zopday-<service>-<id8>`), `service_id` (the ZopDay service id) and `container_runtime`.

| Metric | Type | Extra labels | Used on the tab for |
|---|---|---|---|
| `container_cpu_utilization_ratio` | gauge, fraction of one core | | CPU chart and card |
| `container_cpu_limit` | gauge, cores | | CPU chart ceiling |
| `container_memory_usage_total_bytes` | gauge | | Memory chart and card |
| `container_memory_usage_limit_bytes` | gauge | | Memory chart ceiling |
| `container_network_io_usage_rx_bytes_total` / `_tx_bytes_total` | counter | `interface` | Network chart, rate per interface then summed |
| `container_blockio_io_service_bytes_recursive_total` | counter | `operation` (read/write), `device_major`, `device_minor` | Block I/O chart |
| `container_pids_count` | gauge | | PIDs on the replica card |
| `container_restarts_total` | counter | | Restarts |
| `container_uptime_seconds` | gauge | | Uptime |

**Host series**

| Metric | Type | Extra labels |
|---|---|---|
| `system_cpu_time_seconds_total` | counter | `cpu`, `state` |
| `system_cpu_load_average_1m` / `_5m` / `_15m` | gauge | |
| `system_memory_usage_bytes` | gauge | `state` (used, free, cached, buffered, …) |
| `system_network_io_bytes_total` and related | counter | `device`, `direction` |
| `system_disk_io_bytes_total` and related | counter | `device`, `direction` |
| `system_filesystem_usage_bytes` | gauge | `device`, `mountpoint`, `mode`, `state` |

`system_memory_usage_bytes` is also the liveness signal: the tab treats the agent as up while a host sample is newer than 90 seconds.

**Tip**

Add your store as a Prometheus datasource and filter on `vm_uid` or `space_id`. A `sum by (container_name) (rate(container_network_io_usage_rx_bytes_total[5m]))` panel is the same number the tab's Network chart shows.

## Application metrics per service

The agent can also collect a service's own metrics. Set `configs.metrics` on the deployment and redeploy:

```json
{ "metrics": { "mode": "scrape", "port": 8000, "path": "/metrics" } }
```

- **`scrape`** — the collector discovers the container and scrapes `http://<container>:<port><path>` every 30 seconds. Every scraped series carries the same `org_id`, `space_id`, `vm_uid`, `container_name` and `service_id` labels, plus an `up` series that reads 1 while the endpoint answers and 0 when it does not.
- **`otlp`** — the container is given the collector's OTLP endpoint on the Docker bridge (`http://172.17.0.1:4318`, gRPC on `4317`); push from an OpenTelemetry SDK.
- **`off`**, or no `metrics` key — nothing is collected. `mode` is required whenever the key is present.

Application metrics are shipped as-is, with no name filter; the per-service switch is the gate.

## From the MCP server

The same flow is available to an agent through the [ZopNight MCP server](https://zop.dev/docs/integrations/mcp-server/available-tools). Four tools cover it, in the order you would use them:

| Tool | What it does |
|---|---|
| `test_integration` | Runs the write/read probe on a backend without saving it. `provider` is `victoriametrics`, `mimir` or `prometheus`; `integration_config` and `credential` are the same objects as below. Returns `writeOk`, `readOk` and the per-direction messages. |
| `create_integration` | Connects the backend: `category: "observability"`, `integration_config: { signal: "metrics", writeUrl, readUrl, tenant? }`, `credential: { authType: "basic" \| "bearer" \| "header", … }`. The connect itself repeats the write probe and refuses a backend whose write fails. `update_integration` rotates the credential or changes the URLs; `delete_integration` disconnects it (refused while a space still uses it). |
| `set_infrastructure_metrics_backend` | Attaches the integration to a VM pool (`infrastructure_id` from `list_infrastructure`), switches it, or detaches with `detach: true`. This records the attach only. |
| `install_infrastructure_component` | With `component: "observability-agent"`, installs, re-configures or removes the collector on the pool's VMs according to the current attach. Poll `list_infrastructure_jobs` until the job settles. |

Reading back takes two tools:

- `get_live_metrics` with `scope: "vm"` (and optionally `container`) returns the live container rows the Observability tab's strip shows.
- `get_vm_usage_history` returns one container's CPU, memory, network and block I/O series over `15m`, `1h`, `6h` or `24h` from the backend.

Both address the VM by `resource_id`. Start from `get_service_infrastructure`: it names the pool a VM service runs on, its `target`, its `infrastructureOrgId`, and the pool's machines under `vms`. Which id to pass depends on who owns the pool:

- **Your own pool.** Pass the VM's id from the ZopDay inventory, the value `list_resources` reports as `id` for the machine of that name.
- **A shared pool** (`target: "zopcloud"`). Pass the machine's `instanceId` from `vms`, the provider's id such as `sep-vm-0`. A machine listed with a `name` but no `instanceId` has no key that fits the route and cannot be read this way yet. The VM belongs to the platform org, so also pass `infrastructureOrgId` as `resource_org_id` on both reads while keeping `org_id` as your own org. Without it the reads see no containers.

Container names follow `zopday-<service>-<first 8 characters of the service id>`. A `404` from `get_vm_usage_history` means one of three things, in the order to check them: the VM id is wrong or not visible to your org, the container is not one of your org's deployments on that VM, or the pool has no metrics backend attached yet.

**Info**

`credential` is redacted in the audit log and is not returned by any tool. Rotating it means sending a new `credential` to `update_integration`, which is verified against the store before it replaces the stored one.

## Troubleshooting

| Symptom | Likely cause | What to do |
|---|---|---|
| Test connection: write fails with 401 | Wrong credential or auth type for the store | Check the username/password or token; Mimir also needs the tenant. |
| Test connection: "resolves to a private address" | The store is only reachable inside your network | Expose it on a public https endpoint (a tunnel or ingress) that ZopDay and the VM can reach. |
| Tab shows **Checked on the VM** although a backend is attached | Agent not shipping: VM rebooted, store down, or credential changed | Check the Components card for the agent's state; re-save the backend to re-apply; confirm the store accepts writes. The tab recovers on its own within a minute of samples resuming. |
| Tab is slow and says **Collecting metrics…** | The read fell back to the VM's exec channel | Same as above; the exec read takes 20 to 60 seconds. |
| Charts have a gap | The agent or the store was down for that period, or the service was scaled to zero | Expected. A scaled-down service shows the time of its last sample. |
| Delete backend refused with 409 | A space still uses it | Detach in that space's Settings first; the message names the spaces. |