Observability Backends

An observability backend is a metrics store you run — VictoriaMetrics, Grafana Mimir or Prometheus — connected to ZopDay (the ZopDev deploy app) as an org-level integration. Attach it to a VM deployment space and that space’s VM ships host and container metrics to it every 30 seconds. The service Observability tab then reads those numbers back from your store, so the tab answers in well under a second and shows usage history instead of a single point-in-time reading.

A logs backend works the same way for container logs: a Grafana Loki, VictoriaLogs or Datadog store you run, attached to a VM space as its Logs backend. The VM ships every service container’s output to it, and the service log viewer reads it live (refreshed every 10 seconds) with time-range, search and level filters. A space can have a metrics backend, a logs backend, or both. See Logs backends.

Without a backend attached, or while the backend is unreachable, the Observability tab works exactly as before: it reads the VM directly over the cloud provider’s exec channel and shows the Checked on the VM pill.

Add a backend

Settings → Integrations → Observability → Add backend. Integrations are shared by every project in the organisation, so any VM space can attach the same backend.

The paths are the stores’ own. See the Mimir HTTP API reference for /api/v1/push and /prometheus/api/v1/query_range, the Prometheus HTTP API for /api/v1/write (needs --web.enable-remote-write-receiver) and /api/v1/query_range, and the VictoriaMetrics URL examples for both.

Click Test connection before saving. The probe writes one sample to the write URL and then reads it back from the read URL:

• The write must pass; Save stays disabled until it does.
• A read that answers correctly but does not yet show the sample is reported as a note, not a failure — most stores hide the newest 30 seconds or so.
• A read that is rejected (wrong path, wrong credential) is reported as a warning so you can fix it before attaching.

Editing a saved backend re-runs the same probe, so a rotated credential is verified before it replaces the stored one.

curl -X POST https://api.zop.dev/integrations/test \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-victoria",
    "provider": "victoriametrics",
    "category": "observability",
    "config": "{\"signal\":\"metrics\",\"writeUrl\":\"https://metrics.example.com/api/v1/write\",\"readUrl\":\"https://metrics.example.com/api/v1/query_range\"}",
    "credential": "{\"authType\":\"basic\",\"username\":\"vmadmin\",\"password\":\"…\"}"
  }'

{
  "data": {
    "writeOk": true,
    "readOk": true,
    "readNote": "read endpoint verified; the probe sample was not visible yet (ingestion delay)"
  }
}

POST /integrations, PUT /integrations/{id} and DELETE /integrations/{id} work as for every other integration. For a metrics backend config carries signal: "metrics", writeUrl, readUrl and optional tenant; credential carries authType (basic, bearer or header) with username/password, token, or headerName/headerValue. The credential is never returned by any read.

Logs backends

Settings → Integrations → Observability → Add backend, choose Logs, then the provider. config.signal is logs. Every URL is https only and must resolve to a public address, as for metrics.

The connection test writes one log line, zopday probe <id>, and reads it back. The line carries service.name and vm_uid zopday-probe, so it never mixes with a VM’s container logs. As for metrics, the write must pass, and a read that answers but does not show the line yet is only a note.

{ "signal": "logs", "writeUrl": "https://logs.example.com/otlp", "readUrl": "https://logs.example.com/loki/api/v1/query_range", "tenant": "acme" }
{ "signal": "logs", "options": { "site": "datadoghq.eu" } }

credential.authType is basic, bearer or header as for metrics, or apikey with apiKey and appKey for Datadog.

Attach it to a VM space

Attaching is what installs the agent. Two entry points, same result:

• Infrastructure → your VM space → Settings → Metrics backend or Logs backend. Pick the backend and Save; one Save covers both. The confirmation says exactly what will happen on the VM.
• Infrastructure → your VM space → Components → Observability agent → Install. Pick the backend in the dialog and Install.

Either way ZopDay records the attach on the space and runs the observability-agent component on the VM. The Components card shows the install in progress and then Installed · enabled; the first samples and log lines reach your store within about a minute.

• Switch to another backend by picking it and saving: the agent is re-applied with the new settings and metrics resume within a minute.
• Detach by choosing No backend: the agent stops shipping that signal, and is removed from the VM once no metrics or logs backend is attached and Service Alerts are off. The tab goes back to reading the VM directly.
• A backend that any space still uses cannot be deleted; the Disconnect dialog names the spaces to detach first.

What runs on the VM

The agent is a single pinned OpenTelemetry Collector container (otel/opentelemetry-collector-contrib), capped at 0.25 CPU and 512 MiB. Measured footprint is 120 to 200 MiB for a VM with 20 to 30 live containers. It reads the Docker socket for per-container stats and the host for CPU, memory, disk, filesystem, load and network, every 30 seconds, and pushes them with Prometheus remote write. Its configuration and a root-only environment file with the credential are written to /etc/zop-observability/ on the VM. Docker must already be present, which is true for any VM space that has deployed a service.

With a logs backend attached, the same agent also tails the Docker log of every running zopday-* service container and sends each line to your store, tagged with container.name and service.name (the container name), service.instance.id and vm_uid (the VM id), vm_uid_lc (the VM id in lowercase, which Datadog reads by), org_id and space_id. It starts from the moment of attaching (older lines are not replayed) and remembers its position; re-applying the agent stops it gracefully, so lines neither go missing nor repeat. Detaching the logs backend clears that position, so attaching again later starts from that moment rather than replaying the gap. A space with only a logs backend runs the agent without collecting any metrics.

What the Observability tab shows

Open a VM service → Observability. With a backend attached and the agent shipping, the header shows a Metrics backend pill and a green Live · Ns ago pill that says how old the newest sample is.

• Status strip — status, replicas ready and restarts. A container that exited cleanly (exit code 0 or 143, not out-of-memory) rolls up to Scaled down; a crash stays Degraded; a VM that cannot be reached stays Unreachable.
• Usage — CPU, Memory, Network and Block I/O over 15m / 1h / 6h / 24h. The CPU chart draws the container’s --cpus cap and the Memory chart its memory limit as dashed ceilings. Gaps in collection are drawn as gaps, not bridged. A scaled-down service shows dashes and the time of its last sample.
• Replicas — one card per container: uptime, restarts, PIDs and exit code, plus CPU, memory, network and block I/O when there are no charts.

If the agent stops reporting for more than 90 seconds — the VM rebooted, the store is down, the credential was rotated without updating the integration — the tab falls back to reading the VM directly and the pill switches to Checked on the VM. It returns to the backend on its own within a minute of samples resuming.

Viewing logs

Open a VM service’s logs with View logs. There is one live view; where its lines come from depends on the space:

• Logs backend attached — lines are read from the backend and refreshed every 10 seconds. Pick Last 15 minutes / 1 hour / 6 hours / 24 hours, search case-insensitively, filter by level, and use Show earlier logs to page further back. On a pool with several VMs, each VM is read and the lines are merged by time. This still works when the VM itself cannot be reached.
• No logs backend — lines are read straight from the VM, as before.
• A shared space you do not own — your containers’ lines are read from the space owner’s logs backend, the same way; you only ever see your own services. If your container has no saved line in the last 24 hours (for example, the pool’s agent has not been re-applied yet), the viewer reads its logs straight from the VM instead, and stays there until you reload the page.

Metrics and labels

Everything the agent ships is standard OpenTelemetry hostmetrics and docker_stats data, so you can build your own Grafana dashboards on it. The names below are what lands in a Prometheus-compatible store after the exporter’s unit suffixes.

Every series carries these labels

Container series also carry container_name (zopday-<service>-<id8>), service_id (the ZopDay service id) and container_runtime.

Host series

system_memory_usage_bytes is also the liveness signal: the tab treats the agent as up while a host sample is newer than 90 seconds.

Application metrics per service

The agent can also collect a service’s own metrics. Set configs.metrics on the deployment and redeploy:

{ "metrics": { "mode": "scrape", "port": 8000, "path": "/metrics" } }

• scrape — the collector discovers the container and scrapes http://<container>:<port><path> every 30 seconds. Every scraped series carries the same org_id, space_id, vm_uid, container_name and service_id labels, plus an up series that reads 1 while the endpoint answers and 0 when it does not.
• otlp — the container is given the collector’s OTLP endpoint on the Docker bridge (http://172.17.0.1:4318, gRPC on 4317); push from an OpenTelemetry SDK.
• off, or no metrics key — nothing is collected. mode is required whenever the key is present.

Application metrics are shipped as-is, with no name filter; the per-service switch is the gate.

From the MCP server

test_integration and create_integration cover metrics backends only; connect a logs backend in the app or with the routes above, then attach and read it with the logs tools below.

The metrics and logs flows are available to an agent through the ZopNight MCP server. These tools cover them, in the order you would use them:

Reading back takes three tools:

• get_live_metrics with scope: "vm" (and optionally container) returns the live container rows the Observability tab’s strip shows.
• get_vm_usage_history returns one container’s CPU, memory, network and block I/O series over 15m, 1h, 6h or 24h from the backend.
• get_vm_logs_history returns one container’s saved log lines from the logs backend over the same windows, newest first: limit lines per page (default 200, at most 1000), search for lines containing a text (case-insensitive, at most 200 characters), and cursor set to the previous page’s nextCursor for older lines while hasMore is true.

All three address the VM by resource_id. Start from get_service_infrastructure: it names the pool a VM service runs on, its target, its infrastructureOrgId, and the pool’s machines under vms. Which id to pass depends on who owns the pool:

• Your own pool. Pass the VM’s id from the ZopDay inventory, the value list_resources reports as id for the machine of that name.
• A shared pool (target: "zopcloud"). Pass the machine’s instanceId from vms, the provider’s id such as sep-vm-0. A machine listed with a name but no instanceId has no key that fits the route and cannot be read this way yet. The VM belongs to the platform org, so also pass infrastructureOrgId as resource_org_id on every read while keeping org_id as your own org. Without it the reads see no containers.

Container names follow zopday-<service>-<first 8 characters of the service id>. A 404 from get_vm_usage_history or get_vm_logs_history means one of three things, in the order to check them: the VM id is wrong or not visible to your org, the container is not one of your org’s deployments on that VM, or the pool has no metrics (or logs) backend attached yet. For get_vm_logs_history on a shared pool you do not own, a fourth: with no search, your container has no saved line in the last 24 hours — read its live logs. On a shared pool, pass the pool owner’s org as resource_org_id; your containers’ saved history is read from the owner’s logs backend.

Troubleshooting