ZopDay VM Logs Now Ship to Your Own Observability Backend
Logs that only exist on the VM that produced them disappear with the VM. ZopDay VM spaces can now attach a logs backend that the customer owns, either Loki, VictoriaLogs or Datadog, right beside the metrics backend. You connect it in Settings → Integrations → Logs, then pick it under Space → Settings → Observability.
From there the agent on the VM ships each service’s output to that backend, and View logs on a service reads it back. The viewer refreshes every 10 seconds, keeps working while the VM is unreachable, and offers a time range, search, a level filter and Show earlier logs. A space with no logs backend keeps reading logs straight from the VM, exactly as before.
A Backend Must Accept and Return Before It Is Trusted
Saving a logs integration does more than check that a URL parses. Providers are limited to loki, victorialogs and datadog, URLs must be https, tenant names are checked against an allowed character set, and a Datadog site is checked against the sites Datadog actually runs. Then the save runs a write-and-read-back probe: one test line is written, and the same backend is asked to return it.
That is the same rule the metrics backend follows. A backend that accepts writes but cannot answer a query would look healthy in a status badge and still leave the viewer empty, so the probe tests both directions before the integration is saved.
One Collector, a Bounded Queue, a Checkpoint
On the VM, the collector tails the output files of every service container. Two settings decide how it behaves when things go wrong. Each file is checkpointed, so a collector restart resumes where it stopped instead of re-sending everything. A new container is read from its end, not its beginning, so attaching a backend does not replay old history into it.
The exporter has retry and a queue capped at 100 batches. A slow or unreachable backend therefore drops the oldest batches once that queue fills, instead of growing memory on a customer’s VM until something else gives. A golden test pins one more property: the collector configuration for a space with metrics only is byte-identical to what it was before this release, so adding logs support changes nothing for spaces that do not use it.
Reading Back Without Hammering the Backend
The viewer reads through a range endpoint that offers 15 minute, 1 hour, 6 hour and 24 hour windows, a limit of up to 1000 lines, and cursor paging for earlier lines. It sits behind the same permissions as the metrics range endpoint, so a person who can see a service’s charts is the person who can see its logs.
Three small choices keep the backend from being overloaded by a page that refreshes every 10 seconds. A first page is cached for five seconds, and concurrent identical requests are collapsed into one with singleflight. Each backend has its own query reader, for LogQL, LogsQL and Datadog search. And when the backend answers 429, the response passes through to the viewer with a retry-after value, so the page backs off instead of retrying blindly.
Shared Spaces Needed Their Own Rule
The first version of logs shipping only picked up containers carrying the owner’s label. On a shared ZopCloud space, tenants’ containers and any container deployed before the label existed shipped nothing. The follow-up ships every ZopDay container on every space and stamps each line with the customer organization read from its label. On the read side, each caller is limited to its own deployments’ containers.
| Space type | Where the viewer reads from |
|---|---|
| Space with a logs backend | The backend, through the range endpoint |
| Shared space, container has a line in the last 24 hours | The backend, scoped to the caller’s own containers |
| Shared space, container has no line in the last 24 hours | Live logs from the VM |
| Space with no logs backend | Live logs from the VM |
The 24 hour rule exists so the viewer never shows an empty page for a container that is simply new. If the backend has no history for it yet, the viewer falls back to live VM logs rather than reporting not found.
The Limits We State Up Front
Three limits are named in the release rather than left to be discovered. Log levels are not stored as a severity, so the level filter works on the text of the line. The first second or so of a brand new container’s output can be missed, because the collector starts reading from the end of the file. And Datadog search matches whole words, so a partial word will not match the way it does in the other two backends. These are real edges of the design, and knowing them is what lets an engineer decide whether the viewer or the backend’s own search is the right place for a given question.
This follows the same pattern as the VM metrics work: a space either has a backend you chose and reads from it, or it keeps the built-in behaviour.
