# Observability Stack

> zop.dev deploys 5 Helm releases (Loki, Cortex, Mimir, Tempo, and OpenObserve) into a dedicated namespace, each writing to its own S3 bucket, plus Mimir basic-auth, an ingress, and PrometheusRules alerts. The buckets bill continuously and grow with retention; ingestion limits in observability_config are the throttle.

Source: https://zop.dev/integrations/provision/aws/observability-stack
Updated: 2026-08-19

---

A production platform needs long-term storage for logs, metrics, and traces. zop.dev deploys a complete S3-backed observability backend alongside every cluster: Loki for logs, Cortex and Mimir for metrics, Tempo for traces, and OpenObserve.

## S3-backed Loki, Cortex, Mimir, Tempo, and OpenObserve

Five S3 buckets (loki, cortex, mimir, tempo, openobserve), five Helm releases, a dedicated namespace, ingress, Mimir basic-auth, and an alerts module with PrometheusRules for cortex, loki, mimir, and tempo.

## observability_config per component settings

```text
Per-component configuration through observability_config: replicas, autoscaling, persistence, resource limits, remote_write, and ingestion limits for prometheus, grafana, kubernetes-event-exporter, loki, cortex, mimir, tempo, and openobserve.
```

## OpenObserve is the AWS-only addition

AWS is the only provider whose stack includes OpenObserve alongside the four Grafana-ecosystem backends.

## Storage is the stack's center of gravity

Each of the 5 components writes to its own S3 bucket, and those buckets bill per GB-month,
continuously, from day one. Telemetry only accumulates. Logs, metrics, and traces arrive every
second the cluster runs, so bucket growth is linear with retention. The ingestion limits and
per-component retention in `observability_config` are the levers; without deliberate settings,
the buckets grow until someone notices the S3 line item and has to archaeology their way back
to the cause.

## The compute hides inside the cluster

The 5 Helm releases run in a dedicated namespace on your own nodes. Loki ingesters, the Cortex
and Mimir metric stores, and Tempo are not free to operate: their replicas, autoscaling,
persistence, and resource limits (all settable per component) translate directly into node
capacity the cluster must hold. On an observability-heavy setup, a sizeable share of "cluster
cost" is the observability stack itself, invisible because it bills as EC2 rather than as a
monitoring line item.

## The stack watches itself from day one

An alerts module ships PrometheusRules for cortex, loki, mimir, and tempo, so the pipeline that
would normally tell you about failures also tells you about its own. The practical value: an
ingester falling over pages someone before dashboards silently go dark and a week of telemetry
quietly disappears.

## The ingress is an exposure decision

The dedicated namespace ships with an ingress, which makes query endpoints reachable from
outside the cluster, and Mimir sits behind basic-auth. Treat that hostname as production
surface. It fronts your logs and metrics, which are exactly the data an attacker wants first.
Rotate the basic-auth credential like any other, and keep the ingress scoped to networks that
need it.