# Provisioning VMs

> Production-shape multi-service VM environments; private subnet with NAT, optional cloud LB, IAM-gated exec, no SSH.

Source: https://zop.dev/docs/zopday/provisioning/vms

---

ZopDay provisions VMs for the case where Kubernetes is the wrong abstraction; multi-service VMs running Docker, legacy workloads that aren't containerised, or environments where you want a single shared host with NGINX routing between services. The default shape is production-ready: private subnet, NAT egress, no public SSH, IAM-gated exec.

## When to provision VMs (vs. clusters)

| Situation | Pick |
|---|---|
| Multiple services, container-native, need orchestration | Cluster (EKS / GKE / AKS); see [Clusters](https://zop.dev/docs/zopday/provisioning/clusters) |
| Legacy workload that runs as a process, not a container | VM |
| Multi-service host with NGINX routing between Docker containers | VM (the default cloud-init configures this) |
| Lightweight environment for testing; single host, single port | VM |
| Burst traffic with horizontal scaling | VM in ASG / MIG mode |

VMs are not a "lite cluster"; they're a different abstraction for genuinely different workloads.

## Pool modes

ZopDay supports two VM pool modes:

**Fixed**

    N specific VMs via direct `RunInstances` (AWS) or `compute.instances.insert` (GCP). Each VM has a stable identity. Best for stateful workloads or specific-IP requirements.

**ASG (Auto Scaling)**

    AWS Auto Scaling Group or GCP Managed Instance Group. Min / max / desired capacity, scaling policies, instance lifecycle managed by the cloud autoscaler. Best for stateless web servers.

Azure VMSS provisioning is in preview / not yet GA; when provisioning a VM job with provider=Azure, the wizard surfaces a clean stub return (`Invalid`) and points you at when it will be available. The Azure exec channel is likewise gated on Azure GA.

## What the wizard provisions

The default VM environment lands:

| Resource | Notes |
|---|---|
| **VM(s)** | In a **private subnet** of the chosen VPC. No public IP. |
| **VM security group / firewall** | Inbound: only from within the VPC + LB security group. **No public SSH; port 22 is not publicly reachable.** |
| **Launch template / instance template** | Stores the AMI / image, instance type, IAM role, cloud-init userdata, tags |
| **Cloud LB** (optional) | TLS-terminated. Cert source = `CertSource{AWSACM, GCPManaged, AzureKeyVault}` |
| **IAM role for the VM** | Includes the exec channel role (SSM / OS Config); see below |
| **Tag set** | `zop:orgId`, `zop:jobId`, `zop:role`, `zop:managedBy` on every resource |

## Pipeline steps

The VM pipeline reuses shared steps with the cluster pipeline (`stepCreateNetwork`, `stepCreateIAM`, `stepRegisterSpace`) via `j.ResourceType` branching. The full sequence:

```
stepValidate → stepCreateNetwork → stepCreateIAM → stepCreateLaunchTemplate
            → stepCreateVMs → [LB sub-steps when LB attached]
            → stepRegisterSpace → stepFinalize
```

For a VM job **without** a load balancer: **7 steps total**.
For a VM job **with** a load balancer: **11 steps total** (the LB sub-steps cover the cloud-side load balancer creation, target group / backend pool, listener / forwarding rule, and DNS record).

## Cloud-init bootstrap

Every VM ZopDay provisions runs the same cloud-init userdata across clouds:

1. Install Docker
2. Create an `nginx-on-zop-net` Docker bridge network
3. Pre-install NGINX as a Docker container on the bridge for multi-service routing on a single VM
4. Write a sentinel file once everything is wired

The sentinel file is what ZopDay polls during `stepCreateVMs` to confirm the VM is actually ready to accept workloads. Until the sentinel exists, the VM is in `bootstrap` state and the job waits.

If you don't want NGINX pre-installed, the wizard's "Bare metal" mode skips that part of the cloud-init. You still get Docker and the network setup; NGINX is your problem.

This NGINX baseline is for VMs provisioned as standalone infrastructure; when a VM is used as a ZopDay deploy substrate, Caddy handles routing instead (see [Deploy to a VM](https://zop.dev/docs/zopday/deploy/vm-substrate)).

## Exec channel; no public SSH

ZopDay never provisions public SSH access. Port 22 is not publicly reachable, the VM has no public IP, and ZopDay does not put your public key in `authorized_keys`. Interactive access is only via the cloud's IAM-gated exec channel:

| Cloud | Channel | Command |
|---|---|---|
| **AWS** | SSM Session Manager | `aws ssm start-session --target i-xxxx` |
| **GCP** | IAP tunnel | `gcloud compute os-config-instance-os-policy-assignment-reports describe` (for managed inventory) or `gcloud compute ssh --tunnel-through-iap` for interactive |
| **Azure** | Azure Run Command (in preview / not yet GA) | `az vm run-command invoke` |

Exec is IAM-gated end-to-end; your AWS/GCP/Azure identity gates whether you can even start a session. Because there is no public SSH and no port 22 exposed to the internet, there is no public brute-force surface; access flows only through the cloud provider's IAM-gated channel.

The exec channel for each cloud is set via the `ExecMode` field: `SSM` (AWS), `OSConfig` (GCP), `AzureRunCmd` (Azure).

## Load balancer (optional)

When you toggle "with load balancer" in the wizard:

| Cloud | LB type | TLS termination |
|---|---|---|
| **AWS** | Application Load Balancer | At the LB via ACM certificate |
| **GCP** | HTTPS Load Balancer | At the LB via Google-managed certificate |
| **Azure** | Application Gateway | At the gateway via Key Vault certificate |

Health checks default to `GET /healthz` on port 80; override in the wizard if your service uses a different path. If the LB reports the target unhealthy, a mismatched health path or port is the most common cause; correct it by overriding the check in the wizard (or on the load balancer after provisioning).

### LB attach / detach after provisioning

LBs can be attached or detached after the VM job lands without re-provisioning the VMs. Use the existing PATCH route with the typed `vmOp` discriminator:

```
PATCH /provisioning-jobs/{id}
{ "vmOp": "attach_lb" }
```

…or:

```
PATCH /provisioning-jobs/{id}
{ "vmOp": "detach_lb" }
```

No new route surface; the PATCH route handles VM ops alongside other partial updates.

## VM lifecycle state

ZopDay tracks per-VM lifecycle state on `InfraConfig` inside the job's `config` JSON. Fields include:

| Field | What it carries |
|---|---|
| `VMs[]` | Per-instance IDs, IPs, lifecycle state |
| `VMSecurityGroupID` | The security group / firewall rule attached to the VMs |
| `LoadBalancerID` | The LB if attached |
| `ScalingGroupID` | ASG / MIG ID in ASG mode |
| `LaunchTemplateID` | The current launch template version |
| `ExistingResourceGroup` | (Azure) the resource group the VMs landed in; preserved if pre-existing |
| `BootstrapNginxReady` | Has the cloud-init NGINX bootstrap completed? |

No new tables. Adding a new field to track is a struct extension, not a migration.

## Updating VMs

| Change | What happens |
|---|---|
| `min/max/desired` in ASG mode | Patched on the ASG. Existing instances stay; new ones come from the launch template. |
| Cloud-init userdata change | New launch template version. Existing instances **do not auto-rotate**: terminate them explicitly to get the new userdata, or trigger a rolling instance refresh. |
| AMI / image change | Same as userdata; new launch template version, manual rotation. |
| Security group change | Applied immediately to all instances (security groups are mutable in-place on AWS / GCP). |

Rolling instance refresh on ASG: `aws autoscaling start-instance-refresh` (AWS) or recreating the MIG (GCP). ZopDay does not run rolling refresh automatically; that's a workload-shape decision.

## Cost notes

VM cost = instance hourly × count × hours + EBS / persistent disk storage + LB hourly (if attached) + NAT data processing.

For dev/staging environments, attach a [ZopNight schedule](https://zop.dev/docs/zopnight/concepts/scheduling) to the VM pool to stop instances on weeknights and weekends. The savings on a 10-instance dev pool with weekday-only hours is typically 60–70% vs 24×7.

## Teardown

`DELETE /provisioning-jobs/{id}` on a VM job:

1. Detaches the LB (if attached)
2. Terminates the VMs (Fixed mode) or sets ASG/MIG `min=max=desired=0` then deletes (ASG mode)
3. Deletes the launch template
4. Deletes the security group
5. Tears down the IAM role
6. Tears down networking (if ZopDay created the VPC; existing VPCs are not touched)

VM terminations are typically 1–2 minutes. ASG/MIG deletion can take 5–10 minutes depending on instance count.

## What ZopDay does NOT do on VMs

**Note**

**VMs can also be a deploy target now.** Beyond provisioning the VM as infrastructure, ZopDay's deployer can ship a container straight to a Linux VM (Docker + Caddy, no Kubernetes); see [Deploy to a VM](https://zop.dev/docs/zopday/deploy/vm-substrate). This page covers the *provisioning* of production-shape VM environments; the deploy substrate is a separate capability on top.

| Out of scope | Why |
|---|---|
| Configuration management beyond cloud-init bootstrap | Use Ansible / Chef / SaltStack / Puppet on top of ZopDay's VMs. ZopDay only sets up the Docker baseline. |
| Patch management | Use AWS Systems Manager Patch Manager / OS Config / Update Management; these tie into the exec channel ZopDay enables. |
| SSH keys | None ever provisioned. Use the exec channel. |