Cross-cluster orchestration with groups

Outcome

By the end of this lesson, you will be able to group K8s namespaces across multiple clusters, apply a single schedule across cluster boundaries, and choose the right grouping pattern (global / per-region / workload-based).


Prerequisites	M2.6.L1 - L5 · T1.M1.4 (resource groups)
Time	9 minutes

1. Concept

Most companies of any size run several Kubernetes clusters, usually one per region or one per team.

A group does not have to stay inside one of them. Put namespaces from four clusters into one group, attach one schedule, and you are done. (The namespace is the schedulable K8s unit; every workload inside it follows.)

THE CROSS-CLUSTER GROUP PATTERN:
  Group spans cluster boundaries
  Schedule applies to all members
  Per-cluster K8s API calls happen independently
  Failures in one cluster don't block others

Going from one region to four adds members to a group. It does not add schedules, and it does not add anything for you to keep in step.

Example structure

ORGANIZATION
  Cluster A (us-east-1):    namespace payments-dev (payment-api, order-svc)
  Cluster B (eu-west-1):    namespace payments-dev (regional copy)
  Cluster C (ap-south-1):   namespace payments-dev (regional copy)

GROUP: "global-payments-dev"
  Members: payments-dev namespace in all 3 clusters (3 resources, same name)

SCHEDULE: business-hours-global
  Cron: 0 8 * * 1-5 start
        0 20 * * 1-5 stop
  Attached to group "global-payments-dev"

RESULT:
  At 8 AM in the schedule's timezone (one IANA zone per schedule):
    payments-dev starts in each cluster
  At 8 PM:
    payments-dev stops in each cluster
  All three regional copies follow the same schedule, at the same moment

Cross-cluster grouping lets a single attachment manage the whole footprint.

Implementation behavior

WHEN SCHEDULE FIRES:

1. Each cluster's K8s API is accessed separately
2. ZopNight knows which cluster each workload lives in
3. The group's schedule fires once, in the schedule's single timezone
4. Each cluster's API call is made independently
5. If one cluster's scale fails, the others continue (idempotent)
6. Failure on one cluster is reported in the action log

FAILURE HANDLING:
  Group spanning 3 clusters; 1 cluster fails
  → 2 succeed; 1 fails (cleanly reported)
  → Retry manually or at the next scheduled run
    (the action is idempotent, so a repeat is safe)
  → Customer sees the per-cluster failure in audit log

The independent-per-cluster execution handles failures gracefully.

Three multi-cluster scheduling patterns

PATTERN A: SINGLE GLOBAL SCHEDULE
  All payments-dev namespaces (3 clusters) in one group
  One schedule applied to the group
  Same UTC time for all clusters

  WORKS WHEN:
    Workloads are functionally identical
    Same business hours work across regions
    Customer doesn't need timezone awareness

  TIMING:
    All clusters scale at the same UTC moment
    Variability: ~1-2 seconds between clusters

PATTERN B: PER-REGION GROUPS
  Group "payments-dev-us": us-east-1 payments-dev namespace (1 resource)
  Group "payments-dev-eu": eu-west-1 payments-dev namespace (1 resource)
  Group "payments-dev-asia": ap-south-1 payments-dev namespace (1 resource)
  Each group has its own schedule (different timezones)

  WORKS WHEN:
    Different regions have different business hours
    Multi-region team availability matters

  TIMING:
    Each region scales per its local business hours
    Independent timing per region

PATTERN C: WORKLOAD-BASED GROUPING
  One group "non-prod-stack" spans clusters: the staging database
  namespaces AND the staging app namespaces
  Sequenced execution inside that group: apps stop before
  databases; databases start before apps (sequencing works
  within a group, not across groups)

  WORKS WHEN:
    Service dependencies span clusters
    Coordinated start/stop matters

  TIMING:
    Workload-type matters more than region
    Sequenced execution per workload type

Pattern A is simplest. Pattern B handles timezones cleanly. Pattern C handles dependencies cleanly.

When cross-cluster doesn’t work

SCENARIO                                       ISSUE
─────────────────────────────────────────────────────────
Workloads have different names across clusters Group requires resource UID
                                                Not name; UID per cluster differs
                                                Group needs explicit member adds

Different timezones, want different             Schedule cron is per-schedule
schedules per region                            Not per-resource
                                                Use separate groups per region

Workloads have different configurations          Group is a set of members, not a config-
across clusters (replicas, etc.)                 alignment tool
                                                Each workload scales per its own
                                                config independently

Cross-region dependencies                        Service-A in cluster-A depends
                                                on service-B in cluster-B
                                                Scheduling them via group works
                                                but timing may be off
                                                Use sequenced execution

In these cases, prefer multiple groups (one per region or type) over one global group.

Multi-cluster permissions

ZopNight needs IAM credentials for each cluster’s K8s API:

AWS (EKS):
  IAM with eks:DescribeCluster (to find the cluster endpoint)
  Kubernetes API calls use a short-lived token signed by the
  connected IAM identity (no IRSA role); that identity needs
  Kubernetes permissions on each cluster to patch workloads

GCP (GKE):
  GKE Service Account with cluster read/update permissions

Azure (AKS):
  AKS service principal with cluster read/update permissions

PER-CLUSTER SCOPING:
  Permissions scoped per cluster
  ZopNight cannot modify a cluster it doesn't have credentials for

ADDING A CLUSTER:
  Provide credentials; cluster appears in discovery
  Workloads discovered
  Now eligible for cross-cluster groups

The permissions are explicit per cluster. No silent cross-cluster access.

Keeping groups current

Group membership is explicit. ZopNight never adds or removes a member on its own, whatever tags, labels or names a namespace carries.

WHEN A NEW NAMESPACE APPEARS:
  Discovery finds it within the next cycle
  It is NOT added to any group
  Add it to the right group yourself (it then follows the group's schedule)

WHEN A NAMESPACE IS RETIRED:
  Remove it from the group yourself

EXCLUSIVE MEMBERSHIP:
  A resource belongs to at most one group
  One scheduling source per resource
  Move it between groups by removing it, then adding it

Explicit membership means a group only ever schedules what someone chose to put in it. Make “add the namespace to its group” part of the checklist for standing up a new environment.

Audit + observability across clusters

PER-CLUSTER AUDIT trail:
  Each cluster's K8s API call logged
  Cluster ID + workload + action + result

GROUP-LEVEL summary:
  Group: 3 workloads across 3 clusters
  Schedule cron fired at 2026-05-21T08:00:00Z
  Results: 3 of 3 succeeded
  Detail per cluster available

TROUBLESHOOTING:
  "Why didn't payment-api start in eu-west-1?"
  → Per-cluster audit shows the specific failure
  → Cluster credentials? Network? Workload state?

The audit trail makes cross-cluster debugging tractable.

Failure recovery

PARTIAL FAILURES are normal at scale:

  Group spans 10 clusters
  Schedule fires
  9 clusters: success
  1 cluster: fails (API rate-limited or transient)

RECOVERY OPTIONS:
  Next scheduled run: the schedule fires again next cycle
                   and acts on every cluster, including the
                   one that failed (no automatic retry in
                   between)

  Manual retry: operator starts or stops that
                cluster's workloads again
                Immediate

  Investigation: if persistent failure
                Per-cluster diagnosis
                May require cluster-specific fix

The architecture handles partial failures gracefully.

2. Demo

A team’s multi-cluster scheduling rollout:

ESTATE: 6 clusters across 3 regions, ~280 non-prod workloads

STEP 1: IDENTIFY GROUP STRUCTURE:
  Pattern B (per-region) chosen
  Reasoning: clean timezone handling

  Groups planned:
    payment-api-us
    payment-api-eu
    payment-api-asia
    Similar groups for other services

STEP 2: POPULATE GROUPS:
  payment-api-us: add the payment-api namespaces in us-east-1 clusters
                  (across 2 clusters in that region)
  payment-api-eu: add the payment-api namespaces in eu clusters
  payment-api-asia: same for asia

  Use the region and name filters to find each namespace,
  then add it to its group explicitly

STEP 3: ATTACH SCHEDULES:
  business-hours-us schedule on payment-api-us
    Timezone: America/New_York

  business-hours-eu schedule on payment-api-eu
    Timezone: Europe/London

  business-hours-asia schedule on payment-api-asia
    Timezone: Asia/Kolkata

EXECUTION:
  Each region's payment-apis follow their region's business hours
  No timezone mishaps
  Each cluster's payment-api scales independently in its region

RESULTS (1 month in):
  Aggregate savings: ~$8,200/mo across 6 clusters' non-prod workloads
  Average scaling time: <30 seconds per cluster
  Failure rate: <0.1% (retried manually or at the next scheduled run)
  Incidents: 0
  Engineer satisfaction: high

Pattern B (per-region grouping) handles the multi-region complexity cleanly.

3. Hands-on (5 min)

Design cross-cluster scheduling for your estate:

□ STEP 1: Inventory
  Clusters: _____
  Regions: _____
  Major workload types: __________

□ STEP 2: Pick pattern
  □ A: Single global group (simplest)
  □ B: Per-region groups (timezone handling)
  □ C: Workload-based (dependency handling)

□ STEP 3: Design groups
  Group 1: __________: Members: __________
  Group 2: __________: Members: __________

□ STEP 4: Plan schedules
  Each group's schedule + timezone

□ STEP 5: Verify permissions
  Each cluster has correct IAM? □ Yes □ No
  Test cluster access from ZopNight: __________

A 30-minute design exercise. Per-region pattern often the right starting point.

4. Knowledge check

Q1

A group spans 3 clusters. One cluster’s K8s API fails during a scheduled stop. What happens:

A. All clusters fail (transactional)
B. The 2 successful clusters proceed; the failing cluster’s action is logged
C. The action is queued until that cluster recovers again fully later
D. Discovery stops

Show answer

Correct: B. Each cluster is called independently, so one failure does not cascade, and the failed cluster can be retried on its own.

Q2

For workloads across regions in 3 different timezones, the recommended pattern is:

A. One single global group with a single UTC schedule
B. Per-cluster groups
C. No grouping
D. Per-region groups (each with its own schedule + timezone)

Show answer

Correct: D. Pattern B handles timezones cleanly; each region’s workloads follow their local business hours. Pattern A would force all regions onto one schedule timezone, which doesn’t match local engineer availability.

Q3

A new staging namespace appears in a cluster. How does it join the existing group?

A. You add it to the group yourself; membership never changes on its own
B. Discovery adds it on the next sync if its tags match the group’s
C. It joins automatically when its name shares the group’s name prefix
D. It joins once the group’s schedule next fires a start or a stop

Show answer

Correct: A. Group membership is explicit. Discovery finds the new namespace, but it joins a group only when someone adds it. Tags, labels, names and schedule runs never change membership.

5. Apply

Multi-cluster groups are created the same way as single-cluster ones: filter by region or name to find the namespaces, then add them.

For your team: design groups intentionally; per-region is usually right for multi-region. When a new environment comes up, add its namespace to the right group as part of the rollout.

Module quiz

Complete M2.6 → 10-question module quiz.

Outcome

1. Concept

Example structure

Implementation behavior

Three multi-cluster scheduling patterns

When cross-cluster doesn’t work

Multi-cluster permissions

Keeping groups current

Audit + observability across clusters

Failure recovery

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Module quiz

Related lessons

Start with the bill.