Cross-account scheduling concerns

KEY CONCERNS:

1. CROSS-ACCOUNT DEPENDENCIES
   Workload in account-A depends on service in account-B

2. PER-ACCOUNT SCHEDULING CONSISTENCY
   Each account has its own clock; minor drift between accounts

3. SHARED RESOURCES SERVING MULTIPLE ACCOUNTS
   Scheduling shared services off affects all consuming accounts

4. TIMING / ORDERING ACROSS ACCOUNTS
   Workload A must stop BEFORE workload B starts

5. AUDIT + VISIBILITY ACROSS ACCOUNTS
   Failures might happen in account A; logs in account B

SCENARIO: app in acme-dev calls API in acme-shared-services

  If shared-services is scheduled off:
    App in dev tries to call → connection refused → app fails

  RESOLUTIONS (in order of preference):

  A) Don't schedule shared services off
     Shared infra serves many; outage cascades
     Default position: shared services always-on

  B) Coordinate scheduling
     If dev app off when shared off = no problem
     Schedule them in coordination

  C) Add fallback / queuing in dev app
     App degrades gracefully when shared unavailable
     Best for: apps that need resilience anyway

  D) Use scale-to-one for shared services
     Shared stays available; capacity reduces off-hours
     Cost savings + reliability

EACH ACCOUNT'S ZOPNIGHT INSTANCE:
  Has its own cloud-account entry
  Schedules in each account fire independently
  Cron evaluations per-account (slight clock drift between accounts)

COORDINATION OPTIONS:
  - Group spans accounts (M1.4) → single schedule applies to all
  - Per-account schedules in coordination (manually aligned)
  - Override system for special cases

DEFAULT pattern: groups span accounts; schedules are group-level

GROUP "dev-platform-eu" spans multiple accounts:

  Members:
    Resources in acme-dev (EU region)
    Resources in acme-staging (cross-account dependency for tests)

  Single schedule attached: dev-business-hours
    Stop 8 PM Mon-Fri EU
    Start 8 AM Mon-Fri EU

  EFFECT:
    All group members stop at the same time
    All start at the same time
    Cross-account coordination handled by group membership

  ENGINEER SETUP:
    Tag resources across accounts with team=platform-eu
    Group includes all matching tags
    Schedule attached to group

WHEN ACCOUNTS ARE PER-ENV:
  acme-dev, acme-staging, acme-prod (separate accounts)

  SCHEDULING:
    Dev account: schedule per resource (or all-dev group)
    Staging account: schedule per resource (or all-staging group)
    Prod account: no schedule (always-on)

  CROSS-ACCOUNT GROUP not needed (envs are accounts)

WHEN ENVS WITHIN ONE ACCOUNT (tagged):
  Single account with env=dev, env=stage, env=prod tags

  SCHEDULING:
    Group "env-dev" → schedule off-hours
    Group "env-stage" → schedule off-hours
    Group "env-prod" → no schedule

  Tags drive group membership.

RULE: shared resources rarely scheduled off

REASONS:
  Many dependents → cascading impact
  Hard to coordinate everyone off at same time
  Failure during off-period = many engineers blocked

BETTER patterns for shared services cost:

  Scale-down (not stop) during off-hours
    Database: replica reduction, not full off
    Monitoring: lower-capacity ingest, not off
    Cache: smaller capacity, not off

  Right-size for actual usage
    If shared service is over-provisioned: reduce capacity
    Quarterly review per shared service

  Centralize aggressively
    Get more from fewer shared instances (M5.4.L2)

EXCEPTIONS (when shared CAN be scheduled):
  Genuine off-hours when no one uses (e.g., 3 AM Sat)
  AND clear ownership of when to wake (on-call procedure)
  AND tested failure scenarios (graceful degradation)

SCENARIO: dev-aws-1 and dev-aws-2 both schedule off at 8 PM

  WHAT IF schedule cron evaluates at slightly different moments
  in each account?

  Reality:
    AWS account A's cron evaluates at 8:00:00.123 PM
    AWS account B's cron evaluates at 8:00:00.456 PM
    Usually < 1 second difference

  This is fine for typical scheduling.
  Coordination at second-level rarely matters.

SCENARIO: Workload A must stop BEFORE workload B starts

  Cron-level coordination isn't sufficient for strict ordering.

  Use:
    Sequenced execution (T1.M1.4.L4)
      Define ordering: stop A → wait → stop B
      ZopNight executes in order

    Or: override system
      Manual confirmation per step

SCENARIO: Cross-account audit log correlation

  Failure happens in account-A
  Logs scatter across A and dependent accounts (B)

  Use:
    Centralized log aggregation (M5.4.L2 shared services)
    Distributed tracing (Jaeger, X-Ray)
    Correlation IDs in cross-account requests

ANTI-PATTERN A: SCHEDULE CLUSTER WITH CROSS-ACCOUNT DEPENDENCY
  Schedule cluster-A off in account-1
  Cluster-B in account-2 calls cluster-A
  Cluster-B fails when cluster-A is off

  FIX:
    Schedule them together via group (M1.4)
    Or: cluster-B has retry / circuit-breaker logic
    Or: cluster-A stays on (less savings, more reliability)

ANTI-PATTERN B: DON'T SCHEDULE SHARED DATABASES
  Tempted to schedule shared DB off off-hours
  Many apps fail when DB is off; not all are tracked

  FIX:
    Scale-to-one (not stop) shared DBs
    Or: keep always-on; accept cost

ANTI-PATTERN C: CROSS-ACCOUNT SERVICES WITHOUT MONITORING
  Schedule off; failures during off-hours not noticed
  Wake on Monday to find broken state

  FIX:
    Cross-account monitoring (centralized observability)
    Alert if dependency unhealthy during expected-up periods

ANTI-PATTERN D: SCHEDULE WITHOUT GROUP
  Per-resource scheduling across accounts
  Forget to add new resource to schedule
  Coverage degrades over time

  FIX:
    Group membership via tag selector
    New resources auto-join when properly tagged

ANTI-PATTERN E: INCONSISTENT TIMEZONES
  Account A schedules in UTC
  Account B schedules in local time
  Coordination breaks at DST transitions

  FIX:
    Use UTC everywhere
    Or document the timezone per group + verify quarterly

TEST IN SANDBOX:
  Create equivalent test resources in sandbox accounts
  Schedule them
  Verify:
    Stops happen as expected
    Starts happen as expected
    Cross-account dependencies handled gracefully
    Failures (when test workload calls stopped dependency) are
      logged appropriately
    Recovery (when dependency starts) restores normal operation

TEST FAILURE MODES:
  What if account A's stop fails but B's succeeds?
  What if account B can't reach account A during dependency check?
  What if cron fires at slightly different moments?

  Document expected behavior and recovery procedure.

ROLL OUT GRADUALLY:
  Start with one team's resources (1 account first, then cross-account)
  Monitor first week
  Refine schedule + group as issues emerge
  Expand to other teams

TEAM: platform-team-eu, mid-size org

ARCHITECTURE:
  Accounts:
    acme-dev          (dev workloads, multi-team)
    acme-staging      (staging)
    acme-prod         (prod, no schedule)
    acme-shared       (shared services, no schedule)

  EU resources of platform team scattered across acme-dev + acme-staging
  Some cross-account dependencies between dev and staging

GROUP DESIGN:
  Group "platform-eu-dev":
    Tag selector: team=platform AND region=eu AND environment=dev
    Members: ~40 resources across acme-dev and (some) acme-staging

  Group "platform-eu-staging":
    Tag selector: team=platform AND region=eu AND environment=stage
    Members: ~15 resources in acme-staging

SCHEDULE DESIGN:
  Schedule "dev-business-hours-eu":
    Cron: weekdays 8 AM - 8 PM CET
    Action: start at 8 AM; stop at 8 PM
    Applies to: group "platform-eu-dev"

  Schedule "staging-eu":
    Cron: weekdays 10 AM - 6 PM CET (shorter window; less use)
    Action: start at 10 AM; stop at 6 PM
    Applies to: group "platform-eu-staging"

DEPENDENCY CHECK:
  Workloads in dev-eu group:
    DO NOT depend on staging-eu workloads
    DO depend on shared services (always-on; fine)
  Workloads in staging-eu group:
    Stand-alone for testing purposes

  Verified by:
    Dependency map review
    Test runs in sandbox

ROLLOUT:
  Week 1: Apply to 5 resources (small subset)
          Monitor for 1 week
          No failures

  Week 2: Expand to all 40 dev-eu resources
          Verify off-hours stops correctly
          Verify start-up timing correct

  Week 3: Apply staging-eu schedule
          Verify cross-account dependencies (dev → shared) intact

  Week 4: Documentation + handoff
          Team wiki updated
          On-call runbook references the schedules

OUTCOMES:
  Dev cost: -65% (was always-on; now 12h × 5 days = 36% utilization)
  Staging cost: -75% (was always-on; now 8h × 5 days = 24% utilization)
  Total savings: $1,800/mo across both envs

  Cross-account issues: 0 in first month
  Engineer feedback: positive

□ STEP 1: List your accounts
  Account 1: __________   Purpose: __________
  Account 2: __________   Purpose: __________
  Account 3: __________   Purpose: __________

□ STEP 2: Identify cross-account dependencies
  Workload A in account-X depends on service B in account-Y:
    __________ → __________
    __________ → __________

□ STEP 3: Review existing schedules
  Per-account schedules: _____
  Cross-account groups: _____ (does it exist?)

  Coordination via:
    □ Groups (preferred)
    □ Manually-aligned per-account schedules
    □ No coordination (risky)

□ STEP 4: Identify risky patterns
  Scheduled workloads with cross-account dependents: _____
  Shared resources mis-scheduled: _____

□ STEP 5: Plan improvements
  □ Create cross-account group
  □ Add monitoring for cross-account failures
  □ Test failure modes in sandbox