AWS Glue Job DPU Rightsizing
What does ZopNight detect here?
AWS Glue jobs whose peak needed executors fall below allocated capacity get a DPU rightsizing recommendation. ZopNight requires a remediable MaxCapacity class, a coverage-checked 30-day peak-executor gauge, and an aggregator-priced saving of at least $5: the delta (configured DPU minus peak-needed DPU) times the Glue DPU rate and measured DPU-hours.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-098 |
| Category | rightsizing |
| Severity | low |
| Metric | none — pure configuration read |
| Source | glue_rightsizing.go |
Where it applies
What makes a Glue job over-provisioned and remediable
All three producers landed and the rule now correctly implements rules.MetricsAware, so RC-098 emits a real recommendation when an over-provisioned, remediable Glue job carries a real aggregator-priced saving; it abstains (returns nil) on any missing input. The retired implementation fired on a configured-DPU>10 tag and priced savings off a hardcoded 30% fraction, a WRONG_BINDING_CONSTRAINT; that is gone. Firing requires all of:
- Configured DPU (remediable class). The discoverer’s glue:GetJobs enrichment (GlueProvider, glue.go) stamps configured_dpu (MaxCapacity, or NumberOfWorkers × worker-type-DPU) and dpu_remediable (the MaxCapacity class the provisioner can resize through RecommendTarget → glueMaxCapacity, resize/aws/glue_job.go; worker-type jobs fail closed). The rule gates on getMetadataFloat(“configured_dpu”)>0 and metadataBool(“dpu_remediable”)==true (see discoverer LIVE.md).
- A trusted DPU-utilization peak. numberMaxNeededExecutors (peak executors the load needed) vs numberAllExecutors (allocated) is the authoritative over-provisioning signal, and both counters come straight from the executor gauges Glue publishes to CloudWatch. The rule requires HasSufficientMaxCoverage() (MaxCoverageDays >= MinMaxCoverageDays) on the peak gauge before believing the job is over-provisioned; an untrusted peak abstains. The gauge is fetched via RequiredMetrics() (gluePeakNeededExecutorsMetric, 30-day lookback), which is required for both halves of rules.MetricsAware: it is the reason SetMetrics actually gets called by service.go’s injection loop, and it is what admits the gauge into requiredMetricNames()‘s union / the aggregator’s metric_names allow-set.
- A real aggregator-priced saving. glue-job moved off calcFree to calcGlueDPU (aggregator), which prices the job (configuredDPU × measured DPU-hours × the AWSGlue glue-dpu rate, since Glue bills a job by DPU-hour) and emits the dedicated GlueDPURightsizeSavingsUSD quantity delta (configuredDPU − peakNeededDPU) × rate × DPU-hours, where peakNeededDPU = configuredDPU × clamp(maxNeeded/all, 0..1). It is injected here via the GlueDPURightsizeSavingsAware seam (SetGlueDPURightsizeSavings, mirroring the S3 retier seam) → glueDPUSavingsMap[uid]. RC-098 emits exactly that saving (clamped to the priced cost) and abstains when it is absent or below the $5 floor, never a fraction.
The aggregator quantity delta, not a fraction
SavingsUSD = ComputedCostEntry.glueDpuRightsizeSavingsUsd (the aggregator quantity delta), clamped to the priced CurrentCostUSD. This is not a rate delta and not a fraction of the bill. See aggregator COST.md / METRICS.md for the producer.
Reducing MaxCapacity incrementally
- Open the Glue job and review the current MaxCapacity (DPU count)
- Check the Glue job run history for executor utilization
- Enable Glue job metrics and review glue.driver.ExecutorPool.numberAllExecutors vs. numberActiveExecutors
- Reduce MaxCapacity incrementally (e.g., by 25%) and compare run times
- Enable Glue auto-scaling if available for the job type to let AWS optimise DPUs dynamically
The peak-needed executors gauge
glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors (peak-need gauge that gates firing; coverage-checked), numberAllExecutors + cost/GlueElapsedTimeMs (consumed in the aggregator cost path)