GCP Dataproc Cluster Autoscaling Disabled
The cluster is fixed at its configured size, so it is either too small at peak or paid for while idle.
Free to start. No card. The playground just needs your work email.
Google Cloud
Found, explained, handed over.
Where the estate departs from your policies, frameworks and tagging rules.
Detect
ZopNight checks this automatically across Google Cloud, with read-only access to the account.
Explain
Every finding says exactly what to change: Enable autoscaling on the Dataproc cluster.
Fix
The finding opens with the fix already written out, step by step, so it is one ticket, not an investigation.
- Applies to
- Dataproc Clusters on Google Cloud
- The fix, by hand
- Profile a representative workload to pick min / max worker bounds. Review historical YARN allocation metrics in Cloud Monitoring.
- Create an autoscaling policy: gcloud dataproc autoscaling-policies import ‹policy-id› --region=‹region› --source=policy.yaml (workerConfig minInstances/maxInstances/weight, secondaryWorkerConfig for preemptible).
- Attach to the cluster: gcloud dataproc clusters update ‹name› --region=‹region› --autoscaling-policy=‹policy-id›.
- Use preemptible / Spot secondary workers for the elastic tier (cheap + Dataproc handles task retries).
- Watch the Jobs tab + YARN ResourceManager. Pending containers should drop within the cooldown window.
These are the steps the finding carries in the product.
- Category
- Compliance. Configuration drift from best practice: public access, missing MFA, unencrypted storage, deprecated runtimes.
- Where it appears
- The Compliance tab of Recommendations, with every affected resource listed.
- Rule ID
RC-1211- Full reference
Related checks
See the compliance findings in your account.
Connect a read-only role and the first pass runs on your own estate. This check, and the rest of the catalogue, with it.
Prefer to talk it through first? Book 20 minutes with the team.
- $30M+annualised cloud spend under management
- 550K+resources tracked since launch
- 20-60%off the bill in the first month
- SOC 2Type II report, plus ISO 27001
Figures published on zop.dev.