Snowflake Pipe Ingesting Many Small Files
Each file carries a fixed ingestion charge, so many small files cost far more than the same data delivered in fewer, larger ones.
Free to start. No card. The playground just needs your work email.
Snowflake
Found, explained, handed over.
Where the estate departs from your policies, frameworks and tagging rules.
Detect
ZopNight checks this automatically across Snowflake, with read-only access to the account.
Explain
Every finding says exactly what to change: Batch the files this pipe ingests.
Fix
The finding opens with the fix already written out, step by step, so it is one ticket, not an investigation.
- Applies to
- Snowflake Pipe on Snowflake
- The fix, by hand
- Open PIPE_USAGE_HISTORY for the resource. Verify high file count relative to bytes loaded (high per-file overhead).
- In the source system, batch files to 100 to 250 MB before pushing to the stage (the Snowpipe sweet spot).
- Evaluate whether to switch from Snowpipe to scheduled COPY on a small warehouse: for steady high-volume loads, COPY is often cheaper.
- Re-measure PIPE_USAGE_HISTORY after the batching change. Overhead credits should drop by ~10×.
These are the steps the finding carries in the product.
- Category
- Governance. Missing tags and labels that break cost attribution, such as no environment, team or cost-centre value.
- Where it appears
- The Compliance tab of Recommendations, with every affected resource listed.
- Rule ID
RC-2115- Full reference
Related checks
See the governance findings in your account.
Connect a read-only role and the first pass runs on your own estate. This check, and the rest of the catalogue, with it.
Prefer to talk it through first? Book 20 minutes with the team.
- $30M+annualised cloud spend under management
- 550K+resources tracked since launch
- 20-60%off the bill in the first month
- SOC 2Type II report, plus ISO 27001
Figures published on zop.dev.