Project Name
Recovered 60% of Stalled S3 Storage Capacity and Automated Detection and Purge of Zombie Multipart Upload Parts
Our client is a mid-to-large enterprise operating a high-volume data ingestion pipeline that uploads large files to Amazon S3 using multipart uploads. Cloud Architects, DevOps Engineers, and Data Engineers manage pipeline throughput and S3 storage availability.
Over time, network failures, application timeouts, and process crashes left a growing number of abandoned multipart upload sessions, with successfully uploaded parts continuing to consume storage. This zombie storage eventually reached approximately 60% of the bucket’s capacity, causing new uploads to fail due to storage exhaustion and disrupting the ingestion pipeline.
The organisation needed an automated and reliable way to identify, clean up, and prevent the accumulation of abandoned multipart uploads before they impacted storage availability and pipeline operations.
- Failed Multipart Uploads Holding Reserved Storage: When an upload failed before completion, successfully uploaded parts remained stored because the application never issued the AbortMultipartUpload call to release them.
- Zombie Parts Invisible to the Application but Charged for Storage: Abandoned parts did not appear as regular S3 objects or in application-level inventories, yet continued consuming storage and incurring costs.
- 60% of S3 Storage Occupied by Non-Existent Data: Accumulated zombie parts consumed approximately 60% of total bucket capacity, leaving only 40% available for legitimate data.
- New Uploads Blocked by Storage Exhaustion: As available capacity declined, new large-file uploads began failing, stalling data ingestion and disrupting downstream data delivery.
- No Automated Mechanism to Detect or Purge Abandoned Uploads: Without an automated cleanup process, incomplete multipart uploads continued accumulating with limited visibility and no consistent remediation mechanism.
- Manual Cleanup Impractical at High Volume: Identifying and aborting incomplete uploads manually through the AWS Console or CLI was too time-consuming to keep pace with recurring upload failures.
Ksolves, an AI-first Big Data consulting services company, built an automated S3 Multipart Upload Resource Reclamation script that uses Amazon S3's ListMultipartUploads and AbortMultipartUpload APIs to identify and purge abandoned multipart upload sessions, releasing the storage held by zombie parts.
- ListMultipartUploads-Based Zombie Detection: The script uses S3's ListMultipartUploads API to identify all open multipart upload sessions, capturing key details such as upload ID, initiation time, and object key for complete visibility.
- Age-Threshold Abandonment Classification: Each session is evaluated against a configurable age threshold. Older sessions are classified as abandoned and marked for cleanup, while newer sessions remain untouched to protect legitimate uploads in progress.
- AbortMultipartUpload Programmatic Purge: For each abandoned session, the script calls S3's AbortMultipartUpload API to remove its associated parts and release the reserved storage without manual intervention.
- Scheduled Continuous Reclamation Cycle: The detection and cleanup process runs automatically at configured intervals, continuously reclaiming storage from new upload failures and preventing zombie parts from accumulating.
- Reclamation Volume Audit Logging: Each run records zombie sessions detected, storage reclaimed, upload IDs aborted, and any failures, providing visibility into cleanup activity and helping teams optimize thresholds and schedules.
Technology Stack
| Category | Technology |
|---|---|
| Infrastructure | Amazon S3 Multipart Upload API |
| Architecture | Automated Zombie Part Detection Script |
| Platform | Configurable Abandonment Age Threshold |
| DevOps | Scheduled Reclamation Run Cadence |
| Monitoring | Storage Reclamation Audit Log |
| Methodology | Prevent-Block-Through-Reclamation Design |
From 60% of S3 bucket capacity consumed by zombie parts to restored capacity, an unblocked ingestion pipeline, and continuous automated reclamation, without purchasing additional storage.
- 60% of S3 Storage Capacity Recovered on First Run: The initial run detected and aborted accumulated zombie upload sessions, releasing approximately 60% of bucket capacity and immediately restoring space for new S3 uploads.
- Data Ingestion Pipeline Upload Blocking Eliminated: Reclaiming the occupied storage restored sufficient capacity for normal ingestion, eliminating upload failures by addressing the underlying storage issue rather than expanding capacity.
- Zombie Part Re-Accumulation Prevented Through Automation: Scheduled reclamation continuously detects and purges newly abandoned uploads, preventing zombie parts from building up toward another storage crisis.
- 60% of Storage Cost Eliminated From Abandoned Data: Removing the zombie parts eliminated storage costs associated with approximately 60% of bucket capacity that provided no operational value.
- No Additional Storage Purchased: The solution restored capacity by eliminating the underlying zombie storage problem, avoiding the cost of purchasing additional S3 capacity that would not have addressed the root cause.
What appeared to be an S3 capacity problem was actually a multipart upload lifecycle problem. Abandoned upload parts had consumed 60% of the bucket, blocking new ingestion and increasing storage costs without providing any operational value. Ksolves addressed the root cause with an automated reclamation script that identifies stale multipart uploads, aborts them, and continuously cleans up new failures. The initial run recovered 60% of bucket capacity, unblocked the ingestion pipeline, and eliminated the need for additional storage investment.
Is Your S3 Storage Filling Up With Zombie Parts From Failed Uploads?
