Apache Flink's blob store holds unused blobs 30 minutes before cleanup

Apache Flink's BLOB store holds the binary files a running cluster cannot do without: the JARs users upload to run their jobs, oversized messages passed between tasks, and the TaskManager logs the Web UI shows on request. Before version 1.4, that store had three failure modes the community later addressed in FLIP-19. The same file could end up stored more than once. Files no longer needed by any task stayed on disk. And file contents could be modified without anything noticing.

None of those problems announces itself. A duplicated JAR wastes space in silence. An orphaned blob only matters when the disk fills up. A corrupted file shows up as a failed job, long after whatever wrote it has finished. The redesign, then, was not about throughput or latency. It was about housekeeping that had drifted.

Three components, three jobs

The new design splits the work three ways, and each piece has a narrow remit.

Component	Where it runs	Job
BlobServer	Centrally, for the whole cluster	Stores and serves blobs from a local copy plus a backup copy
BlobCache	On each TaskManager	Serves local reads, fetches missing files from BlobServer, clears its own store
BlobClient	Per request	Opens a connection to BlobServer, tracks the request, confirms delivery

BlobServer keeps two copies of everything: a local store for fast reads and a backup store for recovery. Files land under a fixed path convention, <path>/<jobId>/<BlobKey>, so a blob's job and identity are readable from its location. An upload goes to the local store first and then synchronizes to the backup, which is meant to buy durability without putting the backup write in the caller's way. A read checks local first and reaches for the backup only when the file is missing.

BlobCache runs on each TaskManager and mirrors the same path structure. It can read the central backup store but not write to it. When it needs a file it does not have, it asks BlobServer for a transfer. Each cache decides for itself when to clear files it no longer needs, which keeps cleanup decisions with the machine that owns the disk. That self-cleaning idea has a cousin in Blast Radius, which buries dead agent context and reports that nothing came back.

Reference counting and the deliberate pause before deletion

Deleting a blob the moment its last known reader finishes is how a cluster loses a file another task was about to fetch. FLIP-19 answers that with reference counting: the system tracks how many tasks currently hold a file, and deletion waits for the count to reach zero. Even at zero, the file does not disappear. It sits for a configurable interval, and a new request during that window keeps it alive.

The window has a cost. Every retained blob is disk space the cluster cannot use for anything else, and a busy cluster holds files past their useful life on purpose. That trade-off runs the other way in some systems: TEPA argues a stale memory is worse than no memory at all. The default has moved toward reclaiming that space sooner: blob.retention.interval now defaults to 30 minutes, down from one hour. The source material gives no figures on how much storage the shorter interval frees, so read the number as a setting rather than a result.

Different files get different treatment, which is why a single interval has to cover them all.

Blob type	Purpose	Lifetime
JAR files	User program code	Tied to the job; cleaned up once the job ends
RPC messages	Communication between tasks	Brief; deleted after use
Log files	System monitoring and the Web UI	Stored on demand, cleaned up after use

A log request shows the pattern in miniature. The Web UI asks for TaskManager logs, the TaskManager uploads them to BlobServer, and the UI downloads them from there and renders them. Because nobody needs the file after that, it can be cleaned up instead of kept.

Large messages follow the same route. The sender writes the payload to BlobServer, the receiver downloads it, and the reference count drops once the message has been handled. When every receiver has confirmed, the blob joins the cleanup queue and waits for its scheduled pass.

Checksums and the two-tier store: where the guarantees stop

The integrity work is the part of FLIP-19 that shows up least often in day-to-day operation, and the part that saves the most time when it does. The system verifies a checksum whenever it reads or copies a file, so accidental modification surfaces as a mismatch instead of as a job that dies an hour later with no obvious cause. Combined with the local-then-backup write order, a single lost disk no longer means a blob the cluster still needs is gone. Integrity marks come in other shapes too: Claude's incoming watermark is one, and a full rewrite erases it.

Those guarantees are worth separating from the claims around them. A checksum confirms a file has not changed since it was written; the source describes verification as protection against accidental modification and does not extend it to anything deliberate. Mirroring protects against a damaged or missing copy, but the source material does not say whether the two copies are ever compared against each other, how far the local store can run ahead of the backup before a sync completes, or what a BlobServer handover does when the local store holds files the backup has not seen.

That last gap matters because BlobServer takeover is described as a four-step handover, and the source material does not list the steps. The intent behind the redesign is easier to pin down: FLIP-19 set out to fix concurrency and cleanup problems in the original architecture and to leave room for later work, including large RPC message handling. Where the implementation draws its line between covered and not covered lives in the configuration and the code, not in an overview.