Skip to content

Resume blob downloads after a dropped connection - #949

Open
nandan-varma wants to merge 1 commit into
apple:mainfrom
nandan-varma:resume-blob-download
Open

nandan-varma wants to merge 1 commit into
apple:mainfrom
nandan-varma:resume-blob-download

Conversation

@nandan-varma

Copy link
Copy Markdown

If the connection drops while downloading a layer, the pull fails and all progress on that layer is lost. This is easy to hit with large images (apple/container#2264).

With this change, fetchBlob retries after a network error and asks for the remaining bytes with a Range header. If the registry ignores the range and sends the whole blob again, the bytes we already have are skipped. The number of retries comes from the existing RetryOptions.

Added tests that use a local server which drops the connection partway through a download. I also tried a real container image pull through a proxy that cut the connection twice. It resumed and finished. Without this change it fails with stream ended at an unexpected time.

Fixes apple/container#2264

@dkovba dkovba left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution! 👍

jlbauss pushed a commit to jlbauss/konrad that referenced this pull request Oct 27, 2026
`konrad update` on a Mac failed after about ten minutes, every time,
with `StreamClosed(… HTTP2ErrorCode<0x1 ProtocolError>)`. ghcr's CDN
resets slow blob downloads mid-stream (reproduced with a throttled curl:
three of three slow downloads cut after 4 to 9 minutes, the same
PROTOCOL_ERROR), and apple/container gives up on the first cut: its
registry client retries only before the response starts, and a failed
pull cancels its ingest session, throwing away every layer already
fetched (apple/container#2264). So on a slow link a large pull never
finishes, and re-running it starts from zero.

When `container image pull` fails, eng_pull now falls back to
container_pull_resumable: it reads the image's manifest itself, fetches
the config and layers the engine's content store lacks with `curl -C -`,
resuming each cut download until the blob is whole and its digest
matches, and hands them to `container image load` as an OCI archive
(layers already in the store are skipped by the load as by a pull).
Partial downloads stay under the state dir, so an interrupted fallback
resumes on the next try. All three pull paths (first run, update, the
background refresh) get it through eng_pull.

This is custom code for an upstream bug, tracked in ROADMAP → Upstream
workarounds: remove it once apple/containerization#949, which resumes a
dropped blob download, ships in a container release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Retry on network errors before giving up

2 participants