Repository navigation
Resume blob downloads after a dropped connection - #949
Open
nandan-varma wants to merge 1 commit into
Open
nandan-varma wants to merge 1 commit into
nandan-varma wants to merge 1 commit into
Conversation
3 tasks done
dkovba
approved these changes
Oct 7, 2026
jlbauss
pushed a commit
to jlbauss/konrad
that referenced
this pull request
Oct 27, 2026
`konrad update` on a Mac failed after about ten minutes, every time, with `StreamClosed(… HTTP2ErrorCode<0x1 ProtocolError>)`. ghcr's CDN resets slow blob downloads mid-stream (reproduced with a throttled curl: three of three slow downloads cut after 4 to 9 minutes, the same PROTOCOL_ERROR), and apple/container gives up on the first cut: its registry client retries only before the response starts, and a failed pull cancels its ingest session, throwing away every layer already fetched (apple/container#2264). So on a slow link a large pull never finishes, and re-running it starts from zero. When `container image pull` fails, eng_pull now falls back to container_pull_resumable: it reads the image's manifest itself, fetches the config and layers the engine's content store lacks with `curl -C -`, resuming each cut download until the blob is whole and its digest matches, and hands them to `container image load` as an OCI archive (layers already in the store are skipped by the load as by a pull). Partial downloads stay under the state dir, so an interrupted fallback resumes on the next try. All three pull paths (first run, update, the background refresh) get it through eng_pull. This is custom code for an upstream bug, tracked in ROADMAP → Upstream workarounds: remove it once apple/containerization#949, which resumes a dropped blob download, ships in a container release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the connection drops while downloading a layer, the pull fails and all progress on that layer is lost. This is easy to hit with large images (apple/container#2264).
With this change,
fetchBlobretries after a network error and asks for the remaining bytes with aRangeheader. If the registry ignores the range and sends the whole blob again, the bytes we already have are skipped. The number of retries comes from the existingRetryOptions.Added tests that use a local server which drops the connection partway through a download. I also tried a real
container image pullthrough a proxy that cut the connection twice. It resumed and finished. Without this change it fails withstream ended at an unexpected time.Fixes apple/container#2264