Skip to content

[Bug]: copyOut of a missing path fails with one of two errors at random - "path not found" or "copyOut: no metadata received" #975

Open

Description

@JeronimoColon

I have done the following

  • I have searched the existing issues
  • If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

Through the container CLI, which calls LinuxContainer.copyOut:

container run --detach --name cp-race-repro alpine sleep 600
for i in $(seq 1 30); do container cp cp-race-repro:/does-not-exist /tmp/cp-out 2>&1 | tail -1; done
container delete --force cp-race-repro

Current behavior

Most attempts report the guest's error, but some report the host's internal one instead. From 30 runs on container 1.3.0 (containerization 0.41.0), 29 were:

Error: failed to copy from container cp-race-repro (cause: "internalError: "failed to copy from container cp-race-repro (cause: "unknown: "notFound: "copy: path not found '/run/container/cp-race-repro/rootfs/does-not-exist'""")"")

and 1 was:

Error: failed to copy from container cp-race-repro (cause: "internalError: "failed to copy from container cp-race-repro (cause: "internalError: "copyOut: no metadata received"")"")

I first hit this independently while building on ContainerAPIClient, in an automated test that pins the missing-path error: it failed about 1 run in 6.

It also reproduces on main at 6adea56 with an integration test that copies a missing path 30 times on one container and requires the guest's error each time. 3 of 36 runs failed, for example:

attempt 11: copyOut reported 'internalError: "copyOut: no metadata received"' instead of the guest's not-found error

Cause, from reading LinuxContainer.copyOut (Sources/Containerization/LinuxContainer.swift; the same at 0.41.0, 0.47.0, 0.51.0, and on main at 6adea56):

  • The producer task's defer { metadataCont.finish() }, added by LinuxContainer: finish the copyOut metadata stream on every path #837 to fix the #1927 hang, finishes the metadata stream when vminitd.copy throws.
  • The guest throws copy: path not found before it sends any metadata, so the consumer task wakes to an empty stream and throws its own copyOut: no metadata received.
  • group.waitForAll() reports whichever child fails first. Once finish() runs both tasks are runnable, so the order is not fixed: the producer usually wins, the consumer sometimes does.

#799 addressed exactly this before it was closed: the producer records its error and finishes both the metadata stream and the listener before rethrowing, and the consumer reports the recorded error in preference to its own. From reading the code, that also covers a rarer case I have not reproduced: if the guest sends metadata and then fails to connect the vsock, the consumer appears to wait at listener.first with nothing to finish it.

I have a fix along the lines of #799 and will open a pull request for it, with an integration test that repeats the missing-path copy and checks that every error is the guest's.

Expected behavior

A missing source path always reports the guest's not-found error.

Environment

  • OS: macOS 27.0.1 (26A434)
  • Xcode: 27.0 (27A266a)
  • Container: container CLI 1.3.0 (build: release, commit: d6de569), containerization 0.41.0

Relevant log output

The two errors are quoted under Current behavior.

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions