Skip to content

docling-rs: add build-docling-rs.yml for riscv64 wheels - #4276

Draft
luhenry wants to merge 20 commits into
mainfrom
docling-rs
Draft

luhenry wants to merge 20 commits into
mainfrom
docling-rs

Conversation

@luhenry

@luhenry luhenry commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Compiles docling.rs's Rust document-conversion engine into a PyO3 abi3 extension (docling_rs._native). Upstream publishes no riscv64 wheel.

Mirrors upstream's pypi-publish.yml.

Differs from upstream

  • Built with ort-load-dynamic, like upstream's s390x wheel - pyke ships no riscv64 ONNX Runtime.
  • ML stages need a user-supplied libonnxruntime.so (ORT_DYLIB_PATH); declarative formats and the PDF text layer don't.
  • No cargo vendor step - the riscv64 container fetches crates directly.
  • Repo-root LICENSE staged beside pyproject.toml - upstream's wheels ship none.

Matrix: one cp39-abi3 wheel, built on cp312 and tested on cp312/cp313/cp314; no cp314t, as upstream ships abi3 only.

Testing

  • Adds the bindings' declarative pytest tests/ suite from upstream's README, with repo fixtures staged; upstream CI only imports the module.

License: OK

First riscv64 CI run pending.

Builds the cp39-abi3 wheel from a maturin sdist of the v2.62.0 tag, with
ort-load-dynamic (the feature upstream uses for its s390x wheel, since pyke
ships no ONNX Runtime for that target either), and runs the bindings'
declarative pytest suite against it on cp312/cp313/cp314.
luhenry added a commit that referenced this pull request Nov 2, 2026
ort-load-dynamic (CIBW_CONFIG_SETTINGS above) makes the PDF layout/
tables/OCR pipeline dlopen ONNX Runtime at runtime instead of linking
it in, matching upstream's own s390x wheel (no riscv64 pyke/ort
binary). With nothing on the dlopen path, all 8 PDF tests fail:
"ONNX Runtime library unavailable ... set ORT_DYLIB_PATH to a
libonnxruntime (1.17 or newer) built for this platform". This
registry already publishes onnxruntime for cp312/cp313/cp314 riscv64
(the exact interpreters this port tests), so install it as a test
dependency and point ORT_DYLIB_PATH at its bundled .so instead of
leaving the ML pipeline untested.
cibuildwheel evaluates CIBW_TEST_ENVIRONMENT when it creates the test
virtualenv, before CIBW_TEST_REQUIRES is installed, so importing
onnxruntime there failed with ModuleNotFoundError. Compute the path in
CIBW_TEST_COMMAND instead, once onnxruntime is installed.
In run 42497562052, the manylinux build job failed after ~1.5h with
"Failed: DID NOT RAISE KeyboardInterrupt" in
test_ctrl_c_interrupts_native_chunking (234 passed, this was the only
failure). The test fires one SIGINT 0.5s after starting to chunk a
3000-paragraph document and expects KeyboardInterrupt within 5s, but
PyChunkStream::__next__ only calls check_signals() when its 100ms poll of
the chunk channel times out with nothing queued, and the whole chunk
(~60s on this host, per the log timestamps) ran without that gap ever
appearing, so the one-shot signal was never observed.

Patch the test to re-issue the SIGINT every 0.2s until the call is
interrupted, matching the same disposition already used for kuzu's
test_connection_interrupt (patches/kuzu/0.11.3/0002-*), and drop the fixed
5s completion budget that assumed a one-shot signal lands immediately.
Wire the patch into build-docling-rs.yml: CIBW_TEST_SOURCES copies the
tests from the top-level checkout, not the sdist unpacked for the build,
so the patch applies to that checkout.
_publish-wheel.yml now requires a matrix input (main's "publish:
Validate effective wheel targets"), but this PR branch forked before
that landed and docling-rs has never been merged to main, so
build-docling-rs.yml never picked up the matching one-line update every
already-published package's workflow got. The merge ref combines the
new required-input _publish-wheel.yml with this branch's old
build-docling-rs.yml, so any run has failed startup with "Input matrix is
required, but not provided".

Pass it the same way the mass-update commit did for every other
package's publish job.
…n leak

Run 42731217360's manylinux build job failed with cibuildwheel exit code
130 (SIGINT) during pytest's own session teardown (gc.collect() in
_ensure_unconfigure), right after test_ctrl_c_interrupts_native_chunking
-- the last item in the file -- leaving it uncounted in "16 passed, 4
skipped" instead of passing.

The 2026-11-09 fix for this test's single missed SIGINT re-issues it
every 0.2s from a background thread until the call is interrupted, but
that thread checks `stop` and calls raise_signal() as two separate,
unsynchronized steps. It can commit to one more raise_signal() in the
instant before the main thread observes `stop` and returns from
timer.join(); that extra SIGINT is still pending delivery to the main
thread at that point and, with no later test in the file to catch it,
surfaced as an unhandled KeyboardInterrupt during session teardown
instead.

time.sleep() is itself interruptible, so give the pending signal a
bounded 0.2s window to land and absorb it right there in the test's own
finally block, instead of leaving it for whatever the session runs next.
… racing a sleep

The 2nd revision of this patch (absorb a trailing signal with a bounded
0.2s time.sleep() in the test's finally block) didn't fully close the
race: a real riscv64 run showed the KeyboardInterrupt surfacing instead
inside the *next* test's threading.Thread.start() call, meaning
resignal()'s trailing raise_signal() can still be delivered after that
0.2s window, since nothing bounds its actual delivery latency under
load - the window narrowed the race, it didn't eliminate it.

Install signal.SIG_IGN before setting the stop event and joining the
resignal thread, instead: any raise_signal(SIGINT) the thread calls
while SIGINT is ignored is dropped by the kernel unconditionally, not
queued for later delivery, so no trailing signal can survive
timer.join() under any timing. Restore the previous handler only once
the thread has fully exited and can no longer call raise_signal().

Verified the regenerated patch applies cleanly (git apply --check) against
the real upstream crates/docling-py/tests/test_chunking.py at the pinned
v2.62.0 tag, and that the resulting file parses as valid Python, before
committing.
…he join instead

The 2nd revision's SIG_IGN swap still failed on a real riscv64 run (run
43704902848, job 132783900926): the traceback shows a plain
KeyboardInterrupt raised from inside signal.signal(SIGINT, SIG_IGN)
itself - the signal had already been latched by CPython before SIG_IGN
could take effect. A fixed-duration mask (sleep or handler swap) can
never fully close this: the background thread's raise_signal() and the
main thread's cleanup are concurrent all the way up to the thread's
actual exit, so there is always a window.

Set `stop` and loop on timer.join(), swallowing any KeyboardInterrupt it
raises and retrying immediately. POSIX SIGINT does not queue, so each
retry absorbs at most one outstanding signal; once join() returns without
being interrupted the thread is actually dead and can raise no further
signal, so the loop is bounded with no timing assumption at all.

Verified the regenerated patch applies cleanly (git apply --check) against
the real upstream crates/docling-py/tests/test_chunking.py at the pinned
v2.62.0 tag, and that the resulting file parses as valid Python, before
committing.
The 3rd revision's retry-on-KeyboardInterrupt around timer.join() still
failed on a real riscv64 run (run 43864266189, job 133288954921): the
traceback this time is a raw KeyboardInterrupt out of threading.py,
uncaught, ending the whole pytest session. stop.set() ran before the
try/except, on the assumption that a plain attribute flip can't usefully
be "interrupted" - but CPython checks for a pending signal at any
bytecode boundary, not only inside a blocking call, so the trailing
SIGINT landed on that unguarded statement instead and escaped untouched.

Move stop.set() inside the same retried block as timer.join(), so every
statement this cleanup needs is covered by the one except clause, with
no statement left outside it for a trailing signal to hit. Event.set()
is idempotent, so calling it again on a retry is harmless.

Verified the regenerated patch applies cleanly (git apply --check) against
the real upstream crates/docling-py/tests/test_chunking.py at the pinned
v2.62.0 tag, and that the resulting file parses as valid Python, before
committing.
Real riscv64 CI (job 133351190021) shows the test suite passing 16/255
items, reaching test_abandoning_the_iterator_cancels_the_stream PASSED at
2026-11-16T16:22:31Z, then producing zero further log output for over 10
hours until the job's own 720-minute timeout silently killed it. The very
next test in file order is test_ctrl_c_interrupts_native_chunking.

The prior revision's own commit message timed this same _big_document()
chunk at "on the order of a minute" on these runners when it dropped the
upstream 5s completion budget in favor of retrying the SIGINT every 0.2s
until a channel-empty poll observes it - a retry that only helps if the
channel does go empty often enough during the run. Whether the 10-hour
stall is that same race simply losing every round on a sufficiently
slow/loaded host, or something else (a thread/channel left over from the
previous test's abandoned-iterator cleanup, say), isn't established from
the CI log alone - there's no partial output at all to tell the two apart.

Either way, nothing should silently consume the rest of the job's timeout
budget. Add patch 0002: arm a hard faulthandler watchdog, well beyond any
plausible legitimate duration for this document on this fleet, that dumps
every thread's real stack and aborts the process outright if this test is
still running when it fires - turning a silent multi-hour stall into an
immediate, diagnosable failure instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant