Repository navigation
Description
Feature or enhancement
Proposal:
The current makefile rules for profiled guided optimisation don't support cross at all:
Lines 691 to 692 in fb4cb7c
| @ # FIXME: can't run for a cross build | |
| $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) |
This is understandable, as the point is to run the target code and on the host this isn't generally possible.
However, qemu-user does let us run target code on the build host with only a little fuss and various build tools allow the use of this to run target binaries in limited cases, such as when probing capabilities or running tests. For example, Meson's cross files support an exe_wrapper script (https://mesonbuild.com/Cross-compilation.html) and CMake has a CMAKE_CROSSCOMPILING_EMULATOR variable (https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING_EMULATOR.html).
I propose adding something similar for the PGO execution. Strawman proposal:
- Add
--with-cross-runnertoconfigureto let the user specify a wrapper script that can be used to run target binaries on the build host. - Wrap the execution of the just built
python3when generating the PGO training data with this script
Prior art to demonstrate that this is at least feasible can be found in the Yocto Project at https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3/0001-Makefile.pre-use-qemu-wrapper-when-gathering-profile.patch. We generate the pgo-wrapper script (which simply calls qemu-user with arguments) during the build.
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Activity
diegorusso commentedon Dec 14, 2023
Thanks Ross, I can put this in my todo list.
rossburton commentedon Dec 14, 2023
The elephant in the room, so to speak, is that
qemu-usercan'tfork(). This means any tests that try to fork will fail, but that's worked around by limiting the tests that are used for the guiding or ignoring failures.zanieb commentedon Dec 29, 2024
As a related note, in
python-build-standalonewe drop the clear ofRUNSHAREDto run PGO in some of our cross-compiles since we can run the cross-compiled Python binary (without any wrapping) in some contexts, e.g., macOS x86_64 targets run transparently via Rosetta on an aarch64 host and i686 targets can run on a x86_64 host.pythongh-113123: Respect `HOSTRUNNER` for PGO during cross-builds