Repository navigation
Windows arm64 support - #328
Merged
Merged
Conversation
relenv/build/windows.py already carried arm64 plumbing (OpenSSL VC-WIN64-ARM target, arch_to_plat/arch_to_archname mappings) that was never reachable because arches[WIN32] had arm64 commented out since 2022, pending "Python 11" support that has long since landed. Uncomment it so relenv build/create/fetch accept --arch arm64 on Windows. Add build_windows_arm64 and test_windows_arm64 jobs targeting the windows-11-arm hosted runner (native host==target, unlike the existing x86 job which cross-builds on windows-2022), plus an explicit VS ARM64 VC++ tools install step since install_vc_build.ps1 only ever requests the x86/x64 workload. Untested pending an actual CI run. Tracks saltstack#280.
MSFT_VSInstance (root/cimv2/vs) isn't registered on the windows-11-arm runner image even though VS 2022 Enterprise ships pre-installed there. The CI run against build_windows_arm64 confirmed this in practice: the script reported VS as "Missing" and spent 50+ minutes downloading and laying out a VS 2017 build-tools bundle instead of using the VS 2022 already on the box. Fall back to vswhere.exe -- present on every GitHub-hosted Windows image regardless of CIM provider support -- before concluding VS needs a from-scratch install.
setup-python has no arm64 Windows build for 3.10 ("The version '3.10'
with architecture 'arm64' was not found for Windows Enterprise") --
CPython's official arm64 Windows installers only started at 3.11. All
5 build_windows_arm64 jobs succeeded in the prior run (confirming the
native ARM64 CPython+OpenSSL build itself works); this only fixes the
verify job's own bootstrap interpreter, matching what
build_windows_arm64 already uses.
test_pip_install_cryptography and test_pip_install_idem failed on the windows arm64 build: pip fell back to compiling cryptography from source (no win_arm64 wheel exists yet -- nothing has ever shipped one for this target before), and openssl-sys couldn't find an OpenSSL to link against (OPENSSL_DIR unset, no vcpkg). relenv already builds OpenSSL from source for arm64 (update_openssl's VC-WIN64-ARM path); copy that install tree into the onedir as OpenSSL/ and archive it, then point OPENSSL_DIR at it in the two affected tests. No-op on every platform where OpenSSL comes from prebuilt binaries (prefix is never populated there) or where a wheel already exists.
Was added purely to dispatch CI runs manually while testing this branch without needing a PR. Not part of adding windows arm64 support; drop it to keep the change focused.
Closed
twangboy
approved these changes
Aug 26, 2026
Windows arm64 is a new build target (PR saltstack#328). Four verify tests break because they pin package versions that either don't have arm64 wheels on PyPI or don't compile from source under the MSVC arm64 toolchain: - test_pip_install_salt_pip_dir: Salt's transitive C-extension deps (pymssql, cryptography, cffi at the pinned old versions) have no arm64 wheels and their source builds need FreeTDS/OpenSSL/libgit2 for arm64 which is not available here. Xfail on win-arm64 matching the existing amd64 xfail. - test_install_with_target_cffi_versions: cffi 1.14.6 / 1.16.0 lack arm64 wheels (upstream arm64 support landed in cffi 1.17). Skip those two extra installs on win-arm64; the cffi_version path (1.17.1 / 2.0.0) still runs and covers the arm64-supported release. - test_install_with_target_{no_,}ignore_installed: cffi <1.17 and pygit2 <1.16 have no arm64 wheels. Pin to arm64-supported floors on win-arm64 (cffi 1.17.1/2.0.0, pygit2 1.16.0/1.19.2 by Python version) so the already satisfied: cffi and --ignore-installed flows still get exercised.
Contributor
|
@knightmare2600 I pushed some test fixes. |
Contributor
Author
|
@dwoz Do you need anything from my side...? |
Two test breakages on win-arm64 (see CI run 33614539754):
* test_install_with_target_{no_,}ignore_installed pinned
``pygit2==1.16.0`` for Python 3.11-3.13 on win-arm64. That release
has no win_arm64 wheel on PyPI (only 1.19.0+ do) so pip fell back
to a source build, which needs libgit2 headers not present on the
windows-11-arm runners -- ``fatal error C1083: Cannot open include
file: 'git2.h'``. Bump the win-arm64 pin to pygit2 1.19.2 (already
used for 3.14). Python 3.10 arm64 has no runnable combination
(pygit2's requires_python is >=3.11, so no arm64 wheel exists for
cp310) -- pytest.skip that combo.
* ruff-format wanted to collapse the multi-line ``old_cffi_supported``
ternary onto one line in test_install_with_target_cffi_versions.
Apply the formatter output so pre-commit is green.
Local pre-commit run: all hooks pass.
twangboy
approved these changes
Sep 3, 2026
Previous CI run (33751498126) failed on Verify Windows arm64 for 3.11/3.12/3.13. Logs have expired (GitHub 90-day retention) and ``gh run rerun`` reports the workflow file as broken, likely due to workflow infrastructure that has evolved on main since this branch was last synced. Pulling in current main to retrigger CI with up-to-date workflow files and test infrastructure -- including the recent saltstack#339 Windows xcs.c fix, saltstack#341 saltext-zabbix skip, and saltstack#342 CPython 3.14.10 FIPS hashlib patch -- so we can see fresh failure signatures (if any) and fix accordingly.
The windows-11-arm runner image was upgraded to VS 18 (the log shows the
bootstrapper fetching ``VisualStudio.18.Release.chman``). VS 2022 17.14+
and VS 18 moved MSBuild inside the per-install tree, so v140 PlatformTool-
sets are only populated at::
<VS_INST_LOC>\MSBuild\Microsoft\VC\<vXXX>\Platforms\<Arch>\PlatformToolsets\v140
The legacy ``${env:ProgramFiles(x86)}\MSBuild\Microsoft.Cpp\v4.0\V140``
location stays empty even when the install succeeded. Our post-install
verify was still checking only the legacy path, so the Install VS Build
Tools step failed with "Failed (v140 still not registered with MSBuild)"
despite the bootstrapper returning exit code 0 -- breaking all five
Windows arm64 native build jobs (3.10-3.14).
Extract a ``Test-V140Installed`` helper that checks both locations and
use it for the pre-install "already present" short-circuit and the
post-install verify. Legacy VS 2017/2019 images continue to work
because the first branch of the helper matches their path.
The windows-11-arm runner image was upgraded to VS Enterprise 2026 / VS 18, which removed Microsoft.VisualStudio.Component.VC.140 and Microsoft.VisualStudio.Component.Windows10SDK.19041 from its product graph. All five Python arm64 native-build jobs fail at the "Install VS Build Tools" step with "Cannot find package ... in product graph" followed by "Failed (v140 still not registered with MSBuild)". v140 cannot generate arm64 code (VS 2015 predates the arm64 toolchain) so the install was never useful on an arm64 host anyway -- Python's PCbuild falls through python.props to v143/v144, both of which support arm64 natively. Three correlated changes: - install_vc_build.ps1: short-circuit the v140 install step when $env:PROCESSOR_ARCHITECTURE == "ARM64". Leaves amd64/x86 behaviour (VS 2022 on windows-2022) untouched. - build-native-action.yml: drop the WindowsTargetPlatformVersion "10.0.19041.0" pin on the arm64 job. That SDK was paired with v140 on the amd64 job; windows-11-arm ships Win11 SDK 26100 and does not have 10.0.19041.0 available, so pinning to it would break the PCbuild MSBuild step. - windows.py: patch Python 3.10-3.13 PCbuild/python.props to add a '$(VisualStudioVersion)' == '18.0' -> v143 arm. Upstream 3.14 already carries this line; without it, BasePlatformToolset resolves to empty under VS 18 and MSBuild picks an unexpected default (or errors). Idempotent -- 3.14 is left alone by the "18.0 already present" guard.
Previous attempt (614132f) added a '$(VisualStudioVersion)' == '18.0' -> v143 arm to python.props for Python 3.10-3.13 to mirror the upstream 3.14 line. CI then surfaced the next failure: MSB8020 complains that "the build tools for Visual Studio 2022 (Platform Toolset = 'v143') cannot be found" on the windows-11-arm runner. VS Enterprise 2026 doesn't ship v143 natively, and the "legacy" v143 ARM64 compat component isn't exposed in VS 18's product graph, so installing it on-demand via the VS Installer isn't reliable either. Flip the approach: strip the 18.0 pin instead of adding it. With no match for VS 18 in python.props's BasePlatformToolset chain, MSBuild falls back to DefaultPlatformToolset from Microsoft.Cpp.Default.props, which is whatever VS 18 natively ships on the runner. The regex is a no-op on 3.10-3.13 (never had the 18.0 line) and a no-op on windows-2022 x86/amd64 (still matches '17.0' -> v143).
Second CI iteration after stripping the VS 18 -> v14x pin surfaced the next failure: MSB8020 "Platform Toolset = 'v140' cannot be found". With no 18.0 match in python.props's chain, MSBuild falls through to ``$(VCTargetsPath14) != ''`` -> v140 (VS 18 unconditionally sets that variable, but v140 isn't actually installed on the runner and can't even target arm64 anyway). Pinning to v143 fails the same way. Discover the installed PlatformToolset at build time instead of guessing. ``<VSInstallPath>\MSBuild\Microsoft\VC\*\Platforms\<Platform>\PlatformToolsets\`` lists every vXXX name MSBuild will accept; pick the lexicographically newest and pass it via ``/p:PlatformToolset=<name>``. Only applied on arm64 to avoid perturbing amd64/x86 under VS 2022.
build.bat forwards its positional arguments to msbuild.exe verbatim, but cmd.exe tokenizes those positional args on ``=`` as a delimiter. Passing ``/p:PlatformToolset=v145`` as a single argv element therefore reaches msbuild.exe as two tokens -- ``/p:PlatformToolset`` and ``v145`` -- and fails with ``MSB1008: Only one project can be specified.`` (Previous run 42224369347 confirmed discovery finds v145 under VS 18, but the build still failed at this quoting pitfall.) Set the ``PlatformToolset`` environment variable instead: MSBuild promotes every env var whose name matches a known property into the initial property set, and CPython's build.bat passes the environment through to msbuild.exe intact. python.props's existing ``<PlatformToolset Condition="'$(PlatformToolset)' == ''">`` line is a no-op once the env var is populated, so this reliably pins the discovered toolset without having to muck with batch quoting.
v145 pinning worked -- previous CI iteration got as far as compiling, but hit ``fatal error C1002: compiler is out of heap space in pass 2`` on ``unicodetype_db.h`` and other generated sources. Log confirmed HostX86\arm64\cl.exe was in use: MSBuild's default ``PreferredToolArchitecture`` is x86 for arm64 cross-compile layouts, and the 32-bit compiler caps its heap at ~2 GB. Set ``PreferredToolArchitecture=x64`` via env var so MSBuild picks the HostX64\arm64 cross-tools instead. On Windows 11 ARM64 the x64 binaries run under the OS's native emulation layer, no extra install needed.
Env-var attempt (9658d3c) didn't take: the v145 toolset is correctly selected but MSBuild still drives HostX86\arm64\cl.exe and OOMs on unicodetype_db.h / etc. with "fatal error C1002: compiler is out of heap space in pass 2". ``PreferredToolArchitecture`` from the process env gets reset by Microsoft.Cpp.Default.props's ``Condition='' == ''`` block before it ever reaches the C++ build targets. Patch python.props instead. Every PCbuild vcxproj imports python.props before Microsoft.Cpp.Default.props, so a condition-on-empty set there latches the x64 preference before the default has a chance to leave it empty. Scope the new line to ``'$(Platform)' == 'ARM64'`` so the x86 / x64 builds on windows-2022 (which have no 32-bit heap pressure and would just be slowed by x86-emulation of the x64 tools... but actually on windows-2022 they're already x64-hosted) stay untouched.
Previous commit (68d66ff) patched python.props to set ``PreferredToolArchitecture=x64`` for the ARM64 platform, but Microsoft. Cpp.Default.props still resets it to x86 after our value is applied. CI run 42322494456 confirms: v145 is correctly picked (via env-var promotion of PlatformToolset), but cl.exe is still invoked from HostX86\arm64 and OOMs on unicodetype_db.h for 3.10-3.13 with "fatal error C1002: compiler is out of heap space in pass 2". (3.14 somehow dodges the C1002 and was green on that run -- different compile flags, less heap usage.) Pass ``/p:PreferredToolArchitecture=x64`` via the MSBuild command line instead -- /p: global properties win over anything the project files set. The previous attempt at this hit MSB1008 because cmd.exe splits unquoted positional batch args on ``=``; wrap the arg in literal double quotes and shell out via a single pre-quoted string (shell=True) so the quotes survive subprocess.list2cmdline and build.bat forwards the arg to msbuild.exe intact. Only ARM64 takes this path; amd64/x86 keep the plain argv-list invocation.
The previous attempt at forcing x64-hosted cl.exe on VS 18 went nowhere: ``PreferredToolArchitecture`` in python.props is overridden by Microsoft.Cpp.Default.props, and passing it via /p: hit MSBuild's positional-arg tokenization (``=`` splits the arg). 3.14 happens to skate past the C1002 because its generated unicodetype table or compile flags land on the right side of the 2 GB cap; 3.10-3.13 keep OOMing. Rather than keep fighting the host-tool selection, patch pyproject.props to disable ``WholeProgramOptimization`` and ``LinkTimeCodeGeneration`` for the ARM64/Release configuration pair. That eliminates the pass-2 whole-program optimisation phase where HostX86\arm64\cl.exe runs the heap out, at the cost of some runtime speed (official CPython ARM64 wheels keep WPO/LTCG on; these aren't official builds). Idempotent via ``<!-- relenv-arm64-disable-wpo -->`` sentinel. Other architectures and the Debug configuration stay untouched. Also revert the shell=True + list2cmdline hack from 0364ef7: it carried ``/p:PreferredToolArchitecture=x64`` through to msbuild.exe correctly but had no effect on the host-arch selection for ARM64 cross-compile anyway.
Previous commit (b526f36) only disabled WPO+LTCG for ARM64 Release. Still OOM'd on ``unicodetype_db.h``: /O2's own register allocator pushes the 32-bit cl.exe past the 2 GB heap limit even without whole-program optimization enabled. Patch also forces ``Optimization=MinSpace`` (/O1), ``InlineFunctionExpansion= OnlyExplicitInline``, and ``IntrinsicFunctions=false`` for the ARM64 Release ItemDefinitionGroup, cutting the compile memory footprint deep enough to fit. Confirmed from the uploaded python.log on run 42328670281 that the earlier patch did apply (the ``Patched pyproject.props to disable WPO/LTCG on ARM64 Release`` info line is present), so the root cause is optimization level not just WPO.
Round 3 on the C1002 pass-2 OOM. /O1+MinSpace (907f097) was still not enough -- the 32-bit cl.exe's own code generator exhausts the 2 GB heap on ``unicodetype_db.h`` even at /O1. Drop to /Od (``Optimization= Disabled``) and turn off every remaining optimisation knob (``StringPooling``, ``FunctionLevelLinking``, ``InlineFunctionExpansion =Disabled``). Also widen the override's Condition from ``Platform='ARM64' and Configuration='Release'`` to just ``Platform='ARM64'`` in case ProjectConfiguration selection hasn't set ``$(Configuration)`` by the time pyproject.props is evaluated and the earlier two-term condition was simply being skipped. Gives up all release-mode optimisation on arm64 in exchange for a successful build; we can tighten the knobs back once a 64-bit cl.exe story works on windows-11-arm.
``test_install_with_target_no_ignore_installed`` /
``test_install_with_target_ignore_installed`` pre-install ``cffi``
and then verify that pip re-uses the already-installed copy when it
pulls in ``pygit2`` through ``--target``. For win_arm64 (3.11-3.13 and
3.14) the test picked ``pygit2==1.19.2`` -- the earliest release that
ships win_arm64 wheels -- but only sent 3.14 down the ``cffi==2.0.0``
branch. 3.11/3.12/3.13 pinned ``cffi==1.17.1``, which fails pygit2
1.19.2's ``cffi>=2.0`` METADATA requirement, so pip upgrades cffi
under the hood and the assertion ``"already satisfied: cffi" in out``
fires.
CI run 42332897154 reproduced: 3.10 skipped (no pygit2 win_arm64
wheel), 3.14 passed, 3.11/3.12/3.13 failed with::
AssertionError: assert 'already satisfied: cffi' in
'... Successfully installed cffi pygit2\r\n'
Collapse the arm64 branch to a single ``cffi==2.0.0`` pin so every
runnable Python version uses the version that matches pygit2 1.19.2's
cffi floor. Other platforms are unaffected.
Both ``build_windows_arm64`` and ``test_windows_arm64`` matrix over ``arch: [arm64]``, which GitHub appends to the job label -- producing ``Python Windows arm64 (3.12, arm64)``. Collapse to ``Python Windows`` and ``Verify Windows`` so the matrix is the only source of the arch tag. The x86/amd64 ``build_windows`` / ``test_windows`` already use bare ``Python Windows`` / ``Verify Windows`` names, so this keeps the two jobs' labels consistent.
dwoz
approved these changes
Nov 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Windows arm64 support
Resubmission of #318, which was accidentally auto-closed when my fork was deleted and has since been recloned. Same branch, same commits — nothing has changed content-wise, this is purely to restore the PR after the fork mishap.
Please See #318 for the original discussion and review context. CC:@dwoz