You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_buffer.test_array_alignment() fails on Python 3.15.0~a6 when using mimalloc #144725
Thanks for the ping (and sorry for the slower response here). Is there a time frame you need a fix looking at for? That said, I'm not certain how well I'll be able to help here - I don't have access to i586/armv7l hardware at that level to verify any fix I write.
To me there's two things that would be nice to fix:
From the test output it's not immediately obvious which case / format string failed. One clearly did but which takes a bit of investigating and would be nice if on faillure it output more directly "this case/type failed".
Whatever is causing the failure for the particular system. Given Victor's check I suspect the particular set of OpenSUSE packages/headers/config/etc. matter. Fortunately the full build log has things like size of every type from configure, but how to resolve will probably take some sleuthing.
Aha, I reproduced the issue on Debian 32-bit using commands:
./configure CFLAGS="-O0"
make
PYTHONMALLOC=mimalloc ./python -m test test_buffer -m test_array_alignment -v
CFLAGS="-O0" is just to make the build faster.
I cannot reproduce the issue with PYTHONMALLOC=malloc nor PYTHONMALLOC=pymalloc.
It seems like mimalloc only uses an alignment on 4 bytes (32-bit) on 32-bit systems.
pymalloc uses an alignment on 8 bytes (64-bit) on 32-bit systems (extract of Include/internal/pycore_obmalloc.h):
#ifSIZEOF_VOID_P>4#defineALIGNMENT 16 /* must be 2^N */#defineALIGNMENT_SHIFT 4
#else#defineALIGNMENT 8 /* must be 2^N */#defineALIGNMENT_SHIFT 3
#endif
changed the title [-]test.test_buffer.TestBufferProtocol.test_array_alignment fails on Linux/i586 in 3.15.0~a6[/-][+]test_buffer.test_array_alignment() fails on Python 3.15.0~a6 when using mimalloc[/+]on Feb 19, 2026
By default, mimalloc uses an alignment of MI_MAX_ALIGN_SIZE which is 16 bytes by default. But memory allocations smaller than 16 bytes use a lower alignment. See microsoft/mimalloc#199 (old mimalloc issue).
The failing test uses array.array("B") and then calls a.append(0) which allocates only 4 bytes. When using mimalloc, this allocation is only aligned on 4 bytes (32-bit), whereas an alignment on 8 bytes (64-bit) is expected by the test.
Ah, I think I must have misread the C spec on expectations of malloc - I had thought (and written the test) under the assumption that the return allocation must be aligned for any type, but actually there's a caveat that it's only required to be aligned for any data type that fits into the allocation size or less.
My intention with the test is primarily to ensure that typed buffers don't have weakened alignment restrictions when producing the empty allocations, which is a convenience for interop with languages (Rust, in my case) that impose alignment requirements even on empty buffers (unlike C); the original motivation had been that a particular internal optimisation in bytearray caused the backing allocation of the empty bytearray to be an on odd address on x86-64 Linux, which was awkward when reloading empty buffers (including Numpy arrays) out of pickle v5+ streams, and then trying to interop with them in other languages.
I'd agree that the memory allocator shouldn't be changed here. The principal goal was about preventing zero-size allocation optimisations from permitting misaligned arrays. The failing test is one that's backed by a true allocation, and at that point, the required semantics should be the same as those imposed on allocators in general.
I can modify the test to ensure that the second test (the one that actually invokes the allocator) only imposes the array-item alignment constraint.
diff --git a/Lib/test/test_buffer.py b/Lib/test/test_buffer.py
index ab65a44bda6..87c981b7a55 100644
--- a/Lib/test/test_buffer.py+++ b/Lib/test/test_buffer.py@@ -4465,9 +4465,9 @@ def test_bytearray_alignment(self):
@support.cpython_only
@unittest.skipIf(_testcapi is None, "requires _testcapi")
def test_array_alignment(self):
- # gh-140557: pointer alignment of buffers including empty allocation- # should match the maximum array alignment.- align = max(struct.calcsize(fmt) for fmt in ARRAY)+ # gh-140557: pointer alignment of empty buffers should be at least+ # to `size_t`.+ align = struct.calcsize("N")
cases = [array.array(fmt) for fmt in ARRAY]
# Empty arrays
self.assertEqual(
@@ -4476,9 +4476,11 @@ def test_array_alignment(self):
)
for case in cases:
case.append(0)
- # Allocated arrays+ # Allocated arrays. The minimum alignment is now governed by regular+ # allocator rules; aligned for any type that fits in the allocation.
self.assertEqual(
- [_testcapi.buffer_pointer_as_int(case) % align for case in cases],+ [_testcapi.buffer_pointer_as_int(case) % case.itemsize+ for case in cases],
[0] * len(cases),
)
is suitable?
I can make that as a PR if we're in agreement here (though I can't easily test on a 32-bit mimalloc-based platform myself to verify).
Ah, I think I must have misread the C spec on expectations of malloc - I had thought (and written the test) under the assumption that the return allocation must be aligned for any type, but actually there's a caveat that it's only required to be aligned for any data type that fits into the allocation size or less.
I think the wording is ambiguous. On my 64-bit system, _Alignof(max_align_t) is 16 bytes. These are the minimum alignments I saw with a bunch of allocators:
the pointer returned is suitably aligned so that it may be assigned so that it may be assigned to a pointer to any type and then used to access such an object [...] in the space allocated
so yeah, I guess it's somewhat ambiguous, and I read it wrong, not helped by the macOS manpages of malloc that say it's always aligned for max_align_t. The Linux manpages for malloc (at least on my current system) specifically say it only needs to be allocated for types that fit in the space, which feels like the better choice in general (and the one that matches your table).
I've opened #145026 with the patch I posted above - I believe that's now an accurate test for the actual requirements on allocators, though given my mistake the first time around (sorry!), I would appreciate whatever checks on my work anybody can offer.
Activity
cmaloney commentedon Feb 12, 2026
Test added in gh-140557 to improve alignment interop with other languages.
cc: @jakelishman (if you want to take a look; new Python alpha release means broader testing and more platforms)
jakelishman commentedon Feb 19, 2026
Thanks for the ping (and sorry for the slower response here). Is there a time frame you need a fix looking at for? That said, I'm not certain how well I'll be able to help here - I don't have access to i586/armv7l hardware at that level to verify any fix I write.
vstinner commentedon Feb 19, 2026
I built Python in 32-bit mode on Fedora using my notes: https://vstinner.readthedocs.io/c_language.html#compile-in-32-bit-mode-on-fedora But I fail to reproduce the
test_bufferbuffer issue.I ran
./python -m test test_buffer -m test_array_alignment -F -j4for 1 minute (1576 runs): it didn't fail.cmaloney commentedon Feb 19, 2026
@jakelishman ideally before the next Alpha which is when more comprehensive testing will happen again (Alpha 7 is scheduled for 2026-03-10 from https://peps.python.org/pep-0790/)
To me there's two things that would be nice to fix:
vstinner commentedon Feb 19, 2026
Kernel:
Dependencies:
configure command:
./configure --host=i586-suse-linux --build=i586-suse-linux --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-platlibdir=lib --docdir=/usr/share/doc/packages/python --enable-ipv6 --enable-shared --with-ensurepip=no --with-system-ffi --with-system-expat --with-lto --with-ssl-default-suites=openssl --enable-optimizations --with-system-libmpdec --disable-gil --enable-loadable-sqlite-extensionsTest headers:
Test failure:
vstinner commentedon Feb 19, 2026
Aha, I reproduced the issue on Debian 32-bit using commands:
CFLAGS="-O0"is just to make the build faster.I cannot reproduce the issue with
PYTHONMALLOC=mallocnorPYTHONMALLOC=pymalloc.It seems like mimalloc only uses an alignment on 4 bytes (32-bit) on 32-bit systems.
pymalloc uses an alignment on 8 bytes (64-bit) on 32-bit systems (extract of
Include/internal/pycore_obmalloc.h):[-]test.test_buffer.TestBufferProtocol.test_array_alignment fails on Linux/i586 in 3.15.0~a6[/-][+]test_buffer.test_array_alignment() fails on Python 3.15.0~a6 when using mimalloc[/+]vstinner commentedon Feb 19, 2026
By default, mimalloc uses an alignment of
MI_MAX_ALIGN_SIZEwhich is 16 bytes by default. But memory allocations smaller than 16 bytes use a lower alignment. See microsoft/mimalloc#199 (old mimalloc issue).The failing test uses
array.array("B")and then callsa.append(0)which allocates only 4 bytes. When using mimalloc, this allocation is only aligned on 4 bytes (32-bit), whereas an alignment on 8 bytes (64-bit) is expected by the test.mimalloc provides allocation functions with an explicit alignment: https://microsoft.github.io/mimalloc/group__aligned.html The problem is that the Python C API (memory allocation functions) has no alignment parameter.
cc @colesbury
vstinner commentedon Feb 19, 2026
Quick & dirty patch fixing the issue for me on a regular Python build (with the GIL):
My hack uses mimalloc functions with an alignment of 8 bytes for allocations smaller than 8 bytes.
colesbury commentedon Feb 19, 2026
Why would we change the allocator? This seems like a problem with the test
jakelishman commentedon Feb 19, 2026
Ah, I think I must have misread the C spec on expectations of
malloc- I had thought (and written the test) under the assumption that the return allocation must be aligned for any type, but actually there's a caveat that it's only required to be aligned for any data type that fits into the allocation size or less.My intention with the test is primarily to ensure that typed buffers don't have weakened alignment restrictions when producing the empty allocations, which is a convenience for interop with languages (Rust, in my case) that impose alignment requirements even on empty buffers (unlike C); the original motivation had been that a particular internal optimisation in
bytearraycaused the backing allocation of the emptybytearrayto be an on odd address on x86-64 Linux, which was awkward when reloading empty buffers (including Numpy arrays) out of pickle v5+ streams, and then trying to interop with them in other languages.I'd agree that the memory allocator shouldn't be changed here. The principal goal was about preventing zero-size allocation optimisations from permitting misaligned arrays. The failing test is one that's backed by a true allocation, and at that point, the required semantics should be the same as those imposed on allocators in general.
I can modify the test to ensure that the second test (the one that actually invokes the allocator) only imposes the array-item alignment constraint.
vstinner commentedon Feb 19, 2026
On 32-bit platforms, the test expects an alignment on 8 bytes (64-bit):
The 8 bytes value comes from
struct.calcsize('d')(64-bit float):@jakelishman: Can we use
struct.calcsize('N')alignment (size_ttype) instead ofmax(struct.calcsize(fmt) for fmt in test.test_buffer.ARRAY)?jakelishman commentedon Feb 19, 2026
@vstinner: yeah - I think a patch like
is suitable?
I can make that as a PR if we're in agreement here (though I can't easily test on a 32-bit
mimalloc-based platform myself to verify).colesbury commentedon Feb 19, 2026
I think the wording is ambiguous. On my 64-bit system,
_Alignof(max_align_t)is 16 bytes. These are the minimum alignments I saw with a bunch of allocators:jakelishman commentedon Feb 19, 2026
The C99 spec (sec 7.20.3) says of
malloc:so yeah, I guess it's somewhat ambiguous, and I read it wrong, not helped by the macOS manpages of
mallocthat say it's always aligned formax_align_t. The Linux manpages for malloc (at least on my current system) specifically say it only needs to be allocated for types that fit in the space, which feels like the better choice in general (and the one that matches your table).jakelishman commentedon Feb 19, 2026
I've opened #145026 with the patch I posted above - I believe that's now an accurate test for the actual requirements on allocators, though given my mistake the first time around (sorry!), I would appreciate whatever checks on my work anybody can offer.
mcepl commentedon Feb 26, 2026
Just for the sake of clarity, I don’t think we use
mimallocanywhere in the SUSE universe.colesbury commentedon Feb 26, 2026
You are building with
--disable-gilso it uses the customized bundled version of mimalloc in CPython.mcepl commentedon Feb 28, 2026
I see, thank you.
gh-144725: Relax over-strict alignment test on arrays (#157530)