Repository navigation
Closed
Labels
testsTests in the Lib/test dirTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Buildbot https://buildbot.python.org/all/#/builders/435 has been failing test_gdb with errors like these:
======================================================================
FAIL: test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction) [_testcapi.meth_varargs]
Verify that "py-bt" displays invocations of PyCFunction instances
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb.py", line 944, in test_pycfunction
self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_varargs' not found in 'Breakpoint 1 (meth_varargs) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n2307\t return Py_BuildValue("NO", _null_to_none(self), args);\n#0 meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n#1 PyModuleDef_Type ()\nBacktrace stopped: frame did not save the PC\nUnable to locate python frame\n'
It last succeeded on commit 373bca0 and first failed on fcd5fb4, suggesting that the culprit is one of these commits:
- fcd5fb4 Comprehension inlining: Bug if comprehension contains a lambda #104357: fix inlined comprehensions that close over iteration var (gh-104357: fix inlined comprehensions that close over iteration var #104368)
- 94f30c7 [pathlib] Option so that OSError does not block glob in pathlib library #90208: Suppress OSError exceptions from
pathlib.Path.glob()(GH-90208: Suppress OSError exceptions frompathlib.Path.glob()#104141)
Neither looks likely to affect test_gdb though.
Linked PRs
Metadata
Metadata
Assignees
Labels
testsTests in the Lib/test dirTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
encukou commentedon May 22, 2023
It looks like GDB can't extract the C stack here. I SSHd to the buildbot and tried:
The stack looks corrupted:
PyModuleDef_Typeis data, not a function.This looks like “miscommunication” between clang and gdb (bad debug info?)
The test seems very sensitive to platform/compiler, e.g.:
Note that the “installed” buildbots are green. (AFAIK these test Python after
make install.) But when Imake install, GDB still fails to extract the backtrace on the installed Python.ping @vstinner, the PEP-11 ppc64le/clang maintainer
encukou commentedon May 22, 2023
See #65462 (comment):
Doesn't clang always use LLVM?
The skip is now
'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be'Clang' in platform.python_compiler()?stratakis commentedon May 22, 2023
The failure started after the update of Clang to 16.0.0 from 15.0.7.
thesamesam commentedon May 23, 2023
I took it to mean "not Apple's Clang" which is distinct (has its own versioning and is essentially a fork). I might be wrong here though.
vstinner commentedon May 23, 2023
Maybe tets_gdb can skip a test if "frame did not save the PC" is found in the output. We already have many skips like that.
carljm commentedon Aug 30, 2023
It looks like multiple Clang buildbots are still failing
test_gdb.vstinner commentedon Sep 7, 2023
I closed issue #109053 as a duplicate of this issue.
vstinner commentedon Sep 13, 2023
I did tests on two ppc64le machines. In short, test_gdb only failed on ppc64le if Python is built with clang version 16.0.6 (
clang -Og). Well, that's what we saw on buildbots. But I preferred to confirm manually. I built Python manually and ran test_gdb manually with-u allto no skip any test.Machine 1:
Linux xxx 6.2.8-200.fc37.ppc64le #1 SMP Wed Mar 22 18:34:43 UTC 2023 ppc64le ppc64le ppc64le GNU/LinuxMachine 2:
Linux localhost.localdomain 5.8.17-200.fc32.ppc64le #1 SMP Thu Oct 29 18:00:08 UTC 2020 ppc64le GNU/LinuxGCC
Test:
./configure --with-pydebug && make && ./python -m test test_gdb -u all -v.Build built with:
gcc -Og.Machine 1: test_gdb: SUCCESS
Unable to read information on python framefound in gdb outputMachine 2: test_gdb: SUCCESS
Unable to read information on python framefound in gdb output"Total tests: run=46 skipped=1LLVM clang
Test:
./configure --with-pydebug CC=clang && make && ./python -m test test_gdb -u all -v.Build built with:
clang -Og.Machine 1: test_gdb: FAILURE
(unable to read python frame information)orUnable to read information on python framefound in gdb output.Total tests: run=46 failures=2 skipped=39Machine 2: test_gdb: SUCCESS
(unable to read python frame information)orUnable to read information on python framefound in gdb outputTotal tests: run=46 skipped=41vstinner commentedon Sep 13, 2023
Reproduce
_testcapi.meth_varargs()issue.script.py:cmds:gdb output on ppc64le with
clang -Og:The important message here is
Backtrace stopped: frame did not save the PC. The gdb traceback is really just 2 frames and the 2nd framePyModuleDef_Type ()is wrong :-(Comparison on x86-64, Python built with
gcc -Og:On x86-64 with gcc, I have 20 frames up to main() and the 2nd frame is
cfunction_call()as expected.vstinner commentedon Sep 13, 2023
script2.py:cmds2:gdb output on ppc64 with
clang -Og:Here, I think that the important part is
#1 0x0000000000000000 in ?? (). When gdb says that, usually it's likely that the traceback is truncated or wrong.For comparison, gdb on x86-64 on Python built with
gcc -Og:Here the 19 frames are complete up to main().
pythongh-104736: Fix test_gdb tests on ppc64le with clang
pythongh-104736: Fix test_gdb tests on ppc64le with clang
vstinner commentedon Sep 13, 2023
I wrote PR #109360 which fix test_gdb on Linux ppc64lc when Python is built with
clang -Og(LLVM clang 16).gh-104736: Fix test_gdb tests on ppc64le with clang (#109360)
6 remaining items
pythongh-104736: Fix test_gdb tests on ppc64le with clang (python#109360
pythongh-104736: Fix test_gdb tests on ppc64le with clang (python#109360
[3.12] gh-109972: Enhance test_gdb (#110026) (#110351)
[3.12] pythongh-109972: Enhance test_gdb (pythonGH-110026) (pythonGH-…
[3.11] [3.12] gh-109972: Enhance test_gdb (GH-110026) (GH-110351) (#1…
encukou commentedon Feb 14, 2025
The issue reappeared on Python 3.11 with clang 19 (f41). New failure:
that is:
vstinner commentedon Feb 19, 2025
@encukou: This build is a success, test_gdb didn't fail.
encukou commentedon Feb 20, 2025
Oops! See the next one, or https://buildbot.python.org/#/builders/967/builds/1169
vstinner commentedon Feb 21, 2025
That's the classical
<optimized out>problem. When the Python binary is optimized, sometimes gdb fails to read function arguments.I'm not sure why this problem only occurs on the 3.11 branch. Python internals likely changed in the meanwhile, and so clang optimizes the binary differently. Maybe gdb is able to read function arguments on Python 3.12+.
The 3.11 branch no longer accepts bugfixes, so I don't think that it's worth it to investigate.
encukou commentedon Feb 21, 2025
Should we mark the builder as unstable, then?
vstinner commentedon Feb 21, 2025
It sounds like a reasonable tradeoff. Is it possible to only mark a builder as unstable on a specific branch?
vstinner commentedon Feb 26, 2025
I created #130593 for the 3.11 branch to skip test_gdb if Python if built with
clang -Og(default optimization level when using--with-pydebug). It's a backport from main of #130550 change.