Repository navigation
Read --doc-dir once when stubgen processes multiple C modules #22370
Description
I work on Pystachy, a compiler for typed Python. I was comparing it with mypyc, Cython and PyPy on mypyc name generator and mypy stub generator. While profiling stubgen, I noticed that each C module reads and parses the same documentation again.
The patch reads the docs once per invocation. With 319 files in Doc/library and four modules, that cuts the number of file reads from 1276 to 319.
stubgen --doc-dir Doc/library -m math -m cmath -m zlib -m arrayStubgen runs. Median wall time in milliseconds over seven fresh processes, including startup and writing all four stubs. Compare before and after within each row.
| Configuration | Before (ms) | After (ms) |
|---|---|---|
| CPython 3.13.16, upstream source | 653 | 356 |
| mypyc 2.4, compiled entry points on the released mypy core | 752 | 379 |
| Cython 3.3.1, compiled stubgen, stubgenc and stubdoc | 780 | 435 |
| PyPy 8.0.1, with import and docstring workarounds | 1265 | 900 |
Same parser on different runtimes. Here CPython ran stubgen and each backend ran the same adapted documentation parser. The timings cover the whole command. Worker rows include process startup. The JIT row also includes compilation.
| Documentation parser | Before (ms) | After (ms) |
|---|---|---|
| CPython | 859 | 457 |
| mypyc extension | 887 | 451 |
| Cython extension | 830 | 437 |
| PyPy worker | 1096 | 652 |
| Pystachy extension | 611 | 385 |
| Pystachy AOT worker | 658 | 417 |
| Pystachy JIT worker | 2321 | 2052 |
All CPython-hosted runs produced the same four stubs byte for byte. The PyPy-hosted run inspected PyPy own modules. Its four stubs also matched before and after.
Activity
den-run-ai commentedon Nov 4, 2026
Feel free to cherry pick the fork and let me know if I should submit a PR.
A5rocks commentedon Nov 5, 2026
I think it's a good idea to submit a PR. Thanks!