Repository navigation
Make set thread-safe in --disable-gil builds #112069
Description
Feature or enhancement
The set object is not currently thread-safe in --disable-gil builds. We should make it thread-safe by using the "critical section" API. to acquire the per-object locks around operations. There should be no effective change in the default build (i.e., with GIL) because critical sections are no-ops in the default build.
Notes:
- Unlike
dictandlist, I don't think it's worth the complexity to try to "optimistically avoid locking" around any set operation (exceptset_len). We could consider doing this in the future if there is a performance justification, but not for 3.13. set_lencan avoid locking and instead use relaxed atomics for reading the "used" field. Note that writes to "used" should then also use relaxed atomics.- Some operations require locking two containers (like
set_merge). Some of these will need refactorings so that the critical sections macros can be added in the correct places.
For context, here is the change from the nogil-3.12 fork: colesbury/nogil-3.12@4ca2924f0d. Note that the critical section API is slightly changed in 3.13 from nogil-3.12; In 3.13 Py_BEGIN_CRITICAL_SECTION takes a PyObject instead of a PyMutex.
TODO:
- Improve
set_init(see gh-112069: Make sets thread-safe with the GIL disabled #113800 (comment)). We also want to avoid locking inset_initif possible -
_PySet_NextEntry -
setiter_iternext
Linked PRs
- gh-112069: Make sets thread-safe with the GIL disabled #113800
- gh-112069: Adapt set/frozenset methods to Argument Clinic #115112
- Revert "gh-112069: Make sets thread-safe with the GIL disabled (#113800)" #116525
- gh-112069: Make setiter_iternext to be thread-safe #117935
- gh-112069: Add _PySet_NextEntryRef to be thread-safe. #117990
- gh-112069: Make PySet_GET_SIZE to be atomic safe. #118053
- gh-112069: Do not require lock if the set has never been exposed. #118069
Activity
tomasr8 commentedon Nov 16, 2023
Hi again! This one looks more challenging than the hashlib one but I'd like to try anyway :)
colesbury commentedon Nov 16, 2023
Thanks @tomasr8!
DinoV commentedon Jan 4, 2024
hi @tomasr8, just curious if you're still planning on working on this?
tomasr8 commentedon Jan 5, 2024
Hi! Yes, I took a break for Christmas but I have it almost working, just need to fix some tests. I could open a draft PR this weekend
gh-112069: Adapt set/frozenset methods to Argument Clinic (#115112)
pythongh-112069: Adapt set/frozenset methods to Argument Clinic (pyth…
gh-112069: Make sets thread-safe with the GIL disabled (#113800)
Revert "pythongh-112069: Make sets thread-safe with the GIL disabled (p…
pythongh-112069: Make sets thread-safe with the GIL disabled (python#…
pythongh-112069: Make setiter_iternext to be thread-safe
corona10 commentedon Apr 16, 2024
@colesbury I am going to work on
_PySet_NextEntrythis week, considering the caller-side usage.gh-112069: Make setiter_iternext to be thread-safe (gh-117935)
pythongh-112069: Make sets thread-safe with the GIL disabled (python#…
pythongh-112069: Make setiter_iternext to be thread-safe (pythongh-11…
pythongh-112069: Make _PySet_NextEntry to be thread-safe.
gh-112069: Add _PySet_NextEntryRef to be thread-safe. (gh-117990)
pythongh-112069: Make PySet_GET_SIZE to be atomic operation
gh-112069: Make PySet_GET_SIZE to be atomic safe. (gh-118053)
pythongh-112069: Do not require lock if the set has never been exposed.
Merge remote-tracking branch 'upstream/main' into pythongh-112069-init
gh-112069: Do not require lock if the set has never been exposed. (gh…
corona10 commentedon Apr 25, 2024
@colesbury Now we can close the issue right? Or more thing is left?
colesbury commentedon Apr 25, 2024
Yeah, I think so