Repository navigation
asyncio signal handler receives signals from child processes #94454
Description
Bug report
Today I ran into a very weird behaviour in one of my projects, that uses asyncio and multiprocessing.
When I interrupted the program with a Ctrl-C on the terminal, the signal handler set in asyncio with loop.add_signal_handler was triggered multiple times! Further inspection showed, that if I manually send a SIGINT to one of the worker child processes, it triggered the main process signal handler.
I could trace the issue to the socket used by asyncio to receive the signals via signal.set_wakeup_fd, which seems to receive the signal bytes also from the child processes.
Running signal.set_wakeup_fd(-1) at least once in the worker processes worked as a workaround for me but cannot be the final solution.
Also, it seems, that other signal operations also can clear the issue (for example some automatically manager child processes did not trigger the bug, as they seem to ignore SIGINT).
In my project I used concurrent.futures.ProcessPoolExecutor, but I don't think that matters, as long as the signal handler (and wakeup fd) is set before the child is started.
Your environment
- CPython versions tested on:
Python 3.10.4 - Operating system and architecture:
Kali Linux rolling with kernel: Debian 5.16.18-1kali1 (2022-04-01) x86_64
Bug output
asyncio_signal_bug.py.txt
# python3 asyncio_signal_bug.py
394236 Signal Handler set
394237 Worker 1 started
394240 Worker 2 started
^C394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
^C394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
# python3 asyncio_signal_bug.py --set-in-children
394241 Signal Handler set
394242 Worker 1 started
394245 Worker 2 started
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
Notice, that in the first block, for every ^C, there are three signals received, whereas in the second block (if signal.set_wakeup_fd(-1) was called in the workers) there is only one.
Activity
shivaraj-arch commentedon Jul 1, 2022
On Mac it runs the same as per the docs.
Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication docs
Let me know if I have left something out while testing.
ozppupbg commentedon Jul 1, 2022
I just tried on a different system with Python 3.6.9 on Ubuntu 18.04.6 LTS with Kernel 4.15.0-188 and could confirm the buggy behaviour (after adjusting for some asyncio API naming changes).
I expect, that it has either something to do with Mac handling
signalfd(the underlying syscall) slightly different or how your terminal issues SIGINT to child processes. I used Bash in my experiments, which obviously sends a SIGINT to the entire process group.Could you please try to manually send a signal to one of the worker processes with
kill -2 <worker PID>?For me this is also received in the parent process.
Also, please note that this issue does not concern an inter-thread, but an inter-process scenario.
shivaraj-arch commentedon Jul 1, 2022
Tried with bash 5.1. Also tried sending manual signal. The behaviour is the same ( see complete trace below without redirection to null ) with or without set children argument.
when manual kill signal is given the traceback does not spill on to terminal, there is no Process SpawnProcess-1/2 .
Maybe its difference with how Mac m1 and Linux kernels handle.
ozppupbg commentedon Jul 1, 2022
I think I just found the reason:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
If a different process start method is used, this of course changes the behaviour!
Can you try adding:
import multiprocessing as mpand in main:
mp.set_start_method('fork')When I set it to 'spawn' for me, I saw exectly the same behaviour you see.
shivaraj-arch commentedon Jul 1, 2022
True , spawn vs fork is the issue. I added fork as multiprocessing start method as you mentioned and the result is exactly what you found in beginning. But I am not sure why traceback is spilling over during spawn.
ozppupbg commentedon Jul 1, 2022
I guess, the worker processes terminate and thus print the interrupt output.
You should be able to check, if they are still running after the first ^C.
shivaraj-arch commentedon Jul 2, 2022
Workers are terminating and interrupt is not caught on first press of ctrl-c.
Looks like there is Base Exception asyncio issue with handling KeyboardInterrupt
ozppupbg commentedon Jul 2, 2022
I'm not sure, but I think this is completely unrelated to the
signalfdissue I'm referring to?As a fix my issue, I would suggest adding something along the lines of:
To: https://github.com/python/cpython/blob/main/Lib/asyncio/unix_events.py _UnixSelectorEventLoop.__init__
And I just noticed, that there may be other issues regarding the sockets
_ssockand_csock(from https://github.com/python/cpython/blob/7db1d2eaf367a1073191c80c7baeee41ae1f2f21/Lib/asyncio/selector_events.py BaseSelectorEventLoop) in the fork case:If a worker uses asyncio itself and they use the same socketpair for their internal wakeup, it might get messy!
kumaraditya303 commentedon Sep 29, 2022
asyncioandforkdon't work together. This seems to be a duplicate of #66285.renan-r-santos commentedon Sep 29, 2022
I was able to reproduce the issue using the code below both on Ubuntu 22.04 and MacOS 12.6 Intel using Python 3.10.5. Sending
SIGTERMto the child also terminates the parent. Addingsignal.set_wakeup_fd(-1)to the child or setting the multiprocessing start method to spawn fix the problem.@kumaraditya303 I was reading the issue you mentioned but it doesn't look like a duplicate because the child isn't attempting to use asyncio's event loop.
kumaraditya303 commentedon Sep 30, 2022
The child is using the wakeup fd of the parent process since
asynciodoes not adds fork handlers to change the wakeup fd.I was correct as
asynciodoes not supportsfork, the child continues to use parent's wakeup fd.gvanrossum commentedon Sep 30, 2022
The OP isn't trying to use fork(), nor are they using asyncio in the handlers. But they are using a
ProcessPoolExecutorand the forked subprocess inherits some signal settings. It seems sensible to implement the OP's fix from #94454 (comment).kumaraditya303 commentedon Oct 2, 2022
Multiprocessing fork context uses
forkunder the hood.That could be done although there could be edge cases.
forkdoes not plays well with signal handlers and child watchers etc.gvanrossum commentedon Oct 2, 2022
TBH I don't know anything about multiprocessing and I'm not about to volunteer to become an expert there. I don't know if the responsibility would lay with asyncio or with multiprocessing. But one of these will have to accommodate the other, somehow. The OP's fix seems the simplest thing that could possibly work, so let's do that in 3.12 and see how it goes. (The worst case outcome is that there are still some cases that remain broken, for which we can iterate if someone reports an example.)