Gracefully exiting asyncio application
Continuing yesterday’s work on my simple supervisor library,
I continued trying to propagate signals cleanly to child processes
before exiting. I remembered that it isn’t enough to just propagate
signals - you also have to actually reap them. This meant waiting
wait calls on them to return.
I had a task running concurrently that is waiting on these processes. So ‘all’ I had to do was make sure the application does not exit until these tasks are done. This turned out to be harder than I thought! After a bunch of reading, I recognized that what I needed to do was make sure I wait for all pending tasks before actually exiting the application.
This was more involved than I thought. It also must be done at the application level rather than in the library - you don’t want libraries doing sys.exit, and definitely don’t want them closing event loops.
After a bunch of looking and playing, it looks like what I want is in my application code is something like:
if __name__ == '__main__': loop = asyncio.get_event_loop() try: loop.run_until_complete(main()) finally: loop.run_until_complete(asyncio.gather(*asyncio.Task.all_tasks())) loop.close()
This waits for all tasks to complete before exiting, and seems to make sure all child processes are reaped. However, I have a few unresolved questions:
- What happens when one of the tasks is designed to run forever, and never exit? Should we cancel all tasks? Cancel tasks after a timeout? Cancelling tasks after a timeout seems most appropriate.
- If a task schedules more tasks, do those get run? Or are they abandoned? This seems important - can tasks keep adding more tasks in a loop?
I am getting a better handle on what people mean by ‘asyncio is more complicated than needed’. I’m going to find places to read up on asyncio internals - particularly how the list of pending tasks is maintained.
Testing code that involves asyncio, signals and processes is
hard. I attempted to do so with
os.fork, but decided that is
super-hard mode and I’d rather not play. Instead, I wrote Python
code verbatim that is then spawned as a subprocess, and use
stdout to communicate back to the parent process. The child process’
code itself is inline
in the test file, which is terrible. I am going to move it to its
I also added tests for multiple signal handlers. I’ve been writing a lot more tests in the last few months than I was before. I credit Tim Head for a lot of this. It definitely gives me a lot more confidence in my code.