Python Multitasking - Async/Await vs Green Threads

2 years ago

I've been using Python professionally for over 30 years now, and it is by far my favorite programming language. Python's development has been guided by a philosophy that values simplicity as summed up in The Zen of Python – which turns out to be great for building highly readable and well-structured applications. Readability and structure are crucial when an application needs to handle additional complexity, which often comes in the form of multitasking.

Flavors of Multitasking

Multitasking is the concurrent execution of multiple software-defined "tasks" within a single process. By default, most software has a single "thread of execution" which starts at a program's entry point, passes through the software's instructions and eventually concludes at the exit point. Via requests to the operating system, processes can create and manage additional threads of execution.

Within this context, there are four basic approaches to providing multitasking:

  • Event-driven single thread using futures/promises or callbacks ("futures")
  • Event-driven single thread with async/await continuations ("async/await")
  • Cooperative "pseudo" or "green" threads within a single real thread ("green threads")
  • Multiple, real OS-level threads ("OS threads")

Futures and async/await are the only approach available in limited execution environments, such as the JavaScript interpreter of a web browser. Cooperative green threads and OS threads are additional options available in full programming environments like Go, .NET, Rust, and – of course – Python.

AsyncIO – Shiny!

While Python has supported OS threads from nearly the beginning, it did not support futures until Python 3.2 in 2011. Green threads have been available since 2006 – via the Eventlet module – and 2010 – via the Gevent fork of Eventlet. Eventlet and Gevent (my own preferred choice for multitasking) are both based on the green threading Greenlet Python module that grew out of the Stackless Python fork – which itself dates back to 1998.

The introduction of async/await and its associated library AsyncIO to Python has been the most notable (and at times most controversial: see Asynchronous Python and Databases, I don't understand Python's Asyncio, and more recently assyncio) part of the Python multitasking story. The async/await keywords were released in 2015 in Python 3.5, while AsyncIO was actually released prior to that as part of Python 3.4 in 2013 and later updated to use async/await.

Setting aside most of the details and drama of what made async/await and AsyncIO controversial, I'm going to focus on just the part of the story that I think hurts it in the long-run in comparison to Gevent (green threads). The way async/await was implemented and operates in Python differs from most other languages, which I think results in poor developer experience (DX).

In most languages, async/await is syntactic sugar for futures and "continuations." A continuation captures the current state of a function/method (i.e. the stack frame) as a closure that is then wrapped in a future, making it possible to suspend and resume it at a later time. This enables a developer to use the async and await keywords to write concurrent code that looks like it executes linearly. This is achieved by the language's interpreter/compiler using the placement of the keywords to guide the rewriting of async functions into an equivalent continuation + future implementation. The most important takeaway of this approach is that functions marked with async return a future. The await keyword wraps calls to async functions with a continuation that resumes when the future is resolved.

However, in Python's AsyncIO, generators (sometimes referred to as coroutines) are used as the underlying mechanism to enable async/await style concurrency. While the rules of Python's async and await are similar to implementations in other languages, the semantics are different in a few critical ways, which makes life harder on developers.

Here There Be Dragons

In future + continuation based async/await, execution is resumed when the future is resolved. And like most future mechanisms in event-driven environments, this is ultimately accomplished with a callback of some sort, usually hidden in a library. Most importantly, a function marked async will always resolve its future whether the caller awaits on it or not. Meaning you can call an async function with await or without. If called with await, a continuation and future are used to suspend execution and resume the calling function. If without, the return value of the call is a future – which can be safely ignored. The future will still complete, and the called function will resume on its own.

In Python's generator-based async/await, execution is only resumed when the generator produces its next value – and critically, the generator does not progress at all unless it is explicitly awaited. Meaning, the await keyword is not optional and is required whenever an async function is invoked. If you don't use the await keyword, the call will never finish execution and the interpreter emits a warning indicating this.

Another issue concerns where and how async and non-async Python functions can call each other. While both async/await implementation styles require the use of await at all call points in a nested stack of call frames for the thread of execution to properly suspend along that path (i.e. function coloring), continuation + futures implementations allow you to break this chain and create a new independent chain of async/await calls if you need/want to. In Python's generator-based implementation, you cannot call async functions from non-async ones. Once you enter an async function, all calls to async functions must use await and once a non-async function is called, the call tree below it must be entirely non-async.

These oddities result in unexpected landmines and unfortunate restrictions when using AsyncIO. But why did it end up like this?

The Python Interpreter's Baggage

While the particulars of the story might be gleaned from discussions on the Python maintainers mailing list from around that timeframe, I believe that these oddities have a root cause that's pretty simple to understand and explain: Python stack frames match C stack frames one-to-one. When Python executes compiled byte code and encounters an instruction to call another method in Python, it does so by calling an internal C function, thus establishing a new C stack frame for managing execution of the called Python method.

I believe this fact is why continuations don't exist (yet) in Python and why async/await uses generators instead. Every approach to multitasking other than futures requires stack manipulation of some sort. In other words, switching between Python tasks would normally require manipulating the Python stack. If the Python stack corresponds one-to-one with the C stack, that means multitasking requires manipulating the Python interpreter's internal C stack. The easiest way to accomplish stack management for the purposes of multitasking is to use green threads. However, for various reasons (backward compatibility being an important one) green threads have never been integrated into the core interpreter – despite excellent implementations of them being available from third parties.

Without green threading and no other interpreter ability to manipulate the C stack, the Python maintainers were left with little choice other than to adopt generators as the solution to Python multitasking. They encapsulate their own closure context and state of execution so it seems like a good fit. When switching between generators, additional C stack frames are not required. Generator switching (even within deeply nested hierarchies of async functions) is just a simple change of context in a single C stack frame between data structures that track each generator's execution context.

But using generators to enable multitasking came with the costs outlined in the previous section. Python async/await is not syntactic sugar over futures – you must explicitly await the generator returned from a call to an async function, and the function coloring problem is worse in comparison to other languages.

Paying the Pyper

And this is ultimately why I continue to use Gevent and do not plan to adopt AsyncIO. When cooperative green threads and their absence of function coloring are available, I believe the DX is considerably better than async/await or OS threads in any programming language. And even more so in Python given the situation with generator-based async/await.

Also, I believe that generator-based async/await will eventually be discarded and replaced with a continuation + futures based implementation, as is standard in most other languages. What will unblock this path is the eventual disconnect of Python stack frames from C stack frames ( which is being worked on) and/or the eventual inclusion of green threads as native interpreter functionality. While async/await and AsyncIO are important technologies that helped Python considerably with its tremendous adoption story, at the point that an alternative to generator-based async/await is feasible, it will be just a matter of time before sentiment drives adoption of continuation + futures based async/await that doesn't have the limitations of the current implementation.

Hopefully, that transition fares better than the Python version 2 to version 3 did.