Aug 5, 202621 min read

Notes on C++ Coroutines

Notes on tracing an example C++ coroutine through its compiler-generated assembly

notes, cpp, coroutines, assembly

Notes on C++ Coroutines

Architecture

Compiler turns a coroutine function into
- a coroutine factory (what gets called on first invocation)
- a coroutine "actor" (body of function that handles resume/suspend afaik at specific "breakpoints")
- a coroutine teardown function
an example layout of a coroutine frame (see example program below)

Frame offset	Meaning
`+0`	pointer to the generated `.actor` function
`+8`	pointer to the generated `.destroy` function
`+16`	`promise_type` object
`+24`	saved argument `continuation_out`, which is `&h` from `main`
`+32`	coroutine state/resume index
`+34`	compiler lifetime/deallocation bookkeeping counter
`+36`	whether this frame must be freed
`+37`	whether `initial_suspend().await_resume()` has happened
`+38`	storage for the initial `suspend_never` awaiter
`+40`	local `Awaiter a`, specifically its `hp_` pointer
`+48`	local loop variable `unsigned i`
`+52`	storage for the final `suspend_never` awaiter

Definitions

Coroutines in C++
- functions that can invoke each other but do not share a stack
Coroutine
- can suspend execution & pass execution context to somewhere else without losing local vars
Coroutine handles
- std::coroutine_handle<>
- works like a function pointer except the function doesn't lose stack vars & saves function state
- does not have a destructor
- has a destroy function, however

What the compiler sees/does

co_await

foo() {

  co_await {expr A}
}

compiles into

foo() {
  // save foo's local vars inside heap
  // generate "pointer" (coroutine_handle) to said heap locat4ion
  // call {expr A}.await_suspend(pointer_to_foo_heap)
  //
}

Note here,

{expr A} must support the functions
- await_suspend(coroutine_handle)
- await_resume(coroutine_handle)
- await_ready(coroutine_handle)

An example program

Credit: this example is from David Mazières’s My tutorial and take on C++20 coroutines.

#include <concepts>
#include <coroutine>
#include <exception>
#include <iostream>

struct ReturnObject {
  struct promise_type {
    ReturnObject get_return_object() { return {}; }
    std::suspend_never initial_suspend() { return {}; }
    std::suspend_never final_suspend() noexcept { return {}; }
    void unhandled_exception() {}
  };
};

struct Awaiter {
  std::coroutine_handle<> *hp_;
  constexpr bool await_ready() const noexcept { return false; }
  void await_suspend(std::coroutine_handle<> h) { *hp_ = h; }
  constexpr void await_resume() const noexcept {}
};

ReturnObject
counter(std::coroutine_handle<> *continuation_out)
{
  Awaiter a{continuation_out};
  for (unsigned i = 0;; ++i) {
    co_await a;
    std::cout << "counter: " << i << std::endl;
  }
}

int
main()
{
  std::coroutine_handle<> h;
  counter(&h);
  for (int i = 0; i < 3; ++i) {
    std::cout << "In main1 function\n";
    h();
  }
  h.destroy();
}

compiles into

"ReturnObject::promise_type::get_return_object()":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret"ReturnObject::promise_type::initial_suspend()":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret"ReturnObject::promise_type::final_suspend()":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret"ReturnObject::promise_type::unhandled_exception()":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret"Awaiter::await_ready() const":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        mov     eax, 0        pop     rbp        ret"Awaiter::await_suspend(std::__n4861::coroutine_handle<void>)":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        mov     QWORD PTR [rbp-16], rsi        mov     rax, QWORD PTR [rbp-8]        mov     rax, QWORD PTR [rax]        mov     rdx, QWORD PTR [rbp-16]        mov     QWORD PTR [rax], rdx        nop        pop     rbp        ret"Awaiter::await_resume() const":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret"counter(std::__n4861::coroutine_handle<void>*)":        push    rbp        mov     rbp, rsp        push    rbx        sub     rsp, 40        mov     QWORD PTR [rbp-40], rdi        mov     eax, 56        mov     rdi, rax        call    "operator new(unsigned long)"        mov     QWORD PTR [rbp-24], rax        mov     rax, QWORD PTR [rbp-24]        mov     BYTE PTR [rax+36], 1        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], 0        mov     rax, QWORD PTR [rbp-24]        mov     QWORD PTR [rax], OFFSET FLAT:"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]"        mov     rax, QWORD PTR [rbp-24]        mov     QWORD PTR [rax+8], OFFSET FLAT:"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .destroy]"        mov     rdx, QWORD PTR [rbp-40]        mov     rax, QWORD PTR [rbp-24]        mov     QWORD PTR [rax+24], rdx        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], 1        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+32], 0        mov     rax, QWORD PTR [rbp-24]        add     rax, 16        mov     rdi, rax        call    "ReturnObject::promise_type::get_return_object()"        mov     rax, QWORD PTR [rbp-24]        mov     rdi, rax        call    "counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]"        mov     rax, QWORD PTR [rbp-24]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax-1]        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], dx        mov     rax, QWORD PTR [rbp-24]        movzx   eax, WORD PTR [rax+34]        test    ax, ax        jne     .L23        mov     rax, QWORD PTR [rbp-24]        mov     esi, 56        mov     rdi, rax        call    "operator delete(void*, unsigned long)".L23:        jmp     .L27        mov     rcx, rax        mov     rax, QWORD PTR [rbp-24]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax-1]        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], dx        mov     rbx, rcx        mov     rax, QWORD PTR [rbp-24]        movzx   eax, WORD PTR [rax+34]        test    ax, ax        jne     .L25        mov     rax, QWORD PTR [rbp-24]        mov     esi, 56        mov     rdi, rax        call    "operator delete(void*, unsigned long)".L25:        mov     rax, rbx        mov     rdi, rax        call    "_Unwind_Resume".L27:        mov     rbx, QWORD PTR [rbp-8]        leave        ret.LC0:        .string "counter: ""counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]":        push    rbp        mov     rbp, rsp        push    rbx        sub     rsp, 56        mov     QWORD PTR [rbp-56], rdi        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        and     eax, 1        test    ax, ax        je      .L29        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        movzx   eax, ax        cmp     eax, 7        je      .L38        cmp     eax, 7        jg      .L39        cmp     eax, 5        je      .L37        cmp     eax, 5        jg      .L39        cmp     eax, 1        je      .L62        cmp     eax, 3        je      .L36        jmp     .L39.L29:        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        movzx   eax, ax        cmp     eax, 6        je      .L48        cmp     eax, 6        jg      .L39        cmp     eax, 4        je      .L47        cmp     eax, 4        jg      .L39        test    eax, eax        je      .L43        cmp     eax, 2        je      .L46        jmp     .L39.L43:        mov     rax, QWORD PTR [rbp-56]        mov     BYTE PTR [rax+37], 0        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax+1]        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+34], dx        mov     rax, QWORD PTR [rbp-56]        add     rax, 16        mov     rdi, rax        call    "ReturnObject::promise_type::initial_suspend()"        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 2        mov     rax, QWORD PTR [rbp-56]        add     rax, 38        mov     rdi, rax        call    "std::__n4861::suspend_never::await_ready() const"        xor     eax, 1        test    al, al        jne     .L45        jmp     .L46.L39:        ud2.L45:        mov     rax, QWORD PTR [rbp-56]        lea     rbx, [rax+38]        mov     rax, QWORD PTR [rbp-56]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::from_address(void*)"        mov     QWORD PTR [rbp-40], rax        lea     rax, [rbp-40]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::operator std::__n4861::coroutine_handle<void>() const"        mov     rsi, rax        mov     rdi, rbx        call    "std::__n4861::suspend_never::await_suspend(std::__n4861::coroutine_handle<void>) const"        jmp     .L49.L36:        jmp     .L35.L46:        mov     rax, QWORD PTR [rbp-56]        mov     BYTE PTR [rax+37], 1        mov     rax, QWORD PTR [rbp-56]        add     rax, 38        mov     rdi, rax        call    "std::__n4861::suspend_never::await_resume() const"        mov     rax, QWORD PTR [rbp-56]        mov     rdx, QWORD PTR [rax+24]        mov     rax, QWORD PTR [rbp-56]        mov     QWORD PTR [rax+40], rdx        mov     rax, QWORD PTR [rbp-56]        mov     DWORD PTR [rax+48], 0.L50:        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 4        mov     rax, QWORD PTR [rbp-56]        add     rax, 40        mov     rdi, rax        call    "Awaiter::await_ready() const"        xor     eax, 1        test    al, al        je      .L47        mov     rax, QWORD PTR [rbp-56]        lea     rbx, [rax+40]        mov     rax, QWORD PTR [rbp-56]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::from_address(void*)"        mov     QWORD PTR [rbp-32], rax        lea     rax, [rbp-32]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::operator std::__n4861::coroutine_handle<void>() const"        mov     rsi, rax        mov     rdi, rbx        call    "Awaiter::await_suspend(std::__n4861::coroutine_handle<void>)"        jmp     .L49.L37:        jmp     .L35.L47:        mov     rax, QWORD PTR [rbp-56]        add     rax, 40        mov     rdi, rax        call    "Awaiter::await_resume() const"        mov     esi, OFFSET FLAT:.LC0        mov     edi, OFFSET FLAT:"std::cout"        call    "std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)"        mov     rdx, rax        mov     rax, QWORD PTR [rbp-56]        mov     eax, DWORD PTR [rax+48]        mov     esi, eax        mov     rdi, rdx        call    "std::basic_ostream<char, std::char_traits<char> >::operator<<(unsigned int)"        mov     esi, OFFSET FLAT:"std::basic_ostream<char, std::char_traits<char> >& std::endl<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&)"        mov     rdi, rax        call    "std::basic_ostream<char, std::char_traits<char> >::operator<<(std::basic_ostream<char, std::char_traits<char> >& (*)(std::basic_ostream<char, std::char_traits<char> >&))"        mov     rax, QWORD PTR [rbp-56]        mov     eax, DWORD PTR [rax+48]        lea     edx, [rax+1]        mov     rax, QWORD PTR [rbp-56]        mov     DWORD PTR [rax+48], edx        jmp     .L50.L57:        mov     rax, QWORD PTR [rbp-56]        lea     rbx, [rax+52]        mov     rax, QWORD PTR [rbp-56]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::from_address(void*)"        mov     QWORD PTR [rbp-24], rax        lea     rax, [rbp-24]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::operator std::__n4861::coroutine_handle<void>() const"        mov     rsi, rax        mov     rdi, rbx        call    "std::__n4861::suspend_never::await_suspend(std::__n4861::coroutine_handle<void>) const"        jmp     .L49.L38:        jmp     .L35.L48:        mov     rax, QWORD PTR [rbp-56]        add     rax, 52        mov     rdi, rax        call    "std::__n4861::suspend_never::await_resume() const"        jmp     .L35.L62:        nop.L35:        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax-1]        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+34], dx        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+34]        test    ax, ax        jne     .L61        mov     rax, QWORD PTR [rbp-56]        movzx   eax, BYTE PTR [rax+36]        test    al, al        je      .L61        mov     rax, QWORD PTR [rbp-56]        mov     esi, 56        mov     rdi, rax        call    "operator delete(void*, unsigned long)"        jmp     .L61.L49:        jmp     .L61        mov     rdi, rax        call    "__cxa_begin_catch"        mov     rax, QWORD PTR [rbp-56]        movzx   eax, BYTE PTR [rax+37]        xor     eax, 1        test    al, al        je      .L56        call    "__cxa_rethrow".L56:        mov     rax, QWORD PTR [rbp-56]        mov     QWORD PTR [rax], 0        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 0        mov     rax, QWORD PTR [rbp-56]        add     rax, 16        mov     rdi, rax        call    "ReturnObject::promise_type::unhandled_exception()"        call    "__cxa_end_catch"        mov     rax, QWORD PTR [rbp-56]        mov     QWORD PTR [rax], 0        mov     rax, QWORD PTR [rbp-56]        add     rax, 16        mov     rdi, rax        call    "ReturnObject::promise_type::final_suspend()"        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 6        mov     rax, QWORD PTR [rbp-56]        add     rax, 52        mov     rdi, rax        call    "std::__n4861::suspend_never::await_ready() const"        xor     eax, 1        test    al, al        jne     .L57        jmp     .L48        mov     rbx, rax        call    "__cxa_end_catch"        mov     rax, rbx        mov     rdi, rax        call    "_Unwind_Resume".L61:        mov     rbx, QWORD PTR [rbp-8]        leave        ret"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .destroy]":        push    rbp        mov     rbp, rsp        sub     rsp, 16        mov     QWORD PTR [rbp-8], rdi        mov     rax, QWORD PTR [rbp-8]        movzx   eax, WORD PTR [rax+32]        or      eax, 1        mov     edx, eax        mov     rax, QWORD PTR [rbp-8]        mov     WORD PTR [rax+32], dx        mov     rax, QWORD PTR [rbp-8]        mov     rdi, rax        call    "counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]"        nop        leave        ret.LC1:        .string "In main1 function\n""main":        push    rbp        mov     rbp, rsp        sub     rsp, 16        mov     QWORD PTR [rbp-16], 0        lea     rax, [rbp-16]        mov     rdi, rax        call    "counter(std::__n4861::coroutine_handle<void>*)"        mov     DWORD PTR [rbp-4], 0        jmp     .L66.L67:        mov     esi, OFFSET FLAT:.LC1        mov     edi, OFFSET FLAT:"std::cout"        call    "std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)"        lea     rax, [rbp-16]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<void>::operator()() const"        add     DWORD PTR [rbp-4], 1.L66:        cmp     DWORD PTR [rbp-4], 2        jle     .L67        lea     rax, [rbp-16]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<void>::destroy() const"        mov     eax, 0        leave        ret

Walkthrough:

Enters main function

"main":        push    rbp        mov     rbp, rsp        sub     rsp, 16

enters main function, allocates 16 bytes to stack
- std::coroutine_handle<> h at -16
- int i at -4

Set h to nullptr

 mov     QWORD PTR [rbp-16], 0        lea     rax, [rbp-16]        mov     rdi, rax

set h to nullptr

Load address of h

        lea     rax, [rbp-16]

rax = &h

Call coroutine factory

        mov     rdi, rax        call    "counter(std::__n4861::coroutine_handle<void>*)"

call the coroutine factory

Function prologue

"counter(std::__n4861::coroutine_handle<void>*)":        push    rbp        mov     rbp, rsp        push    rbx        sub     rsp, 40

save 40 bytes of stack space
- layout roughly like
  - -40 - &h
  - ???

This is what the stack looks like

Higher addresses

rbp + 8  ┌────────────────────────────┐
         │ factory return address     │  pushed by call
rbp + 0  ├────────────────────────────┤
         │ caller's saved rbp         │  push rbp
rbp - 8  ├────────────────────────────┤
         │ caller's saved rbx         │  push rbx
rbp - 16 ├────────────────────────────┤
         │ unused/reserved            │
rbp - 24 ├────────────────────────────┤
         │ coroutine-frame pointer F  │  [rbp-24]
rbp - 32 ├────────────────────────────┤
         │ unused/reserved            │
rbp - 40 ├────────────────────────────┤
         │ continuation_out argument  │  [rbp-40]
rbp - 48 ├────────────────────────────┤
         │ unused/alignment space     │
         └────────────────────────────┘
         rsp after prologue

Lower addresses

Save argument to local stack

        mov     QWORD PTR [rbp-40], rdi

stack_local_continuation_out_ptr = continuation_out;

Malloc the frame for the coroutine

        mov     eax, 56        mov     rdi, rax        call    "operator new(unsigned long)"

malloc the frame for the coroutine

Store the pointer of frame into stack

        mov     QWORD PTR [rbp-24], rax

[rbp-24] = frame

Compile bookkeeping

        mov     rax, QWORD PTR [rbp-24]        mov     BYTE PTR [rax+36], 1        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], 0

rax = [rbp-24] = frame
[rax + 36] = [frame + 36] = frame->needs_free = true
[rax + 34] = [frame + 34] = frame->lifetime_count = 0

Store pointers to continue and destroy functions

        mov     rax, QWORD PTR [rbp-24]        mov     QWORD PTR [rax], OFFSET FLAT:"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]"        mov     rax, QWORD PTR [rbp-24]        mov     QWORD PTR [rax+8], OFFSET FLAT:"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .destroy]"

[frame + 0] = frame->actor_fxn = &counter_actor
[frame + 8] = frame->destroy_fxn = &counter_destroyer

Copy source argument into frame

        mov     rdx, QWORD PTR [rbp-40]        mov     rax, QWORD PTR [rbp-24]        mov     QWORD PTR [rax+24], rdx

rdx = continuation_out;
[frame + 24] = frame->continuation_out = rdx = continuation_out;

Initialize frame logistics

        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], 1        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+32], 0

[frame + 34] = frame->lifetime_count = 1
[frame + 32] = frame->state = 0
state = 0 implies start of invocation

Call get return object

        mov     rax, QWORD PTR [rbp-24]        add     rax, 16        mov     rdi, rax        call    "ReturnObject::promise_type::get_return_object()"

rax = frame + 16 = &frame->promise;
rdi = rax (first argument)
call get return object function

Note this is the get return object function

"ReturnObject::promise_type::get_return_object()":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret

(question, what is the point of this return object i.e. in a real scenario?)

Call the actor for the first time

        mov     rax, QWORD PTR [rbp-24]        mov     rdi, rax        call    "counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]"

counter_actor(frame);

First actor call

        .string "counter: ""counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]":        push    rbp        mov     rbp, rsp        push    rbx        sub     rsp, 56        mov     QWORD PTR [rbp-56], rdi

actor stack is 56 bytes
local_frame = frame

Test low bit of state (for destroy fxn call)

mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        and     eax, 1        test    ax, ax        je      .L29

state = frame->state;
if ((state & 1) == 0) goto .L29;
else continue;

it then casts it to an eax (uint16_t)

        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        movzx   eax, ax

Odd destroy fxn call jump

        cmp     eax, 7        je      .L38        cmp     eax, 7        jg      .L39        cmp     eax, 5        je      .L37        cmp     eax, 5        jg      .L39        cmp     eax, 1        je      .L62        cmp     eax, 3        je      .L36        jmp     .L39

This is

switch (frame->state) {
    case 1: goto .L62;
    case 3: goto .L36;
    case 5: goto .L37;
    case 7: goto .L38;
    default: goto .L39;
};

Even resume fxn call jump

.L29:        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        movzx   eax, ax        cmp     eax, 6        je      .L48        cmp     eax, 6        jg      .L39        cmp     eax, 4        je      .L47        cmp     eax, 4        jg      .L39        test    eax, eax        je      .L43        cmp     eax, 2        je      .L46        jmp     .L39

This is

switch (frame->state) {
    case 6: goto .L48;
    case 4: goto .L47;
    case 2: goto .L46;
    case 0: goto .L43;
    default: goto .L39;
};

`.L43` first entry

.L43:        mov     rax, QWORD PTR [rbp-56]        mov     BYTE PTR [rax+37], 0

frame->initial_resume_started = false

        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax+1]        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+34], dx

frame->lifetime_count++
note now lifetime_count == 2

        mov     rax, QWORD PTR [rbp-56]        add     rax, 16        mov     rdi, rax        call    "ReturnObject::promise_type::initial_suspend()"

push frame->object address onto rdi
call ReturnObject::promise_type::initial_suspend()

"ReturnObject::promise_type::initial_suspend()":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        nop        pop     rbp        ret

        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 2

Question: how is this initial_awaiter = frame->promise.initial_suspend();?
- we only called a function, where is this = coming from?

        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 2

save frame->state = 2

        mov     rax, QWORD PTR [rbp-56]        add     rax, 38        mov     rdi, rax        call    "std::__n4861::suspend_never::await_ready() const"

run await_ready function of the expression suspend_never
ready = frame->initial_awaiter.await_ready();

Note that await_ready effectively always returns false in this scenario

"Awaiter::await_ready() const":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        mov     eax, 0        pop     rbp        ret

for suspend_never, ready == True

        xor     eax, 1        test    al, al        jne     .L45        jmp     .L46

the above means

should_suspend = !ready;

if (should_suspend)
    goto .L45;
else
    goto .L46;

`.L39` invalid state trap

ud2

invalid opcode exception

`.L45`

`.L46` second entry (after initial suspend)

.L46:        mov     rax, QWORD PTR [rbp-56]        mov     BYTE PTR [rax+37], 1

frame->initial_resume_started = true

        mov     rax, QWORD PTR [rbp-56]        add     rax, 38        mov     rdi, rax        call    "std::__n4861::suspend_never::await_resume() const"

Call await_resume for initial suspend_never awaiter note await_resume is a no-op in this case

 mov     rax, QWORD PTR [rbp-56]        mov     rdx, QWORD PTR [rax+24]        mov     rax, QWORD PTR [rbp-56]        mov     QWORD PTR [rax+40], rdx        mov     rax, QWORD PTR [rbp-56]        mov     DWORD PTR [rax+48], 0

rdx = frame->continuation_out = &main.h
frame->a.hp_ = rdx = &main.h
then, initializ [frame + 48] = frame->counter_i = 0

`.L50` execute `co_await a`

.L50:        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+32], 4

save [frame + 32] = frame->resume_state = 4

        mov     rax, QWORD PTR [rbp-56]        add     rax, 40        mov     rdi, rax        call    "Awaiter::await_ready() const"

prepare the function call and call Awaiter::await_ready()

        xor     eax, 1        test    al, al        je      .L47

if (ready) goto .L47

        mov     rax, QWORD PTR [rbp-56]        lea     rbx, [rax+40]

rbx = &frame->a;

        mov     rax, QWORD PTR [rbp-56]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::from_address(void*)"

get the "typed" handle by passing the frame ptr to from_address(void*)
Question: what is a "typed handle"?

        mov     QWORD PTR [rbp-32], rax

put it on the stack

        lea     rax, [rbp-32]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<ReturnObject::promise_type>::operator std::__n4861::coroutine_handle<void>() const"

convert to untyped coroutine handle

        mov     rsi, rax        mov     rdi, rbx        call    "Awaiter::await_suspend(std::__n4861::coroutine_handle<void>)"

rsi = untyped_handle = frame
rdi = &frame->a
call await_suspend

Await suspend

"Awaiter::await_suspend(std::__n4861::coroutine_handle<void>)":        push    rbp        mov     rbp, rsp        mov     QWORD PTR [rbp-8], rdi        mov     QWORD PTR [rbp-16], rsi

save arguments in stack slots

        mov     rax, QWORD PTR [rbp-8]        mov     rax, QWORD PTR [rax]

rax = [rbp-8] = rdi = &frame->a
rax = [rax] = frame->a.hp_ = &h

        mov     rdx, QWORD PTR [rbp-16]

make rdx point at the memory backing the value of h

        mov     QWORD PTR [rax], rdx

[rax] = *frame->a.hp_ = rdx = frame
set the value of the memory backing the value of h to the coroutine frame pointer
the magic sauce behind the coroutine is that it sets the value of what the main corotuine handle h points to to the frame ptr of the current coroutine frame, so next time h() is called it sets program counter to the coroutine's frame which contains the execution/continuation instructions

        nop        pop     rbp        ret

return

Back to `.L50`

        jmp     .L49

suspend the function actually and pop off the stack

`.L49`

.L49:        jmp     .L61

another jump

`.L61:`

        mov     rbx, QWORD PTR [rbp-8]        leave        ret

return

Now we're back at the counter wrapper again

right after calling the actor in the coroutine factory, we're now at

        mov     rax, QWORD PTR [rbp-24]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax-1]        mov     rax, QWORD PTR [rbp-24]        mov     WORD PTR [rax+34], dx

frame->lifetime_count-- (decrement reference)
- history was
  - coroutine factory initialized to 1
  - actor bumped it to 2
  - coroutine factory decremented back to 1

        mov     rax, QWORD PTR [rbp-24]        movzx   eax, WORD PTR [rax+34]        test    ax, ax        jne     .L23

if frame->lifetime_count == 0 then delete the frame

deletion code below

mov     rax, QWORD PTR [rbp-24]mov     esi, 56mov     rdi, raxcall    "operator delete(void*, unsigned long)"

`.L23`

        jmp     .L27

return to counter

`.L27`

.L27:        mov     rbx, QWORD PTR [rbp-8]        leave        ret

return to main

We're now back in main land

        mov     DWORD PTR [rbp-4], 0

set i = 0

        jmp     .L66

go to the loop

        cmp     DWORD PTR [rbp-4], 2        jle     .L67

check i <= 2, if so, go to start of loop body

`.L67` loop body

        mov     esi, OFFSET FLAT:.LC1        mov     edi, OFFSET FLAT:"std::cout"        call    "std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)"

call std::cout << "In main1 function\n"
first argument is rdi = &std::cout
second argument is rsi = &"In main1 function\n"

        lea     rax, [rbp-16]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<void>::operator()() const"

call the coroutine frame
conceptually it does - frame = h.address() - frame->resume_fn(frame)
- using frame layout as specified earlier
  - frame = h.address()
  - actor = *(function_pointer *)(frame + 0)
  - actor(frame)
- note we don't call the coroutine factory thing again. We just call the actor

We enter frame w/ state = 4

This part I will skip, it is relatively straightforward. It prints i = {N} and goes to co_await again.

The co_await once again stores the value of its current coroutine frame into the memory backing h's value through the pointer stored in a.hp_.

Then we go back into the main loop, add 1 to i add DWORD PTR [rbp-4], 1 and repeat the loop until we have reached i > 2, which takes us past jle .L67

After the end of the loop, call the destroy function on the coroutin frame

        lea     rax, [rbp-16]        mov     rdi, rax        call    "std::__n4861::coroutine_handle<void>::destroy() const"

pass h's pointed to coroutine frame to the destroy function

Destroy function for coroutine

Note that here we enter through w/ state = 4
Worth noting this destroy function is really a wrapper around calling the actor w/ state + 1 (which is destroy mode in this compilation)

"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .destroy]":        push    rbp        mov     rbp, rsp        sub     rsp, 16        mov     QWORD PTR [rbp-8], rdi

function prologue + stack.frame = frame

        mov     rax, QWORD PTR [rbp-8]        movzx   eax, WORD PTR [rax+32]        or      eax, 1        mov     edx, eax        mov     rax, QWORD PTR [rbp-8]        mov     WORD PTR [rax+32], dx

frame->state |= 1

        call    "counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]"

Entering counter actor in destroy mode

"counter(_Z7counterPNSt7__n486116coroutine_handleIvEE.Frame*) [clone .actor]":        push    rbp        mov     rbp, rsp        push    rbx        sub     rsp, 56        mov     QWORD PTR [rbp-56], rdi        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        and     eax, 1        test    ax, ax        je      .L29

function prologue
if lower bit not set, then jump to .L29 which is the even cases
since it is set, don't jump, keep going

        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+32]        movzx   eax, ax        cmp     eax, 7        je      .L38        cmp     eax, 7        jg      .L39        cmp     eax, 5        je      .L37        cmp     eax, 5        jg      .L39        cmp     eax, 1        je      .L62        cmp     eax, 3        je      .L36        jmp     .L39

depending on which state it is in, go to appropriate destruction section
note .L39 is raising invalid opcode instruction
Here we're in 5 so we will visit that

Coroutine destructor (for real this time) at `.L35`

.L35:        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+34]        lea     edx, [rax-1]        mov     rax, QWORD PTR [rbp-56]        mov     WORD PTR [rax+34], dx

frame->lifetime_count--;

        mov     rax, QWORD PTR [rbp-56]        movzx   eax, WORD PTR [rax+34]        test    ax, ax        jne     .L61

if (frame->lifetime_count != 0) return;
- Note .L61 is just
  .L61: mov rbx, QWORD PTR [rbp-8] leave ret ```
in reality we probably should never hit this branch

        mov     rax, QWORD PTR [rbp-56]        movzx   eax, BYTE PTR [rax+36]        test    al, al        je      .L61

if (!frame->needs_free) return;

        mov     rax, QWORD PTR [rbp-56]        mov     esi, 56        mov     rdi, rax        call    "operator delete(void*, unsigned long)"        jmp     .L61

delete 56 byte frame and return

`.L61

.L61:        mov     rbx, QWORD PTR [rbp-8]        leave        ret

returns to prev part of stack (which was the tail of destroy wrapper calling actor in destroy mode)

Destroy wrapper tail

        nop        leave        ret

Main post-loop

        mov     eax, 0        leave        ret

And that's it!

Syntax

there is too much syntax to cover best to just go through assembly code for a concrete example of what goes on

Not to be confused w/

std::future, std::promise

Notes

mov A B = let A = B
rdi = first argument
rsi = second argument
rdx = third argument
rax = return value
co_await protocol
- determine its awaiter type
- evaluate await_ready
- possibly evaluate await_suspend
- eventually evaluate await_resume
std::coroutine_handle<MyPromise>
- a typed handle
- additionally carries, at compile time, the assertion
  - promise object inside has type MyPromise, enables .promise()
- has implicit conversion to untyped coroutine
std::coroutine_handle<void>
- type erased handle
- means "I do not know promise type", does not mean it returs void
std::coroutine_handle<>
- same as above
basically

coroutine_handle<P>
    = frame identity
    + resume/destroy capability
    + compile-time permission to access P

coroutine_handle<>
    = frame identity
    + resume/destroy capability
    - knowledge of the promise type

typed → untyped
    = preserve the exact same frame address
    + discard promise-type knowledge

The gist of how this example program works is that we pass in a slot for a coroutine handle. The function sets the value at that pointer to its own coroutine frame when it suspends. So the next time we call the handle, it calls the routine as defined at the coroutine frame of the function.

Notes on C++ Coroutines

Architecture

Definitions

What the compiler sees/does

co_await

An example program

Walkthrough:

Enters main function

Set h to nullptr

Load address of h

Call coroutine factory

Function prologue

Save argument to local stack

Malloc the frame for the coroutine

Store the pointer of frame into stack

Compile bookkeeping

Store pointers to continue and destroy functions

Copy source argument into frame

Initialize frame logistics

Call get return object

Call the actor for the first time

First actor call

Test low bit of state (for destroy fxn call)

Odd destroy fxn call jump

Even resume fxn call jump

.L43 first entry

.L39 invalid state trap

.L45

.L46 second entry (after initial suspend)

.L50 execute co_await a

Await suspend

Back to .L50

.L49

.L61:

Now we're back at the counter wrapper again

.L23

.L27

We're now back in main land

.L67 loop body

We enter frame w/ state = 4

After the end of the loop, call the destroy function on the coroutin frame

Destroy function for coroutine

Entering counter actor in destroy mode

Coroutine destructor (for real this time) at .L35

`.L61

Destroy wrapper tail

Main post-loop

Syntax

Not to be confused w/

Notes

Sources

`.L43` first entry

`.L39` invalid state trap

`.L45`

`.L46` second entry (after initial suspend)

`.L50` execute `co_await a`

Back to `.L50`

`.L49`

`.L61:`

`.L23`

`.L27`

`.L67` loop body

Coroutine destructor (for real this time) at `.L35`