Note: Shadow Stack

(This idea is based on Nimrod's debugging facilities.)

We cannot integrate very well with GDB out of the box. We emit many local C variables that are of no interest to Flect programmers, stack traces will contain mangled names, types will not look natural, etc. Eventually, when the GDB developers reach their goal of making Python scripting able to add support for entirely new languages, we should be able to integrate properly with it. But for now, we need to roll our own debugging infrastructure.

So what we do to get some sort of debugging support is that we instrument programs with a shadow stack when they're compiled in debug mode. All local variables get an entry in the shadow stack which is kept up to date whenever a local variable changes. This shadow stack is then used to retrieve the state of the program when it is paused. It's also useful to construct stack traces (although not as effectively as e.g. libunwind would be able to).

At the entry of every function, we insert a local frame object which we link to the last frame object (stored in a TLS variable). We then initialize the object with some information about the function (name, containing module, source code location, address, etc). Then, we initialize the list of local variables and their types in the frame. Now, for every local variable reference (those that are not compiler-generated temporaries), we replace local_name with frame.local_name. This means that all locals live on the shadow stack.

Whenever the program is stopped (e.g. via the breakpoint function), the debugger API can trivially ask the shadow stack what the values of variables are, and even mutate them.

A stack frame object looks like this:

priv struct GenericShadowStackFrame {
    pub prev : *mut GenericShadowStackFrame;

    pub fn_name : str;
    pub fn_module : str;
    pub fn_file : str;
    pub fn_line : u32;
    pub fn_column : u32;
    pub fn_address : fn() -> unit;
}

For every function, a more specialized frame is generated:

priv struct ShadowStackFrame__func {
    pub base : GenericShadowStackFrame;

    // For every local, holds its name, type info, and a pointer to
    // the appropriate field emitted below.
    pub locals : [(str, &core::meta::TypeInfo, *mut u8) .. $num_locals];

    // For each local, emit:
    // pub local_$name : $type;
}

And we have a TLS variable in the core library:

priv mut tls last_frame : *mut GenericShadowStackFrame = null;

So let's look at an example translation of a function.

pub fn mul(i : i32, j : i32) -> i32 {
    let val = i * j;
    val;
}

This will be translated (roughly, and skipping mangling) to:

/* In the core library. */
__thread GenericShadowStackFrame *last_frame = 0;

/* In the executable. */
void mul(int i, int j) {
    struct ShadowStackFrame__mul {
        GenericShadowStackFrame base;

        struct {
            struct {
                const char *p;
                size_t l;
            } e1; /* Local name (pointer/length pair). */
            void *e2; /* Type info pointer. */
            void *e3; /* Local storage pointer. */
        } locals[1];

        int local_val;
    } frame;

    frame.base.parent = &last_frame;
    last_frame = &frame.base; /* Strictly speaking, could just be &frame. */

    frame.base.fn_name.p = "mul";
    frame.base.fn_name.l = 3;
    frame.base.fn_module.p = /* ... */;
    frame.base.fn_module.l = /* ... */;
    frame.base.fn_file.p = /* ... */;
    frame.base.fn_file.l = /* ... */;
    frame.base.fn_line = /* ... */;
    frame.base.fn_column = /* ... */;
    frame.base.fn_address = &mul;

    frame.locals[0].e1.p = "val";
    frame.locals[0].e1.l = 3;
    frame.locals[0].e2 = &/* ... type info symbol ... */;
    frame.locals[0].e3 = &frame.local_val;

    frame.local_val = i * j;

    return frame.local_val;
}

This does of course mean significantly larger stack frames in debug builds. Unfortunately, we can't do much about that.