Pushing the limits of RISC-V emulation

Lately, I’ve been working on OpenVM, a RISC-V virtual machine we’re building at Axiom. It runs a program and, along with the output, produces a succinct cryptographic proof¹ that the execution was correct. During my time at Axiom, proof generation has scaled from a single CPU to clusters of GPUs by distributing the workload across multiple nodes. However, the first step remains unchanged: we still have to run the program and record its execution. The proving phase is embarrassingly parallel and scales well with additional hardware, whereas execution is inherently sequential. As proving continues to scale, execution speed increasingly becomes the bottleneck, and this post is about making that step faster.

When I started looking into execution, I kept coming back to a simple question: how fast should a program run? OpenVM runs RISC-V programs compiled from normal languages like Rust, so I could compile the same program natively for my machine and use that as a baseline. On an M3 MacBook, the native arm64 program finished in about 100 milliseconds, while the same program took three or four seconds through our interpreter. I expected the virtual machine to be slower, since running through it necessarily adds overhead, but not by orders of magnitude.

The fastest way to run a program is to get out of its way and let the hardware execute it directly. A virtual machine cannot simply hand the program to the hardware because it has to maintain control over the execution and bridge the gap between the guest² and the host. Instead, it either interprets the program or translates it into instructions the host can understand. The rest of this post explores how far we can push this idea, starting with a basic interpreter and progressively removing the layers of overhead between the guest and the host.

The interpreter baseline

A straightforward way to run RISC-V³ code is with an interpreter that loops through the classic fetch-decode-execute cycle.

while (running) {
    uint32_t word = fetch(memory, pc);
    Instruction inst = decode(word);
    pc += 4;

    switch (inst.opcode) { // dispatch + execute
    case OP_ADDI:
        write_reg(inst.rd, regs[inst.rs1] + inst.imm);
        break;

    case OP_ADD:
        write_reg(inst.rd, regs[inst.rs1] + regs[inst.rs2]);
        break;

    // ...
    }
}

L_dispatch:
    ldr     w9, [x20, x19]             // fetch the guest word
    add     x19, x19, #4               // advance the guest pc
    bl      decode                     // decode its fields
    ldr     x10, [x21, x0, lsl #3]     // select the opcode handler
    br      x10                        // dispatch

h_add:
    ldr     x9, [x22, #t0_offset]
    ldr     x10, [x22, #t1_offset]
    add     x11, x9, x10
    str     x11, [x22, #t2_offset]
    b       L_dispatch                 // return to the loop

The problem is that guest instructions are often tiny. An add or a shift may only perform a single arithmetic operation, while the interpreter has to fetch the instruction, decode its fields, select the right case in the switch, and maintain its own execution state along the way. The interpreter can end up spending more time managing instructions than executing them.

+--------------+----------+----------+--------+----------+--------------+
|    funct7    |   rs2    |   rs1    | funct3 |    rd    |    opcode    |
+--------------+----------+----------+--------+----------+--------------+
|    31:25     |  24:20   |  19:15   | 14:12  |   11:7   |     6:0      |
+--------------+----------+----------+--------+----------+--------------+

+--------------+----------+----------+--------+----------+--------------+
|    funct7    |   rs2    |   rs1    | funct3 |    rd    |    opcode    |
+--------------+----------+----------+--------+----------+--------------+
|    0000000   |  00111   |  00110   |  000   |  00101   |    0110011   |
+--------------+----------+----------+--------+----------+--------------+

Predecoding ahead of time

Decoding the same instructions every time they execute is unnecessary. Program code is usually static⁴, so we can decode each instruction once up front and reuse the result. This is predecoding. For each instruction we store what it does, its operands, and a pointer to the small function (handler) that executes it.

typedef struct Decoded {
    void (*handler)(State *, const struct Decoded *);
    uint8_t rd, rs1, rs2;
    int32_t imm;
} Decoded;

void h_addi(State *state, const Decoded *inst) {
    state->regs[inst->rd] = state->regs[inst->rs1] + inst->imm;
}

while (running) {
    const Decoded *inst = fetch(code, state->pc);
    state->pc += 4;

    inst->handler(state, inst); // dispatch, execute
}

L_loop:
    ldr     x19, [x20, #pc_offset]      // read the guest pc
    lsr     x9, x19, #2
    ldr     x10, [x21, x9, lsl #4]      // fetch decoded handler
    add     x19, x19, #4
    str     x19, [x20, #pc_offset]      // publish the next pc
    blr     x10                         // call handler
    b       L_loop                      // return to the loop

h_add:
    ldr     x9, [x20, #t0_offset]
    ldr     x10, [x20, #t1_offset]
    add     x11, x9, x10
    str     x11, [x20, #t2_offset]
    ret                                 // return to the loop

Decoding is no longer part of the hot loop. What remains is dispatch through an indirect call to the handler pointer, followed by a return back to the interpreter loop for every instruction.

Note: The switch interpreter jumps directly to the instruction implementation, while the predecoded interpreter requires a function call and return for every instruction.

Dispatch via computed goto

Predecoding removed the cost of decoding. The remaining bottleneck is the indirect dispatch. Every instruction still has to return to a shared loop so the interpreter can select what runs next. A classic fix for this is a computed goto⁵. Instead of returning to a shared loop, each handler looks up the next opcode’s label in a table and jumps directly to it with a goto *.

The predecoded representation is simpler here than before. There is no handler pointer, just the same Instruction from the first section, decoded once into an array indexed by pc / 4. Each opcode gets its own label as a dispatch target.

Instruction prog[NSLOT]; // decoded once, indexed by pc / 4

void *targets[OP_COUNT];
targets[OP_ADDI] = &&do_addi;
targets[OP_ADD] = &&do_add;
// ...

Instruction inst = fetch(prog, state->pc);
goto *targets[inst.opcode];

do_addi:
    state->regs[inst.rd] = state->regs[inst.rs1] + inst.imm;
    inst = fetch(prog, state->pc += 4);
    goto *targets[inst.opcode];

do_add:
    state->regs[inst.rd] = state->regs[inst.rs1] + state->regs[inst.rs2];
    inst = fetch(prog, state->pc += 4);
    goto *targets[inst.opcode];

// ...

L_dispatch:
    lsr     x9, x19, #2
    ldr     w9, [x20, x9, lsl #4]      // fetch decoded instruction
    ldr     x10, [x21, x9, lsl #3]     // look up its label
    br      x10                        // computed goto

h_add:
    ldr     x9, [x22, #t0_offset]
    ldr     x10, [x22, #t1_offset]
    add     x11, x9, x10
    str     x11, [x22, #t2_offset]
    add     x19, x19, #4               // advance the guest pc
    b       L_dispatch                 // fetch and dispatch the next one

Each instruction still has to be fetched and dispatched, but the interpreter no longer performs a function call and return for every handler. Instead, each handler transfers control to the dispatch sequence, which selects the next handler and jumps directly to it. Execution therefore moves from one handler to the next through indirect jumps rather than repeated call-and-return boundaries.

Tail-call threading

Tail calls⁶ provide another way to thread handlers together while keeping each handler as a normal function. A handler can tail-call the next one instead of returning to a shared dispatch loop:

[[gnu::always_inline]] static inline void dispatch(State *state, const Decoded *inst) {
    const Decoded *next = fetch(code, state->pc);
    [[clang::musttail]]
    return next->handler(state, next);
}

void h_addi(State *state, const Decoded *inst) {
    state->regs[inst->rd] = state->regs[inst->rs1] + inst->imm;
    state->pc += 4;
    dispatch(state, inst);
}

void h_add(State *state, const Decoded *inst) {
    state->regs[inst->rd] = state->regs[inst->rs1] + state->regs[inst->rs2];
    state->pc += 4;
    dispatch(state, inst);
}
// ...

const Decoded *inst = fetch(code, state->pc);
inst->handler(state, inst);

h_add:
    ldr     x9, [x20, #t0_offset]
    ldr     x10, [x20, #t1_offset]
    add     x11, x9, x10
    str     x11, [x20, #t2_offset]
    ldr     x19, [x20, #pc_offset]
    add     x19, x19, #4
    str     x19, [x20, #pc_offset]      // publish the next pc
    lsr     x13, x19, #2
    ldr     x12, [x21, x13, lsl #4]     // fetch the next handler
    br      x12                         // tail jump, no return

Each handler executes its instruction, moves pc forward, and calls an inlined dispatch helper, which finds the next handler and tail-calls it. Since nothing in the caller runs afterward, the compiler can lower the call to a plain jump without retaining the caller’s stack frame. This is tail-call elimination, and it allows a chain of handlers to run indefinitely at constant stack depth. Clang’s musttail attribute makes this lowering mandatory, emitting a tail call or rejecting the program.

Computed goto and tail-call threading are two ways to express the same handler-to-handler control flow. One uses labels inside a function, while the other uses separate functions and relies on mandatory tail calls.

Keeping state in registers

Every handler still reaches through State * for pc, reading and writing it on every instruction. Since these values are already passed through the tail-call chain, they can be passed as function arguments instead. The host calling convention typically assigns those arguments to registers⁷, allowing the compiler to keep frequently used values in host registers rather than repeatedly loading and storing them through State:

typedef void (*handler_fn)(State *, const Decoded *, uint64_t pc, uint8_t *mem);

[[gnu::always_inline]] static inline void dispatch(
    State *state, const Decoded *inst, uint64_t pc, uint8_t *mem
) {
    const Decoded *next = fetch(code, pc);
    [[clang::musttail]]
    return next->handler(state, next, pc, mem);
}

void h_add(State *state, const Decoded *inst, uint64_t pc, uint8_t *mem) {
    state->regs[inst->rd] = state->regs[inst->rs1] + state->regs[inst->rs2];
    dispatch(state, inst, pc + 4, mem);
}

h_add:
    ldr     x9, [x20, #t0_offset]
    ldr     x10, [x20, #t1_offset]
    add     x11, x9, x10
    str     x11, [x20, #t2_offset]
    add     x19, x19, #4                // pc stays in a register
    lsr     x13, x19, #2
    ldr     x12, [x21, x13, lsl #4]     // fetch the next handler
    br      x12                         // tail jump

Now pc can stay in a register throughout the chain instead of being loaded from and stored back to State on every instruction. The same idea applies to other frequently accessed interpreter state such as mem.

Going native

Predecoding removed repeated decoding, and threaded dispatch removed the shared loop. What remained was still an interpreter, with every guest instruction passing through a handler before it could run. That overhead still left a substantial gap to native execution. Since the program was already known ahead of time, the next step was to translate it directly into something the host could execute.

A hand-written assembly backend

A simple ahead-of-time (AOT)⁸ translator works one instruction at a time. Each RISC-V instruction is translated into a short sequence of host instructions, and the generated pieces are stitched together with the required control flow. The translation is almost mechanical: an add becomes a host addition, a load becomes a host load, and so on. Sequential instructions can fall through naturally, and jumps to fixed targets become ordinary host jumps. Jumps whose targets are computed at runtime require a lookup through a dispatch table.

The translator emits assembly for the host architecture in a single linear pass, after which the generated code only needs to be assembled into a shared object and loaded. The tradeoff is portability - since instruction encodings are architecture-specific, each target requires its own backend. In practice, the translator is a switch statement with one case per RISC-V opcode, each writing the corresponding host assembly.

switch (inst.op) {
case OP_ADD: {
    const char *src1 = src_x_arm(out, inst.rs1, T1);  // read rs1
    const char *src2 = src_x_arm(out, inst.rs2, T2);  // read rs2
    const char *dst = dst_x_arm(inst.rd, T0);         // rd's register
    fprintf(out, "\tadd %s, %s, %s\n", dst, src1, src2);
    commit_x_arm(out, inst.rd, dst);  // write rd back to state
    break;
}
// ...

case OP_JAL: {
    if (inst.rd) {
        // rd = pc + 4, the return address
        const char *dst = dst_x_arm(inst.rd, T0);
        mov_imm_arm(out, dst, pc + 4);
        commit_x_arm(out, inst.rd, dst);
    }
    // the target is fixed, so this is a direct branch to a label
    branch_arm(out, "", pc + (uint64_t)inst.imm, lo, hi);
    break;
}
// ...
}

L_1000:  ; pc 0x1000  add t2, t0, t1
    ldr x9, [x28, #40]   ; t0
    ldr x10, [x28, #48]  ; t1
    add x8, x9, x10      ; t2 = t0 + t1
    str x8, [x28, #56]
L_1004:  ; pc 0x1004  jal ra, 8
    ; ra = pc + 4, the return address
    movz x8, #4104
    str x8, [x28, #8]
    ; the target (0x100c) is fixed, so this is a direct branch
    b L_100c
; ...
L_100c:  ; pc 0x100c  ecall
    b Lhalt

This removes the remaining interpreter overhead from the hot path. Execution no longer incurs handler dispatch, and the generated host instructions run directly, with sequential execution falling through naturally and fixed jumps becoming ordinary host branches.

Register mapping

The remaining interpreter cost is moving guest state between memory and registers. A simple translator still reads and writes every guest register through State, just as the interpreter did.

The same idea used earlier applies here. Values that are carried through execution do not need to live in memory. Instead of keeping the guest register file inside State, the translator can map frequently used guest registers to host registers, loading them once at entry and writing them back when execution exits.

static const char *const POOL_X_ARM[NHOT] = {"x1", "x2", "x3", /* ... */ "x30"};
static const int HOT_REG[NHOT] = {1, 2, 10, /* ... */ 21};

// Mapping from guest register to host register
const char *host_x_arm(int g) {
    for (int i = 0; i < NHOT; i++) {
        if (HOT_REG[i] == g) {
            return POOL_X_ARM[i]; // g is pinned to POOL_X_ARM[i]
        }
    }
    return nullptr; // g is cold
}

// ...

const char *src_x_arm(FILE *out, int g, const char *tmp) {
    const char *h = host_x_arm(g);
    if (h) {
        return h;
    }
    fprintf(out, "\tldr %s, [x28, #%d]\n", tmp, g * 8);
    return tmp;
}

run_guest:
    mov x28, x0
    ; other hot registers are loaded here
    ldr x3, [x28, #80]      ; a0, loaded once at entry
    ldr x4, [x28, #88]      ; a1, loaded once at entry
    b L_1000

L_1000:                     ; add a0, a0, a1
    add x3, x3, x4          ; no register-file loads or stores

Lhalt:
    ; other hot registers are stored here
    str x3, [x28, #80]      ; a0, written back once at exit

The loads and stores move to the boundaries of translated execution. Inside the hot path, the guest add becomes a single host instruction.

Letting Clang be the backend

At this point, the translator is effectively reimplementing a compiler backend by hand. It selects host instructions, assigns registers, and maintains a separate backend for each target architecture. A more natural approach is to generate code in a language that an off-the-shelf compiler already understands. The translator can then leave instruction selection, register allocation, and other target-specific details to Clang.

One giant function

The most direct way to use a compiler is to make the entire program one function. Decode the program, translate each instruction into C, keep frequently used guest registers as local variables, and represent the control flow inside that function with labels and goto. This gives the compiler the entire program as context. It can track values across translated instructions, decide where registers should live, and optimize across the generated code.

Control flow is represented directly inside the function. Straight-line execution falls through, fixed targets become labels, and only targets unknown until runtime need an indirect jump table.

for (uint64_t pc = lo; pc <= hi; pc += 4) {
    if (!reachable(pc)) continue;

    // every reachable instruction gets a label
    fprintf(out, "L_%llx:\n", pc);

    switch (inst.op) {
    case OP_ADD:
        // fall-through: no goto emitted at all
        fprintf(out, "\t%s = %s + %s;\n", reg_name(inst.rd), reg_name(inst.rs1), reg_name(inst.rs2));
        break;
    case OP_JAL:
        // static: known target, a plain goto
        if (inst.rd != REG_ZERO)
            fprintf(out, "\t%s = %lluULL;\n", reg_name(inst.rd),
                    (unsigned long long)(pc + 4));
        fprintf(out, "\tgoto L_%llx;\n", (unsigned long long)(pc + inst.imm));
        break;
    case OP_JALR:
        // dynamic: computed goto
        fprintf(out, "\tuint64_t target = (%s + (int64_t)%lld) & ~1ULL;\n",
                reg_name(inst.rs1), (long long)inst.imm);
        if (inst.rd != REG_ZERO)
            fprintf(out, "\t%s = %lluULL;\n", reg_name(inst.rd),
                    (unsigned long long)(pc + 4));
        fprintf(out, "\tstate->pc = target;\n");
        fprintf(out, "\tuint64_t index = __builtin_rotateright64(target - TEXT_START, 2);\n");
        fprintf(out, "\tif (index >= NSLOT) goto Lret;\n");
        fprintf(out, "\tgoto *tbl[index];\n");
        break;
    // ...
    }
}

void run_guest(State *state) {
    // Abbreviated hot-register set.
    uint64_t a0 = state->regs[REG_A0];
    uint64_t a1 = state->regs[REG_A1];

    // one entry per instruction
    void *tbl[NSLOT];
    for (size_t i = 0; i < NSLOT; i++) {
        tbl[i] = &&Lret;
    }
    tbl[(0x1000 - TEXT_START) / 4] = &&L_1000;
    tbl[(0x1004 - TEXT_START) / 4] = &&L_1004;
    tbl[(0x1008 - TEXT_START) / 4] = &&L_1008;
    tbl[(0x100c - TEXT_START) / 4] = &&L_100c;
    tbl[(0x1014 - TEXT_START) / 4] = &&L_1014;
    // ...

    uint64_t entry = __builtin_rotateright64(state->pc - TEXT_START, 2);
    if (entry >= NSLOT) goto Lret;
    goto *tbl[entry];

L_1000: // pc 0x1000  addi a0, a0, 1
    a0 = a0 + 1;
L_1004: // pc 0x1004  slli a0, a0, 2
    a0 = a0 << 2;
L_1008: // pc 0x1008  add a0, a0, a1
    a0 = a0 + a1;
L_100c: // pc 0x100c  jal zero, 8
    goto L_1014;

// ...

L_1014: // pc 0x1014  jalr zero, a1, 0
    uint64_t target = a1 & ~1ULL;
    state->pc = target;
    uint64_t index = __builtin_rotateright64(target - TEXT_START, 2);
    if (index >= NSLOT) goto Lret;
    goto *tbl[index];

Lret:
    state->regs[REG_A0] = a0;
    state->regs[REG_A1] = a1;
    return;
}

run_guest:
    adr     x16, Ltable                 // materialize the dispatch table
    ldr     x10, [x0, #regs_a0]         // load register locals
    ldr     x12, [x0, #regs_a1]
    ldr     x9, [x0, #state_pc]         // enter through the guest pc
    sub     x9, x9, #0x1000             // compact table starts at TEXT_START
    ror     x9, x9, #2                  // divide by four and reject misalignment
    cmp     x9, #NSLOT
    b.hs    Lret                        // reject an out-of-range target
    ldr     x14, [x16, x9, lsl #3]
    br      x14

L_1000:
    add     x10, x10, #1                // addi a0, a0, 1
L_1004:
    lsl     x10, x10, #2                // slli a0, a0, 2
L_1008:
    add     x10, x10, x12               // add a0, a0, a1
L_100c:
    b       L_1014                      // fixed target is a direct branch

L_1014:
    and     x13, x12, #-2               // compute jalr target from a1
    str     x13, [x0, #state_pc]
    sub     x13, x13, #0x1000           // compact table starts at TEXT_START
    ror     x13, x13, #2                // divide by four and reject misalignment
    cmp     x13, #NSLOT
    b.hs    Lret                        // reject an out-of-range target
    ldr     x14, [x16, x13, lsl #3]
    br      x14

Lret:
    str     x10, [x0, #regs_a0]         // publish register locals
    str     x12, [x0, #regs_a1]
    ret

Ltable:
    .xword  L_1000, L_1004, L_1008, L_100c, Lret, L_1014

For small programs, this gives the compiler broad visibility across the guest program. Register allocation and optimization can track every value from entry to exit without boundaries between translated regions.

But the approach breaks down as programs grow. Compilers handle small functions well, but a function containing thousands or millions of lines gives Clang one enormous body to analyze and optimize. Compilation can then take an impractical amount of time, and because all the work sits in a single function, it cannot be distributed effectively across cores.

Recompiling one basic block at a time

Translating the entire program at once gives Clang too much to analyze, while translating one instruction at a time would give it too little context. A basic block provides a useful middle ground.

A basic block is a maximal sequence of straight-line instructions with a single entry. Control can enter only at the first instruction, and control can leave only at the end.

The recompiler emits each basic block as a separate C function. This gives Clang enough context to optimize across multiple guest instructions while keeping each generated function small enough to compile efficiently.

For example, the block

# pc 0x1000
slli a0, t0, 13
xor  t0, t0, a0
add  a1, t0, a2

becomes:

typedef void (*block_fn)(State *);

void B_100c(State *state);

void B_1000(State *state) {
    state->regs[REG_A0] = state->regs[REG_T0] << 13;
    state->regs[REG_T0] ^= state->regs[REG_A0];
    state->regs[REG_A1] = state->regs[REG_T0] + state->regs[REG_A2];

    [[clang::musttail]]
    return B_100c(state);
}

ldr x23, [x0, #regs_t0]        // load t0 from State
lsl x9, x23, #13               // compute a0
str x9, [x0, #regs_a0]         // store a0 in State
eor x10, x23, x23, lsl #13     // t0 ^ (t0 << 13), one host instruction
str x10, [x0, #regs_t0]        // store t0 in State
ldr x24, [x0, #regs_a2]        // load a2 from State
add x11, x10, x24              // the remaining guest operation
str x11, [x0, #regs_a1]        // store a1 in State
b   B_100c                     // known successor, no dispatch lookup

The important difference is that the compiler sees the entire sequence as one unit rather than three independent instructions. It can keep intermediate values in registers and combine several guest operations into one host instruction. Here, the shift and xor fold into a single arm64 instruction with a built-in shifted operand.

Identifying blocks

Given an ELF executable⁹ containing a sequence of instructions, the recompiler must determine where each generated function begins and ends. It does this by identifying basic blocks and connecting them into a control-flow graph (CFG) where each node represents one block, and each edge represents a possible transfer of control.

An instruction that may begin a basic block is called a block leader. The program entry point is a leader, as is the target of any direct branch or jump. The instruction following a control-flow instruction is also a leader when execution may fall through to it. Analysis may identify additional leaders by recovering targets that are not encoded directly in the instruction, such as a jalr target determined through constant propagation.

Starting from each leader, the recompiler decodes forward until it reaches another leader or an instruction that may redirect control, such as a branch, jump, call, return, or trap. This keeps every identified jump target at the beginning of a generated function.

This process may leave parts of the executable text undiscovered because no known control-flow edge reaches them. To ensure that those regions can still be compiled, the recompiler also scans the remaining text linearly and divides it into blocks, starting new ones at section boundaries, after control-flow instructions, and at any direct targets it encounters.

Once the instruction ranges are known, the recompiler connects them according to their successors. A conditional branch has a taken successor and a fall-through successor, while a direct jump has only its target. These successors become edges in the CFG.

The example below applies these rules to a function that sums an array. The conditional branch in B_1008 ends one block and produces two successors, while the direct jump in B_100c forms the loop back edge. These transfers encode their targets directly. The final jalr reads its destination from ra, so its target is known only at runtime.

void B_1000(State *state) {
    state->regs[REG_T0] = 0;
    state->regs[REG_T1] = 0;

    [[clang::musttail]]
    return B_1008(state);
}

void B_1008(State *state) {
    if ((int64_t)state->regs[REG_T1] >= (int64_t)state->regs[REG_A1]) {
        [[clang::musttail]]
        return B_102c(state);
    }

    [[clang::musttail]]
    return B_100c(state);
}

void B_100c(State *state) {
    int64_t v = (int32_t)load32(state, state->regs[REG_A0]);
    state->regs[REG_T0] += (uint64_t)v;
    state->regs[REG_A0] += 4;
    state->regs[REG_T1] += 1;

    [[clang::musttail]]
    return B_1008(state);
}

void B_102c(State *state) {
    state->regs[REG_A0] = state->regs[REG_T0];

    uint64_t target = state->regs[REG_RA] & ~1ULL;
    state->pc = target;
    uint64_t index = __builtin_rotateright64(target - TEXT_START, 2);
    if (index >= NSLOT) {
        state->halted = 1;
        return;
    }
    block_fn next = dispatch_table[index];
    [[clang::musttail]]
    return next(state);
}

Static analysis can still recover many indirect targets. It marks return addresses as reachable, propagates constants through auipc + jalr sequences used for distant calls, and scans read-only data for pointers into executable code, including function-pointer arrays and switch jump tables.

Indirect-target analysis remains conservative, however. It records a target only when there is evidence that execution may begin there. The whole-text sweep adds blocks for coverage, but targets known only at runtime remain unresolved in the static CFG and use the runtime dispatch path.

Once the blocks and their successors are known, direct branches and jumps can become tail calls to the corresponding generated functions.

Connecting blocks

A block transfers control only at its end. Direct branches and jumps have statically known destinations, so the generated function can end with a direct tail call:

if (taken) {
    [[clang::musttail]]
    return B_taken(state);
}

[[clang::musttail]]
return B_fallthrough(state);

Clang can lower these calls to ordinary jumps. No dispatch lookup or explicit program-counter update is needed along a direct edge. Executing B_1000 already identifies the current guest address as 0x1000.

A jalr, including a function return, computes its destination from a register at runtime and therefore uses the dispatch table:

state->pc = target_pc;

uint64_t index = __builtin_rotateright64(target_pc - TEXT_START, 2);
if (unlikely(index >= NSLOT)) {
    state->halted = true;
    return;
}

block_fn next = dispatch_table[index];
[[clang::musttail]]
return next(state);

Before accessing the table, the dispatch path validates that the target is aligned and falls within the dispatchable text; otherwise, execution halts. The table has one entry for each instruction slot in the text section. Entries for block leaders point to generated functions, while the remaining entries use a fallback.

The fallback handles targets that are valid instruction addresses but not block leaders, such as an instruction in the middle of a generated block. In that case, the dispatch table invokes the interpreter at the target address. The interpreter continues until execution reaches a compiled block, then transfers control back to generated code.

Registers across blocks

Spilling all 32 guest registers at the end of one block and reloading them in the next would add substantial memory traffic to every transfer. The block recompiler therefore applies the same idea used by the tail-call interpreter. Each block receives a fixed set of frequently used guest registers as arguments so those values can remain in host registers:

typedef uint64_t reg_t;

// Abbreviated hot-register set.
[[clang::preserve_none]]
void B_100c(State *, reg_t ra, reg_t sp, reg_t a0, reg_t a1);

[[clang::preserve_none]]
void B_1000(State *state, reg_t ra, reg_t sp, reg_t a0, reg_t a1) {
    a0 = a0 + 1;
    a0 = a0 << 2;
    a0 = a0 + a1;

    [[clang::musttail]]
    return B_100c(state, ra, sp, a0, a1);
}

The arguments include the State pointer and a shared set of hot guest registers, while less frequently used registers remain in State.

An ordinary calling convention exposes only a limited number of argument registers.¹⁰ Clang’s preserve_none convention makes nearly every general-purpose register available and requires the callee to preserve none of them.

This would normally force the caller to save any live values before a call. A block transfer is a musttail call, however, so the caller never resumes and has nothing to preserve. Together, preserve_none and musttail make far more host registers available for carrying guest state between blocks.

The x86-64 implementation leaves one preserve_none argument register unused to avoid exhausting LLVM’s register allocator during link-time optimization. After accounting for the State pointer, this leaves room for roughly 23 hot guest registers on arm64 and 10 on x86-64.

Every block uses the same hot-register set because musttail requires compatible signatures and matching calling conventions. Registers that a block does not modify pass through without additional moves. The recompiler chooses this set, while Clang assigns it to the available host registers.

With these pieces in place, the design is complete. Basic blocks give the compiler enough context to optimize across instructions, tail calls turn known control flow into direct jumps, and a dense jump table handles unresolved targets. Combined with preserve_none, this keeps frequently used guest state in host registers across block boundaries.

Performance

The benchmarks measure how each step affects preparation time, execution throughput, and generated shared-object size.

Benchmark setup

Within each workload, all guest backends run the same RISC-V binary on the same host. The native baseline is compiled separately from equivalent C. Every version produces the same output, and the timed path contains no instruction counting or other bookkeeping. Preparation is measured end to end for a freshly generated artifact and includes code generation, compilation or assembly, loading the generated shared object, and initialization. Throughput is calculated from the minimum runtime across repeated executions to reduce transient noise. It is reported in millions of guest instructions per second (MIPS), rounded, and is best read as ratios within each workload.

The benchmark programs and inputs¹¹ are trusted, so guest memory accesses use a guarded linear mapping without per-access bounds checks. However, the generated backends still check indirect control-flow targets for range and alignment before dispatch.

Small workloads

The first chart combines three hand-written loops with eight C programs compiled for RISC-V.

Preparation time

All three backends finish these workloads in under a second. The single-function backend becomes the most expensive as the generated function grows, reaching about 600 milliseconds on matrix multiplication.

Throughput

The native host C row runs a separately compiled host version of the same workload. Its throughput is reported in equivalent guest MIPS.¹² The loops have no native result because they have no corresponding C implementation.

The cleanest comparison is between per-instruction, pinned and recompiler, preserve_none. Both keep the same hot guest registers in hardware, so the remaining difference should mainly reflect the compiler’s ability to optimize across a basic block rather than one instruction at a time. The difference is small on tightly serial code and much larger when the compiler can combine or vectorize operations.

preserve_none helps most when values cross block boundaries frequently. It performs well on recursive workloads, but can underperform, likely because the wide argument set increases register pressure inside a tight loop.

On this Ackermann benchmark, the translated code exceeds the equivalent native C baseline. Its guest calls and returns become tail transfers, so recursion grows the guest stack without repeatedly creating and removing host stack frames.

Note: These are small benchmarks from a minimal reference implementation, chosen to isolate individual mechanisms rather than represent performance on a real workload.

Larger workloads

The larger benchmarks are four C workloads compiled for RISC-V and linked bare-metal, with no operating system, system calls, or floating point.

Preparation time

The per-instruction translator remains the cheapest because it emits fixed instruction sequences and assembles them. The block recompiler spends more time optimizing tens of thousands of small functions. The generated functions are split across files and compiled in parallel, while the final link uses ThinLTO to allow optimization across file boundaries.¹³

The runner gives the single-function backend up to ten minutes for each workload. CoreMark finishes in 30 seconds. LZ4, zstd, and SQLite do not finish compiling within the limit, so this backend is absent from their throughput and code-size results.

Throughput

On CoreMark, LZ4, and zstd, the per-block recompiler reaches roughly 84 to 89% of native throughput. On SQLite, it reaches about 40%. Native Clang can optimize across complete functions and reason directly about source-level pointers. The recompiler only sees one basic block at a time, and guest memory accesses must first be translated into host addresses, leaving it with fewer opportunities to optimize.

SQLite also suffers from incomplete static coverage. Its bytecode interpreter and runtime-registered function pointers contain many destinations that static analysis cannot recover.¹⁴ Without broad code coverage, execution falls back to interpretation more often. Compiling the full text section improves that coverage, although SQLite’s short and irregular blocks leave less room for optimization than LZ4 or zstd.

Note: The RISC-V builds of LZ4 and zstd use custom memcpy and memmove implementations, while the native baselines use the host versions.

Code size

The per-instruction translator produces the smallest object because it emits one fixed, unoptimized template per guest instruction. The per-block recompiler lets Clang inline, unroll, and align the same code, making its output roughly 2 to 2.7 times as large. The single-function CoreMark object is about twice the size of the per-instruction translator’s output.

The per-block SQLite object reaches about 10 MiB, so larger translated programs must balance this growth against the throughput gained from optimization.

Tradeoffs

Basic-block recompilation works best when guest programs are known ahead of time and run long enough, or often enough, to amortize compilation. Regular, compute-heavy workloads benefit most because they give the compiler more work to optimize within each block.

It is less suitable for short-lived programs, frequently changing code, or workloads whose indirect control flow repeatedly falls back to interpretation. Compared with simpler translation, it improves throughput at the cost of longer preparation, larger artifacts, and greater dependence on static control-flow coverage.

Closing thoughts

There is still a large design space between interpretation and compiling the entire program eagerly. Hot basic blocks could be combined into larger regions, lighter optimization could reduce compilation time and code size, and unresolved targets could trigger JIT compilation instead of falling back to the interpreter. The approach could then be extended to compile more of the program on demand, avoiding the full upfront cost of AOT compilation.

The core techniques described in this post are not new in isolation. What I had not seen was this combination of musttail, preserve_none, and basic-block recompilation used to run guest programs efficiently. Together, they make the design both fast and portable across host architectures.

Resources

• Computed goto for efficient dispatch tables
• Efficient interpreters with musttail
• Clang documentation for musttail and preserve_none
• RVR, the initial recompiler implementation in Rust

OpenVM v2.1 will include the recompiler as an optional execution backend.