Methodology

The evaluation compares three ways to bring up 2x2 and 4x4 OpenPiton meshes of Ariane and PicoRV32. A run passes when Verilator simulation of the RTL reports Simulation -> PASS (HIT GOOD TRAP). The finish mask has one digit per tile, so that line appears only after every tile hits its good trap.

Approaches

Approach

What it does

Driver

(a) Manual

Builds and runs the configuration by hand through the adapter, with no LLM.

OpenPitonWorkspaceNode.build() and run() called directly

(b) One-shot LLM

Asks the LLM once for a configuration, then builds and runs it, with no tools and no retry.

examples/baseline_one_shot_llm.py

(c) MACE loop

Plans a task graph, runs it level by level, gates each task on simulation, and replans on failure.

examples/mace_end_to_end.py --mesh <X>x<Y>

All LLM calls use Gemini 2.5 Flash on Vertex AI.

Workloads

  • Ariane runs barrier_atomic.c: all harts meet at a barrier and add to a shared atomic counter.

  • PicoRV32 runs addi.S. PicoRV32 explains why it runs only assembly workloads.

  • OpenSPARC T1 is left out. OpenSPARC T1 explains why.

PicoRV32 needs the CONFIG_DISABLE_BIST_CLEAR RTL define, and all three approaches get it: (a) sets it by hand, and the objectives for (b) and (c) name it. The one-shot config line has an optional config_rtl field, and the loop’s planner requests the define through a CONFIG_RTL: line.

Time basis

All approaches share one build cache, so an approach pays for a build only when it picks a new configuration. Times are end-to-end wall time, including LLM calls for (b) and (c). Manual times count machine time only.

Environment

  • One job at a time on a single OpenPiton checkout, patched with fixes 1 to 12 of scripts/patch_openpiton.sh, the fixes the 0.0.1 pre-release had.

  • MAKEFLAGS=-j1 for all builds, as Ariane (CVA6) explains.

  • Verilator 5.020 with --no-timing. The Nix shell pins Verilator 5.052.

  • The one-shot baseline allows 15000 s for a build and 3600 s for a run; the loop’s run limit is also 3600 s.