Baselines

Two baselines run the same core and gate workload as the loop:

Baseline (a), manual mesh scaling: configure, build, and run a mesh by hand through the adapter.
Baseline (b), one-shot LLM: one prompt proposes a configuration, which is built and run once, with no tools, no verification loop, and no retry.

The full loop is approach (c). Methodology describes the comparison, and Results reports it. Evaluation Harness describes the larger set of baselines, ablations, and co-design searches the batch runner runs.

Baseline (a): manual mesh scaling

Baseline (a) drives OpenPitonWorkspaceNode directly:

import ray
from chia.base.ChiaFunction import get
from chia_openpiton.openpiton_workspace import OpenPitonWorkspaceNode
from mace.workloads import RECOMMENDED_RTL_TIMEOUT, WORKLOADS_DIR

ray.init(address="local", resources={"openpiton": 1})
node = OpenPitonWorkspaceNode("/home/you/openpiton")
try:
    cfg = get(node.configure.chia_remote(x_tiles=2, y_tiles=2, core="ariane"))
    art = get(node.build.chia_remote(cfg))
    res = get(node.run.chia_remote(
        cfg, "barrier_atomic.c",
        asm_diag_root=str(WORKLOADS_DIR), rtl_timeout=RECOMMENDED_RTL_TIMEOUT,
    ))
    print(art.success, res.verdict)
finally:
    node.close()
    ray.shutdown()

For PicoRV32, configure with core="pico" and config_rtl=("MINIMAL_MONITORING", "CONFIG_DISABLE_BIST_CLEAR"), and run "addi.S". examples/run_barrier_atomic.py wraps the Ariane case with --piton-root, --core (ariane or sparc), --x-tiles, and --y-tiles. It always runs barrier_atomic.c and rebuilds with clean=True.

Baseline (b): one-shot LLM

examples/baseline_one_shot_llm.py does not import mace.loop, mace.planner, or mace.orchestrator. It sends one prompt with no tools, parses one configuration from the reply, then configures, builds, and runs it through the adapter.

Flag	Default
`--piton-root`	required
`--core`	`ariane`; also `sparc` or `pico`
`--workload`	`barrier_atomic.c`
`--objective`	`Verify the barrier_atomic gate workload passes on a 1x1 mesh.`
`--backend`	`vertex`
`--model`	`gemini-2.5-flash` on `vertex`
`--project`	none; see LLM Backends

The script has no --mesh flag. The LLM chooses x_tiles and y_tiles, so state the mesh in --objective.

The CONFIG: line

The prompt asks for exactly one line in this format:

CONFIG: x_tiles=<int> | y_tiles=<int> | l1i_size=<bytes> | l1i_assoc=<int> | l1d_size=<bytes> | l1d_assoc=<int> | l15_size=<bytes> | l15_assoc=<int> | l2_size=<bytes> | l2_assoc=<int> | config_rtl=<comma-separated RTL defines to add, or none>

The script uses the last CONFIG: line in the reply. The config_rtl field is optional. The script adds the listed defines to the default MINIMAL_MONITORING, and a missing field or none adds none. Each define must match ^[A-Z][A-Z0-9_]*$. The script stops before any build and prints BASELINE (one-shot): FAILED TO PARSE A CONFIG when the reply has no CONFIG: line, a missing field, or a malformed define. A value that PitonConfig rejects, such as a cache size of 0, stops it the same way.

A parsed configuration builds with a 15000 s timeout, and the workload runs with rtl_timeout=RECOMMENDED_RTL_TIMEOUT. After the run, the script prints a summary with status, llm_calls: 1, execution_time_s, compute_usd (lower bound), and verdict. It exits with 0 only on a pass.

python examples/baseline_one_shot_llm.py \
    --piton-root ~/openpiton \
    --core pico \
    --workload addi.S \
    --objective "Verify the addi.S gate workload passes on a 2x2 pico mesh. The build needs the CONFIG_DISABLE_BIST_CLEAR RTL define."

Running (b) and (c) back to back

bash scripts/local_baselines_b_and_c_test.sh ~/openpiton

The script activates the chia_env conda environment from ~/miniconda3 or ~/anaconda3. It runs examples/baseline_one_shot_llm.py, then examples/mace_end_to_end.py, against the same checkout, one after the other because they share it. Output and exit codes go to runs/baseline_b_output.log and runs/baseline_c_output.log in the repository root. It passes only --piton-root, so both scripts use their default core and workload. (c) builds its default 1x1 mesh, and (b)’s default objective asks for one. Pass the checkout path as its argument, or set PITON_ROOT; with neither, it stops with a usage message. For 2x2 and 4x4 comparisons, run the two scripts directly with matching flags.

Seeded-failure runs

examples/recovery_seeded.py breaks the loop’s first plan, so the detect, diagnose, and replan cycle runs against Verilator builds and simulations:

export GOOGLE_CLOUD_PROJECT=<your-gcp-project> MAKEFLAGS=-j1
python examples/recovery_seeded.py --piton-root ~/openpiton --core ariane --mesh 2x2 \
    --workload barrier_atomic.c --fault build --runs 3

--fault build adds PITON_FPGA_SYNTH, a define for FPGA synthesis, to every config and workload task in the first plan. The Verilator build then fails with %Error-PINNOTFOUND.
--fault sim removes CONFIG_DISABLE_BIST_CLEAR from the first plan. It needs --core pico, whose simulation then fails.
--fault also takes any fault name from mace/eval/faults.py, such as crossbar or l1d_eight_way.

Later plans are left alone. The script passes each run a plan_hook that breaks the first plan, prints each iteration’s results and diagnoses, and records the runs in runs/recovery_seeded.db. --objective sets the objective. Pass the one from the run being compared: the PicoRV32 runs in Results named CONFIG_DISABLE_BIST_CLEAR in theirs.