Troubleshooting

Rebuilds and stale models

Each model lives in $PITON_ROOT/build/<sys>/<build_id>, where <sys> is manycore for every task except a unit test. A build ID is mace_ plus 12 hex digits. The loop builds each PitonConfig directly, without configure(), so its build IDs cover only the configuration: core, mesh, network, RTL defines, cache geometry, extra flags, for a unit test the environment name, and for a build with RTL edits a hash of those edits.

A successful build leaves a .mace_build_ok marker that holds the configuration key and a fingerprint of the checkout. A later build with the same ID reuses the model only while both match. After an RTL or monitor edit, a new fix from scripts/patch_openpiton.sh, a new commit, or a change of Verilator, the next build of each configuration rebuilds its model and logs the checkout or Verilator changed since it was built. A model built before the fingerprint existed rebuilds once. OpenPitonWorkspaceNode lists what the fingerprint covers.

The fingerprint misses edits inside the Ariane submodule’s own submodules unless they move a submodule commit. If the same failure repeats in every iteration after such an edit, rebuild with clean=True, which first deletes the model’s obj_dir and marker:

art = get(node.build.chia_remote(cfg, clean=True))

The shell prints each config task’s model_dir, and PitonConfig gives the ID the loop uses for a configuration:

from chia_openpiton.state_def import PitonConfig

cfg = PitonConfig(core="pico", x_tiles=4, y_tiles=4,
                  config_rtl=("CONFIG_DISABLE_BIST_CLEAR", "MINIMAL_MONITORING"))
print(cfg.build_id)

Ray times out during startup

On a loaded machine, ray.init() can fail when the raylet stops waiting for the dashboard agent, with Timed out waiting for file .../dashboard_agent_listen_port_.... Run the command again. The drivers start a fresh local instance with ray.init(address="local", ...), so a stale cluster marker is not the cause, and the include_dashboard=False in examples/mace_end_to_end.py does not prevent the race.

Memory and MAKEFLAGS for 4x4 builds

sims compiles Verilator’s generated C++ with make. Its parallel cc1plus processes take 100 MB to 850 MB each, and a 16-tile Ariane build can run out of memory. Export MAKEFLAGS=-j1 before a 4x4 run:

export MAKEFLAGS=-j1

Fix 9 of scripts/patch_openpiton.sh removes the bare make -j in sims that would override MAKEFLAGS. Do not raise MAKEFLAGS to -j2: the boot ROM Makefile then races on rv64_platform.dtb. On a chia up cluster, set MAKEFLAGS in the worker’s worker_env_commands, as the openpiton_gcp node type in cluster/local.yaml does. A driver’s environment does not reach remote workers. A fresh 4x4 build at -j1 can take over an hour, and the adapter’s build timeout defaults to 7200 s.

Verilator versions

Verilator 5 refuses OpenPiton’s bare #1 delays unless it gets --timing or --no-timing. The adapter reads verilator --version inside OpenPiton’s environment and adds --no-timing for version 5 and later; Verilator 4 has no such flag. Do not pass --timing: my_top.cpp advances simulation time itself, and --timing aborts at startup with Missed a time slot?. A build that fails with %Error-NEEDTIMINGOPT (tag verilator_needs_timing_flag) did not get --no-timing. Check which verilator OpenPiton’s environment finds through PATH and VERILATOR_ROOT.

Version	Result with this RTL
5.020	Builds and runs Ariane and PicoRV32 with `--no-timing`; used for the evaluation. OpenSPARC T1 stalls. Ubuntu 24.04’s `5.020-1` package has the same missing `-c` as 5.040 and 5.048, so the evaluation machines add `-c` before `$(CFG_CXXFLAGS_PCH)` in the two precompiled-header rules of `/usr/share/verilator/include/verilated.mk`.
5.028	Internal crash `Wide Op w/ no temp` in `V3EmitCFunc.cpp` (verilator#5820), fixed in 5.036.
5.040, 5.048	Their `verilated.mk` sets `CFG_CXXFLAGS_PCH` without `-c`, so every build fails to link the precompiled header with undefined reference to `main' (tag `pch_link_failure`).
5.049 devel	Builds, but its `verilator_coverage` faults on `--version`.
5.052	Pinned by `flake.nix`; has none of the 5.028, 5.040, 5.048, and 5.049 problems, and runs OpenSPARC T1.

Tool name length

CHIA’s API backends name each tool function <tool>__<function>, cut to 64 characters. When two names collapse to the same prefix, Vertex rejects the request with Duplicate function declaration found. MACE therefore names its unit-test edit tool ut_edit_ plus 8 hex digits of a hash of the task ID. Keep <tool>__<function> within 64 characters for any tool you add.

Other build errors

Symptom	Cause	Fix
Diags fail with `string.h: No such file`	A distribution `riscv64-unknown-elf-gcc` without newlib comes first on `PATH`	Set `RISCV` to your toolchain’s install directory.
Boot ROM build error at the `init_uart` call in `src/main.c`	GCC 15 and later compile C as C23, where `void init_uart();` takes no arguments, and the checkout lacks fix 2	Run `scripts/patch_openpiton.sh` on every checkout.
`/usr/bin/env: 'python3\r'`, or `dtc` fails on a path string	Checkout on a Windows-mounted drive	Clone on native Linux storage; fixes 3 and 4 repair an existing checkout.