Results and Metrics

mace.metrics records every loop run in a SQLite database. examples/mace_end_to_end.py writes to runs/mace_end_to_end.db, mace shell to runs/mace_cli.db, and examples/generic_integrate.py to runs/generic_integrate.db; all accept --db-path. The one-shot baseline prints its result and records nothing. Git ignores runs/ and *.db.

Tables

Every insert except into llm_calls uses INSERT OR REPLACE on the table’s primary key, so recording the same row again overwrites it. llm_calls rows are appended with the next seq.

Table	Primary key	Other columns
runs	run_id	objective, core, x_tiles, y_tiles, started_at, finished_at, status, method, task, repeat, seed, meta
iterations	run_id, iteration	num_tasks, num_passed, wall_s, usd
tasks	run_id, iteration, task_id	kind, spec, passed, build_success, run_verdict, wall_s, caches, module, build_s, run_s, programs, evidence, config_rtl, rtl_edits
failures	run_id, iteration, task_id	diagnosis, fix, recovered
post_mortems	run_id	assessment, explanation, next_steps
llm_calls	run_id, seq	iteration, phase, input_tokens, output_tokens, thinking_tokens, usd, wall_s, ok, reply
evaluations	run_id, idx	round, design, passed, feasible, sim_time, area_um2, read_energy_nj, area_source, wall_s, simulated, rejected, hart_cycles, note, storage_bits
resimulations	run_id, task_id, program	verdict, passed, wall_s
plan_checks	run_id, iteration, attempt	action, problems, stripped
probes	run_id, iteration, probe	failed_task, task_id, undone, verdict, ran, passed, build_success, run_verdict, build_s, run_s
fast_paths	run_id, iteration	kept, dropped
directed_tests	run_id, iteration, task_id, attempt, name	target, form, source, verdict, wall_s, evidence
test_checks	run_id, iteration, task_id, attempt, name	fault_verdict, clean_verdict, outcome, wall_s

• run_id is 12 hex digits.

• runs.method names what ran: mace for a plain loop run, or the baseline or ablation an evaluation run was. runs.task, runs.repeat, and runs.seed identify the evaluation task and repeat, and runs.meta holds the run’s environment as JSON. A database written before these columns existed gets them empty.

• iterations.wall_s runs from the start of the planner call to the end of the iteration’s last task, and excludes triage. iterations.usd is the cost of the iteration’s planner, task, and triage calls; see LLM Backends.

• tasks.build_s and tasks.run_s are the build time and the summed simulation time, and tasks.wall_s is their sum; a reused build counts as 0. tasks.programs lists each gate workload run with its verdict, as JSON. tasks.caches holds the build configuration’s cache map as JSON, and tasks.config_rtl its sorted RTL defines as a JSON list. tasks.rtl_edits holds the unified diff of the RTL edits the build carried, or nothing. tasks.module names the module of a unit_test task. tasks.evidence holds, for a task that did not pass, the text triage reads: the build flags and their changes from the defaults, then the build’s errors and output tail or the failing simulation’s transcript tail and status log, then the diff of any RTL edits the build carried. It stays after an evaluation job’s cache clear removes the logs.

• plan_checks holds one row for each look the plan check took at an iteration’s plan that found a problem or removed a define, and for each retry’s look. attempt counts the plans the iteration asked for, action is rejected, accepted, kept, or unusable (a planner reply with no plan), and problems and stripped are JSON lists. probes holds one row per narrowing probe, tied to the task whose failure it examined by failed_task; undone lists the changes the probe put back as JSON, and verdict repeats the narrowing’s verdict on every row. Probes stay out of tasks. fast_paths has a row for each iteration whose plan ran as one task: kept is that task and dropped the ids of the tasks it replaced, as JSON.

• directed_tests holds each test a test task wrote, with its verdict on the design the task built, and test_checks the evaluation’s mutation check of each test (see Directed Tests and Directed-Test Evaluation). In evaluations, hart_cycles holds an RTL-parameter design’s per-hart cycle counts as JSON, storage_bits the flip-flop bits it adds per tile, and note the bottleneck line of the MACE proposer that chose it (see RTL-Parameter Searches).

• llm_calls holds one row per LLM call, numbered in the order the calls finished. phase is plan, plan_retry (a plan asked for again after the plan check rejected one), task, triage, or post_mortem for the loop, agent for the retry-agent baseline, and propose for the co-design LLM proposer; a post-mortem call has no iteration. ok is 0 for a call that failed; such a call records no tokens. reply holds the reply’s text, such as a plan’s TASK: lines or a triage diagnosis.

• failures.diagnosis is free text. The triage prompt suggests test_bug, config_error, timeout, maxcycles, rtl_suspect, and testbench_mismatch. The loop records unknown when the reply has no DIAGNOSIS: line, and config_change when narrowing decided the diagnosis without a triage call. When a run passes after earlier failures, all of its failures are marked recovered.

• The post-mortem prompt asks for the assessment fixable_config, likely_hardware_limitation, or inconclusive.

Run statuses

Status	Meaning
running	The run started and has not finished. A killed process leaves its run in this state.
passed	Every task of one iteration passed, or one task’s design passed every gate workload.
budget_exceeded	max_iterations, max_wall_s, or max_usd ran out before any iteration passed.
planning_failed	The planner’s reply held no valid task DAG, and the two retries the plan check allows gave none either.
failed	An iteration produced no task results.
checksum_mismatch	A C program in mace/workloads/ failed its checksum; nothing ran.
unsupported_core	The generic tile cannot run the core descriptor’s core, or the checkout lacks its files; nothing ran. LoopResult.unsupported holds the reasons.
error	An exception stopped the run.

A run that ends failed or budget_exceeded after at least one iteration gets a post-mortem, unless the LLM’s reply cannot be parsed.

The five metrics

mace.metrics.summary(db, run_id) returns:

Key	Computed as
successful_tasks	Count of tasks rows with passed = 1.
iterations	Count of iterations rows.
failures_recovered	Count of failures rows with recovered = 1.
execution_time_s	Sum of iterations.wall_s.
compute_usd	Sum of llm_calls.usd, which includes the post-mortem; for a run recorded before llm_calls existed, the sum of iterations.usd.

from mace.metrics import DBReader, all_runs

db = DBReader("runs/mace_end_to_end.db")
for run in all_runs(db):
    print(run["run_id"], run["status"], run["successful_tasks"], run["execution_time_s"])

DBReader reads the database with Python’s sqlite3 and starts no Ray. It adds the columns that an older database lacks, such as caches, config_rtl, and reply, then refuses writes. The loop writes through open_db(path), a CHIA SQLiteNode; the first call on one starts Ray, because CHIA’s profiler looks up its collector actor. all_runs(db) returns every run, newest first, with its summary merged in. trace_run(db, run_id) gives one run’s iterations with their tasks and failures. failure_taxonomy(db, run_id) counts failures by diagnosis. module_status(db, run_id) gives the latest status of each unit_test module. get_post_mortem(db, run_id) returns the post-mortem.

mace results

mace results reads the database through DBReader, so it needs no shell session and starts no Ray. It exits with an error when the file does not exist. Its --db-path default is runs/mace_cli.db, so name the loop script’s database explicitly:

mace results --db-path runs/mace_end_to_end.db
mace results --db-path runs/mace_end_to_end.db --run-id <run_id>
mace results --db-path runs/mace_end_to_end.db --run-id <run_id> --trace

With no other option, it prints one row per run (run_id, core, mesh, status, tasks, iterations, wall_s, usd) and a line with the pass count, total execution time, and total cost. The tasks column counts passed tasks. --run-id prints that run’s failure taxonomy. --trace with --run-id prints the run as a tree: one branch per iteration, the tasks its plan dispatched with their build result and verdict, and a TRIAGE branch when a failure was diagnosed. --trace without --run-id is an error.

SQL

You can also query the tables directly:

SELECT iteration, task_id, kind, build_success, run_verdict, wall_s
FROM tasks
WHERE run_id = '<run_id>'
ORDER BY iteration, rowid;