Failure Analysis
When an iteration holds a failed task, the orchestrator diagnoses the first failed result with mace.triage.triage() and feeds the diagnosis into the next plan.
Other failed tasks in that level get a row in the tasks table and no triage.
Triage
triage(result, llm, tools=(), objective="", rtl_edits=False, narrowing="") makes one LLM call.
The orchestrator passes the run’s objective, which leads the prompt: it can name what a build needs, such as an RTL define the failed plan left out.
The prompt gives the task’s id, kind, and instruction, the build outcome, the run verdict, and the build’s sims flags, which name its RTL defines and cache sizes.
A line of changes from the defaults picks out the plan’s own choices among them: the RTL defines it adds, each cache whose geometry differs, and a crossbar interconnect (changes_from_defaults()).
A failed build adds the failure reason, its error lines (see PitonBuildArtifact), and the last 1500 characters of its stderr, or of its stdout when stderr is empty, as sims leaves it.
A failed run adds the last 1500 characters of the simulation log and up to 8000 characters of the status log.
When the build carried RTL edits, the prompt then adds their diff, up to 3000 characters, and the feedback line below adds it too.
For an rtl task whose agent recorded no edit, the prompt says so and quotes up to 600 characters of the agent’s reply.
The prompt then says what the next plan can change: the build configuration only (RTL defines, cache sizes and associativity), or, under LoopOptions(rtl_edits=True), also files under piton/design/ (ACTIONS_NO_RTL, ACTIONS_RTL).
Without RTL edits, a failure no configuration can fix gets the label needs_rtl_change.
It adds that each change from the defaults is a candidate cause, to rule out before blaming the design.
The prompt asks for two directive lines:
DIAGNOSIS: <a short label, e.g. test_bug, config_error, timeout, maxcycles, rtl_suspect, testbench_mismatch>
FIX: <a short, concrete instruction for what to try next>
parse_diagnosis() returns the last DIAGNOSIS: value, lowercased, and parse_fix() the last FIX: value; a missing fix is recorded as empty.
KNOWN_DIAGNOSES lists the six labels, but any label is recorded.
Without a DIAGNOSIS: line, triage raises TriageError, and the orchestrator records unknown with the fix retry with more context.
A failed unit_test build whose output contains %Error-PINNOTFOUND skips the LLM call and gets testbench_mismatch, with a fix aimed at the testbench’s port connections rather than the module.
The orchestrator narrows a failed config or workload task before it triages it (see Fixed Checks).
When the probes name the change to undo, triage does not run, and the failure is recorded as config_change with that fix.
When they settle nothing, narrowing adds the probes’ note to the end of the context above, so triage knows which causes the rebuilds ruled out.
Diagnostic Tools
When Ray is initialized, the orchestrator starts a PitonToolServer named triage-<run ID> on the first checkout for each triage.
It stops the previous server first and hands the new one the failed task’s build and run.
It exposes four read-only tools:
Tool |
Returns |
|---|---|
|
Regex matches from the run’s |
|
Small text files from the run directory, by glob |
|
The first divergence between |
|
|
A task that failed at build has no run, and the tools report an error for it. Triage gets the caller’s tools plus this server. If the server fails to start, the run goes on without it. The last server also serves the post-mortem and stops when the run ends. See Tool Server.
Replanning
record_failure() writes the diagnosis to the failures table, and replan_feedback() writes a note for the run’s feedback history:
Task <id> (<instruction>) failed (build succeeded: <b>, run verdict: <v>).
Changes from the defaults: <changes>
What the build and simulation reported: <digest>
<the probes' note, when probes ran>
Triage, an LLM's reading of this evidence: diagnosis=<label>, suggested fix=<fix>
The digest is the build’s error lines, or the failing program’s verdict and the end of its simulation log, in about 800 characters.
It keeps a wrong label from hiding the line that names the cause.
When the probes decided the diagnosis, the last line reads Diagnosis from the probes above instead.
With LoopOptions(raw_evidence=False), the note is one line: the task, the changes from the defaults, the label, and the fix.
With LoopOptions(triage="raw"), the line instead holds whether the build succeeded, the run verdict, and the failure’s raw evidence, and with triage="off" the orchestrator adds no line (see Architecture).
Every later plan() call receives the whole history.
If the run then passes, mark_all_recovered() marks all of its failures recovered.
A replanned DAG can use new task ids, so recovery is tracked per run.
A tier-1 test, mace/test/cluster/orchestrator_e2e_test.py, exercises the detect, diagnose, and replan cycle with a stub sims that fails its first run and passes after that.
Results reports which runs needed the cycle, including the seeded-failure runs.
Post-Mortem
A run that ends with failed or budget_exceeded after at least one iteration gets one more LLM call, mace.report.generate_post_mortem().
Its prompt holds the objective, core, mesh, stop reason, and one line per task tried, with the diagnosis on the triaged task.
The prompt asks for three directive lines:
ASSESSMENT: <fixable_config|likely_hardware_limitation|inconclusive>
EXPLANATION: <the specific evidence that led to this assessment>
NEXT_STEPS: <what to try next, or why nothing more is worth trying>
record_post_mortem() stores the result in the post_mortems table.
A response without ASSESSMENT: is dropped, and the run status stands.