% Copyright (c) 2026 Rana Umar Nadeem, Samrah Mumtaz, Muhammad Imran

# Failure Analysis

When an iteration holds a failed task, the orchestrator diagnoses the first failed result with `mace.triage.triage()` and feeds the diagnosis into the next plan.
Other failed tasks in that level get a row in the `tasks` table and no triage.

## Triage

`triage(result, llm, tools=(), objective="", rtl_edits=False, narrowing="")` makes one LLM call.
The orchestrator passes the run's objective, which leads the prompt: it can name what a build needs, such as an RTL define the failed plan left out.
The prompt gives the task's id, kind, and instruction, the build outcome, the run verdict, and the build's `sims` flags, which name its RTL defines and cache sizes.
A line of changes from the defaults picks out the plan's own choices among them: the RTL defines it adds, each cache whose geometry differs, and a crossbar interconnect (`changes_from_defaults()`).
A failed build adds the failure reason, its error lines (see [PitonBuildArtifact](../04_chia_openpiton/workspace_node.md)), and the last 1500 characters of its stderr, or of its stdout when stderr is empty, as `sims` leaves it.
A failed run adds the last 1500 characters of the simulation log and up to 8000 characters of the status log.
When the build carried RTL edits, the prompt then adds their diff, up to 3000 characters, and the feedback line below adds it too.
For an `rtl` task whose agent recorded no edit, the prompt says so and quotes up to 600 characters of the agent's reply.
The prompt then says what the next plan can change: the build configuration only (RTL defines, cache sizes and associativity), or, under `LoopOptions(rtl_edits=True)`, also files under `piton/design/` (`ACTIONS_NO_RTL`, `ACTIONS_RTL`).
Without RTL edits, a failure no configuration can fix gets the label `needs_rtl_change`.
It adds that each change from the defaults is a candidate cause, to rule out before blaming the design.
The prompt asks for two directive lines:

```text
DIAGNOSIS: <a short label, e.g. test_bug, config_error, timeout, maxcycles, rtl_suspect, testbench_mismatch>
FIX: <a short, concrete instruction for what to try next>
```

`parse_diagnosis()` returns the last `DIAGNOSIS:` value, lowercased, and `parse_fix()` the last `FIX:` value; a missing fix is recorded as empty.
`KNOWN_DIAGNOSES` lists the six labels, but any label is recorded.
Without a `DIAGNOSIS:` line, triage raises `TriageError`, and the orchestrator records `unknown` with the fix `retry with more context`.

A failed `unit_test` build whose output contains `%Error-PINNOTFOUND` skips the LLM call and gets `testbench_mismatch`, with a fix aimed at the testbench's port connections rather than the module.

The orchestrator narrows a failed `config` or `workload` task before it triages it (see [Fixed Checks](fixed_checks.md)).
When the probes name the change to undo, triage does not run, and the failure is recorded as `config_change` with that fix.
When they settle nothing, `narrowing` adds the probes' note to the end of the context above, so triage knows which causes the rebuilds ruled out.

## Diagnostic Tools

When Ray is initialized, the orchestrator starts a `PitonToolServer` named `triage-<run ID>` on the first checkout for each triage.
It stops the previous server first and hands the new one the failed task's build and run.
It exposes four read-only tools:

| Tool | Returns |
|---|---|
| `grep` | Regex matches from the run's `sim.log`, `status.log`, or `fake_uart.log` |
| `collect` | Small text files from the run directory, by glob |
| `compare_to_fixture` | The first divergence between `sim.log` and a reference transcript in `chia_openpiton/test/fixtures/` |
| `symbol_check` | `objdump -f` and `-t` output for the run's `diag.exe`, beside its `symbol.tbl` |

A task that failed at build has no run, and the tools report an error for it.
Triage gets the caller's tools plus this server.
If the server fails to start, the run goes on without it.
The last server also serves the post-mortem and stops when the run ends.
See [Tool Server](../04_chia_openpiton/tool_server.md).

## Replanning

`record_failure()` writes the diagnosis to the `failures` table, and `replan_feedback()` writes a note for the run's feedback history:

```text
Task <id> (<instruction>) failed (build succeeded: <b>, run verdict: <v>).
Changes from the defaults: <changes>
What the build and simulation reported: <digest>
<the probes' note, when probes ran>
Triage, an LLM's reading of this evidence: diagnosis=<label>, suggested fix=<fix>
```

The digest is the build's error lines, or the failing program's verdict and the end of its simulation log, in about 800 characters.
It keeps a wrong label from hiding the line that names the cause.
When the probes decided the diagnosis, the last line reads `Diagnosis from the probes above` instead.
With `LoopOptions(raw_evidence=False)`, the note is one line: the task, the changes from the defaults, the label, and the fix.

With `LoopOptions(triage="raw")`, the line instead holds whether the build succeeded, the run verdict, and the failure's raw evidence, and with `triage="off"` the orchestrator adds no line (see [Architecture](architecture.md)).
Every later `plan()` call receives the whole history.
If the run then passes, `mark_all_recovered()` marks all of its failures recovered.
A replanned DAG can use new task ids, so recovery is tracked per run.

A tier-1 test, `mace/test/cluster/orchestrator_e2e_test.py`, exercises the detect, diagnose, and replan cycle with a stub `sims` that fails its first run and passes after that.
[Results](../06_mace_evaluation/results.md) reports which runs needed the cycle, including the seeded-failure runs.

## Post-Mortem

A run that ends with `failed` or `budget_exceeded` after at least one iteration gets one more LLM call, `mace.report.generate_post_mortem()`.
Its prompt holds the objective, core, mesh, stop reason, and one line per task tried, with the diagnosis on the triaged task.
The prompt asks for three directive lines:

```text
ASSESSMENT: <fixable_config|likely_hardware_limitation|inconclusive>
EXPLANATION: <the specific evidence that led to this assessment>
NEXT_STEPS: <what to try next, or why nothing more is worth trying>
```

`record_post_mortem()` stores the result in the `post_mortems` table.
A response without `ASSESSMENT:` is dropped, and the run status stands.