Parallel Task Execution

mace.integrator.integrate_parallel() executes a task DAG across the OpenPiton checkouts listed in piton_roots.

Checkouts and Placement

OpenPiton’s pyHP preprocessor writes generated .tmp.v files into the source tree on every build, so two builds in one checkout corrupt each other. MACE therefore gives each concurrent task its own checkout.

open_nodes() constructs one OpenPitonWorkspaceNode per checkout concurrently, each waiting up to 120 s for its placement group. If any construction fails, it closes the nodes that succeeded and raises. Each node reserves a one-bundle Ray placement group of {"CPU": 1, "openpiton": 1}, and each of its member functions requests {"openpiton": 1}. All calls through one node therefore land on the same worker. A worker that advertises {"openpiton": 2} must host two separate checkouts. See Workspace Node.

Dispatch

Levels run in order (see Integration and Verification). A level with more tasks than checkouts runs in batches of len(piton_roots), one after another. The i-th task of a batch runs on the i-th checkout.

Within a batch, each config, workload, or rtl task runs on its own thread as up to three CHIA remote calls, each resolved with get() before the next:

  • llm.prompt.chia_remote() with the task’s instruction and the caller’s tools, only when LoopOptions.task_prompts is on;

  • node.build.chia_remote() with the task’s configuration;

  • node.run.chia_remote() for each gate workload in turn, if the build succeeded, stopping at the first that does not pass.

The orchestrator turns task_prompts off for every plan when LoopOptions.fast_path is on, which is the default, so a config or workload task normally makes no LLM call (see Fixed Checks). Each call carries a replay tag (see Budgets and Replay). The prompt’s reply changes nothing that is built or run, so when that call fails, for example with a reply cut off at the model’s output limit, the task logs a warning and still builds and runs. Tasks in a batch advance through these stages independently. The optional on_task_progress(task_ids, stage) callback reports prompting, building, and running.

A unit_test task skips this pipeline and runs locally through mace.loop.run_mace_step() on its slot’s checkout. A batch’s unit_test tasks run one after another before its remote tasks start. An rtl task replaces the first call with an edit session, described under RTL Tasks below.

Per-Task Configuration

mace.loop._config_for_task(spec, task) builds each task’s PitonConfig:

  • core, x_tiles, and y_tiles come from the spec;

  • extra_flags holds -vlt_build_args=--coverage-line when spec.coverage is set;

  • caches takes the task’s CACHES: override, and PitonConfig fills in the caches it leaves out from DEFAULT_CACHES;

  • config_rtl is the sorted union of the default ("MINIMAL_MONITORING",) and the task’s CONFIG_RTL: flags, which include those of the tasks it depends on (see Planner);

  • network_config takes the task’s network, which only co-design searches set;

  • source_edits holds a hash of the RTL edits the build carries, described under RTL Tasks below, and is empty for a build without edits.

A task’s configuration depends only on the spec, the task’s own overrides, and its dependencies’ RTL defines and edits. Two tasks in one level can therefore build different cache geometries. On Ariane, build() refuses a task whose L1D or L1I has more ways than its L1.5, so the task fails with the failure reason way_rule, and triage and the re-plan see which cache broke the rule.

RTL Tasks

An rtl task changes the design’s source. Its agent call runs locally through usage.prompt_retrying(), which waits and retries a rate-limited call, with the prompt from rtl_prompt(): the run’s inputs, the task’s instruction, and the editing rules. The call runs whether or not LoopOptions.task_prompts is on, because it makes the edit. When Ray is initialized, the call gets a mace.tools.RtlEditTool, placed with the node’s task_options and stopped after the call. Its functions {name}_list, {name}_read, {name}_search, and {name}_replace take paths relative to the checkout and refuse any path outside piton/design/, so the testbench and the monitors under piton/verif/ cannot change. A replacement must match exactly one place in the file. The tool never writes the checkout: it keeps each changed file’s full new content in a staging file under build/.mace_edit_staging/.

Before the build, node.apply_edits() writes the edits into the task’s checkout: first the edits of the tasks it depends on, then the task’s own from the staging file. After the task’s runs, node.revert_edits() restores every file, however the task ended. A task with no edits calls node.revert_edits() before its build, so edits a killed task left are never built. The build’s source_edits gives an edited design its own model directory. A task of any kind builds with the edits of the tasks it depends on, directly or through other tasks: integrate_parallel() keeps each passed task’s edits for the levels after it. An rtl task whose agent recorded no edit of its own still builds and runs the design as it stands, so a failure carries the build error with its file and line. The task’s result has edit_recorded false, and the failure evidence says so and quotes the agent’s closing words. The task’s result carries the edits and their unified diff, which triage, the re-plan’s feedback, and the tasks.rtl_edits column quote.

Both the planner’s offer of the rtl kind and the edit tool need LoopOptions(rtl_edits=True), which is off by default; see Planner. With it off, an rtl task runs as a config task. The serial integrate() and run_mace_step() give no task an RtlEditTool.

Unit-Test Tasks

_run_unit_test_step() reads task.spec as an RTL path. It first checks the path: it must be relative, stay inside the checkout, end in .v, .sv, or .pyv, and name an existing file. A path that fails the check fails the task at once, with failure reason unit_test_bad_module_path and an error line that names the problem, so triage sees it; nothing is scaffolded, prompted, or built. It scaffolds an environment named by unit_test_env_name() with OpenPiton’s create_env.py, reads the module’s ports, and prompts the agent to fix the scaffolded <env>_top.v. It then builds PitonConfig(sys=env_name).

When Ray is initialized, the step creates a mace.tools.TestbenchEditTool for the LLM call and stops it afterward. The tool exposes {name}_read_testbench, {name}_write_testbench, and {name}_read_dut_source, and none of them takes a path. The agent can rewrite that testbench and read that module’s source, and no other file.

_unit_test_tool_name(task.id) gives it a fixed-length name:

f"ut_edit_{hashlib.sha1(task_id.encode()).hexdigest()[:8]}"

CHIA’s API backends truncate each function name, {tool}__{tool}_{method}, to 64 characters. The fixed 16-character name keeps the three function names distinct for any task id.