diff --git a/agent-ops/.version b/agent-ops/.version index fef593f..0f0b255 100644 --- a/agent-ops/.version +++ b/agent-ops/.version @@ -1 +1 @@ -1.1.178 +1.1.179 diff --git a/agent-ops/skills/common/code-review/SKILL.md b/agent-ops/skills/common/code-review/SKILL.md index 7918f9f..bd0d50e 100644 --- a/agent-ops/skills/common/code-review/SKILL.md +++ b/agent-ops/skills/common/code-review/SKILL.md @@ -233,7 +233,7 @@ The follow-up handoff contains the selected `{task_name}`, the current plan's re - `prepare-follow-up` must return `status: routed`, the exact routed basenames, `prepared_plan`, `prepared_review`, `plan_number`, `current_plan_archive_name`, `current_plan_archive_number`, `current_review_archive_name`, `current_review_archive_number`, `plan_log_number`, `review_log_number`, and `gitignore_repair_needed`. It must have executed `finalize-task-routing` in `isolated-reassessment` mode. - Verify that the returned current archive names/numbers equal the values derived before preparation, and that `plan_log_number` / `review_log_number` are the post-archive counts embedded in the new review stub for its future archive. -- Materialize `prepared_plan` only as a temporary candidate outside the repository and run `python3 agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan `. Require exit code `0` before archiving either active file. The candidate must contain exactly one non-empty `Modified Files Summary` with only exact workspace files. Globs, directories, workspace root, URLs, outside-workspace paths, malformed paths, and placeholders are invalid. Remove the temporary candidate after validation. +- Materialize `prepared_plan` only as a temporary candidate outside the repository and run `python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan `. Require exit code `0` before archiving either active file. The candidate must contain exactly one non-empty `Modified Files Summary` with only exact workspace files. Globs, directories, workspace root, URLs, outside-workspace paths, malformed paths, and placeholders are invalid. Remove the temporary candidate after validation. - If preparation returns `needs_evidence`, collect all named new evidence and rerun after the input changes; never rerun with unchanged evidence. If the evidence cannot be obtained in the current scope, leave the verdict-appended pair in place and report the exact finalization blocker. - If preparation returns `blocked` or prepared PLAN validation fails, leave the verdict-appended active PLAN/CODE_REVIEW pair in place, do not check archive/next-state items, and report a resumable finalization blocker. A later code-review invocation resumes this step without appending another verdict. @@ -267,7 +267,7 @@ For `WARN` or `FAIL`, materialize the next state prepared in Step 5 immediately - If the user-review gate triggered, write the prepared body to `agent-task/{task_name}/USER_REVIEW.md`. It must use exactly one supported type, `milestone-lock` or `external-execution`, contain every archived loop entry plus the exact required user action or decision, and contain no placeholder. Do not write active PLAN/CODE_REVIEW files or `complete.log`. - Otherwise write `prepared_plan` and `prepared_review` byte-for-byte to their routed basenames. Do not rerun, adjust, compare, or upgrade their lane/G after archive. - Verify the written follow-up pair contains the predicted archived plan/review paths in identical `Archive Evidence Snapshot` sections and contains no unresolved token from the review-stub template inventory. Unrelated braces in commands or code are allowed. -- Re-run `python3 agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan ` and require exit code `0` to confirm that the byte-for-byte materialized PLAN retained the validated write claim. +- Re-run `python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan ` and require exit code `0` to confirm that the byte-for-byte materialized PLAN retained the validated write claim. - Do not adjust the prepared route after finalization. For a `local-fit` base, `review_rework_count >= 2` or `evidence_integrity_failure=true` must produce `recovery-boundary`; `capability-gap` and `grade-boundary` keep their own basis. If the task group is `m-` and the user-review gate triggered, report that the milestone task is blocked on user review; do not emit PASS completion metadata and do not call `update-roadmap`. @@ -327,7 +327,7 @@ Report Required/Suggested counts, archive names, the final task archive path for - PASS with `Roadmap Targets`: `complete.log` contains `Roadmap Completion` with Milestone path, Task ids, archived plan/review evidence, and verification evidence. - PASS without `Roadmap Targets`: `complete.log` omits `Roadmap Completion` and reported metadata says `roadmap-completion=none`. - WARN/FAIL without user-review gate: the plan skill was invoked for the exact task path with verified `review_rework_count` and `evidence_integrity_failure`, completed `finalize-task-routing`, and created new active `PLAN-{build_lane}-GNN.md` and `CODE_REVIEW-{review_lane}-GNN.md` files matching the fresh routed output; no `complete.log`. -- WARN/FAIL prepared PLAN passed `python3 agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan ` before active-pair archive and again after byte-for-byte materialization. Invalid write claims leave the verdict-appended prior pair active. +- WARN/FAIL prepared PLAN passed `python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan ` before active-pair archive and again after byte-for-byte materialization. Invalid write claims leave the verdict-appended prior pair active. - WARN/FAIL follow-up: the plan input omitted prior route fields, revalidated outcome/acceptance/exclusions from current evidence, used the completed in-memory PLAN as the packet, and copied identical `Archive Evidence Snapshot` sections into the new plan/review pair. - Follow-up plans and review stubs keep implementation agents limited to implementation/test/evidence and contain no implementation-owned user-review request section. - USER_REVIEW: `USER_REVIEW.md` exists from template, no active `PLAN-*.md` or `CODE_REVIEW-*.md` remains, and no `complete.log` was written. diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/SKILL.md b/agent-ops/skills/common/orchestrate-agent-task-loop/SKILL.md new file mode 100644 index 0000000..dbc519b --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/SKILL.md @@ -0,0 +1,295 @@ +--- +name: orchestrate-agent-task-loop +description: Run agent-task work and autonomously execute active PLAN/CODE_REVIEW loops on request. Use when dispatching dependency-ready work in parallel by predecessor completion and workspace write claims, running lane/G-specific Codex, Claude, agy, and Pi workers, adding Pi self-checks, converging official Codex reviews, and escalating cloud context until the task loop finishes. +--- + +# Orchestrate Agent Task Loop + +## ๐Ÿšจ ABSOLUTE PRIORITY โ€” NEVER SEND `final` EXCEPT IN THE TWO CASES BELOW + +> [!CAUTION] +> **This section overrides every success, blocker, exit-code, error-handling, and termination rule below.** +> +> **Never send on the `final` channel or end the caller turn unless at least one of the two titled permissions below applies. Never infer another exception from a lower section or runtime condition.** + +### `final` Permission 1 โ€” Verified Successful Completion + +Allow `final` only after every condition below is true: + +- Every user-defined completion condition is satisfied. +- Every observed task in every in-scope task group has a verified archived `complete.log`. +- Every generated `WORK_LOG.md` is archived as `work_log_N.log`. +- No active pair or running, pending, or blocked task remains. +- The final dispatcher exit code is `0`. + +### `final` Permission 2 โ€” Explicit User Instruction to Stop This Run + +Allow `final` when the user explicitly instructs the caller to stop the current run and return through `final`. + +### Persistent-Run Instructions Revoke Successful-Completion Permission + +If the user says โ€œdo not stop,โ€ โ€œnever send final,โ€ โ€œkeep going,โ€ or gives an equivalent persistent-run instruction, verified success alone does not permit `final`. Only an explicit user instruction to stop the current run or return through `final` releases this restriction. + +### Every Other User-Visible Message Must Use `commentary` + +Use only the `commentary` channel for every user-visible message before `final` is permitted. This includes status, partial success, completion candidates, blockers, failures, questions, apologies, waits, retries, and recovery guidance. + +Partial success, FAIL/WARN, USER_REVIEW, a blocker, retry exhaustion, timeout, a tool error, plan-generation failure, dispatcher exit code `2` or `3`, child exit, loss of a session/cell, and context compaction never permit `final`. + +Dispatcher stdout streamed directly by the execution layer is tool output, not a caller-authored message. Never spend an LLM turn restating, summarizing, or relaying a routine dispatcher event. + +### Child Prompt Text Never Grants Caller `final` Permission + +The prompt-contract phrase `Final in Korean.` controls only the child model response language. It never authorizes the caller to use the `final` channel. + +## Purpose + +Monitor the file-based state contract under `agent-task/` and converge the workflow from ready PLAN implementation through official code review and follow-up PLANs. Let the script determine filenames, dependencies, slots, and session locators; let each CLI agent make semantic implementation and review decisions. + +Treat Korean text inside code spans or fenced examples as exact runtime or file-contract literals. Keep all surrounding instructions in English, and never translate those literals unless the runtime contract changes. + +## Inputs + +- `workspace`: Trusted repository root containing `agent-task/` (optional; defaults to the current directory). +- `task_group`: Name of a specific `agent-task/` to run (optional). +- `dry_run`: Inspect state, routes, and dependencies without starting a CLI (optional). +- `max_parallel`: Non-negative integer cap on unique active task-stage attempts across the physical workspace. Omission defaults to `3`; explicit `0` is unlimited. `--task-group` does not narrow occupancy, adopted external attempts count, internal helper coroutines do not count separately, and an override must be supplied again after restart. +- `retry_blocked`: Explicitly retry the same PLAN blocked by a previous dispatcher run in non-dry-run mode (optional). With `task_group`, reset only that group's blockers and 10-attempt counters while preserving other group state. + +## Preconditions + +- [ ] Read the current state contracts in `agent-ops/skills/common/plan/SKILL.md` and `agent-ops/skills/common/code-review/SKILL.md`. +- [ ] Verify that `codex`, `claude`, `agy`, and `pi` are on PATH and their login/provider configuration is valid. +- [ ] Limit automatic approval to PLAN execution inside the current workspace; do not expand scope to external-system changes or destructive work. +- [ ] Verify that no other dispatcher is running in the same workspace. Never bypass a workspace-lock failure. +- [ ] Run `--dry-run` before the first live run to inspect active-task classification and dependency state. + +## Routing Contract + +| PLAN route | Worker | +|---|---| +| `local-G01`โ€“`local-G06` | Pi `iop/ornith:35b`, thinking high | +| `local-G07`โ€“`local-G08` | KST `[07:00,23:00)` agy `Gemini 3.6 Flash (Medium)`; `[23:00,07:00)` Pi `iop/laguna-s:2.1` | +| `local-G09`โ€“`local-G10` | Claude `claude-opus-4-8`, effort xhigh | +| `cloud-G01`โ€“`cloud-G02` | agy `Gemini 3.6 Flash (Low)` | +| `cloud-G03`โ€“`cloud-G04` | agy `Gemini 3.6 Flash (Medium)` | +| `cloud-G05`โ€“`cloud-G06` | agy `Gemini 3.6 Flash (High)` | +| `cloud-G07`โ€“`cloud-G08` | Claude `claude-opus-4-8`, effort xhigh | +| `cloud-G09`โ€“`cloud-G10` | Codex `gpt-5.6-sol`, reasoning xhigh | +| Every `CODE_REVIEW-*` | Codex `gpt-5.6-sol`, reasoning xhigh | + +Concurrency limits: + +- Global physical-workspace limit: omitting `max_parallel` caps execution at `3`; explicit `max_parallel=0` is unlimited. A positive value caps unique active task-stage attempts and is not narrowed by `task_group`. The cap applies across worker, self-check, review, and verified external-active attempts in the same physical workspace. +- Pi `ornith:35b`: 3. +- agy: 1. +- Official Codex review: no separate review-only limit; subject to the global + cap. +- Run worker/self-check and official review in parallel only when they belong to different dependency-ready tasks and their canonical PLAN write sets do not collide in the current physical workspace. Prevent duplicate execution of the same task. +- Even with `complete.log`, treat an explicit predecessor as unfinished while live model/review execution evidence for that task remains. Delay only its consumers; do not propagate the delay to dependency-free siblings or other task groups. +- Run official reviews for different dependency-ready tasks with disjoint workspace claims in parallel. +- Before the first review batch, normalize the Agent-Ops-managed `.gitignore` block once so reviews do not concurrently modify the same shared control file. +- Require exactly one valid, non-empty `Modified Files Summary` (and legacy `์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ`) in the active or recovery PLAN. Fail the task closed when any path is broad, outside the workspace, a directory, malformed, or missing. +- Atomically claim every canonical modified-file path before admitting worker, self-check, or review. A collision is a runtime wait, not a predecessor dependency. Retain the task's claim through every stage, retry, dispatcher restart, and follow-up PLAN; replace or expand its own claim only when the new set does not collide, and release it only after verifying the completed archive. +- Scope write claims to the canonical physical workspace. Separate worktrees and clones use independent state and may run in parallel; task-group filtering never narrows the claim ledger inside one workspace. + +## Prompt Contract + +Keep control prompts in English, insert absolute paths only, and do not expand these sentences unnecessarily. + +- A dispatcher child runs only while `AGENT_TASK_EXECUTION_ID` is present. +- Prefix every worker and review prompt with: `You are a child agent already launched by the dispatcher, not the orchestration caller. Execute only the assigned role directly. Do not start, monitor, or wait for orchestration through dispatch.py or orchestrate-agent-task-loop. You may run dispatch.py --validate-plan only when required by plan or code-review finalization because that mode validates one candidate PLAN without starting or monitoring orchestration.` +- Keep local self-check prompts short. Start them with: `Think in English. Final in Korean.` + +- Cloud worker: `Read {PLAN_PATH} and complete the task. Keep artifact content in English. Final in Korean.` +- Pi worker: `Think in English. Keep artifact content in English. Final in Korean. Read {PLAN_PATH} and complete the task.` +- Pi self-check full pass: `Think in English. Final in Korean. Read {PLAN_PATH}; review all work once, fix omissions, and update {CODE_REVIEW_PATH}. Keep files in English.` +- Pi self-check unchecked-item retry: `Think in English. Final in Korean. Read {PLAN_PATH}; complete every unchecked implementation item and update {CODE_REVIEW_PATH}. Keep files in English.` +- Official review: `Read {CODE_REVIEW_PATH} and start the review. Keep artifact content in English. Final in Korean.` +- Review-exit recovery: `Continue the review for {TASK_PATH}. Keep artifact content in English. Final in Korean.` +- Context escalation: `Continue from {LOCATOR_PATH}. Check the saved context and current workspace. Keep artifact content in English. Final in Korean.` + +Never ask a worker, self-check, or review model to create, edit, or summarize `WORK_LOG.md`. + +Do not treat Pi self-check exit code `0` as success by itself. Set `selfcheck_done=true` only when `## Implementation Checklist` (or legacy `## ๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ`) in `CODE_REVIEW_PATH` contains at least one Markdown list checkbox and every `[...]` checkbox value has at least one non-whitespace character. If both canonical and legacy checklist headings are present in the same file, fail closed. Accept any non-empty value, including `x`, `v`, and `โœ…`. Do not inspect `## Implementation Item Completion`, `Deviations from Plan`, `Key Design Decisions`, `Verification Results`, or final CODE_REVIEW synchronization text. Run the full self-check prompt exactly once. If its checklist condition fails, resume that successful pass's Pi native session and run the unchecked-item retry prompt up to 10 times. Each retry must resume the locator returned by the preceding successful pass so the same conversation context is preserved; never repeat the full review prompt or start a fresh retry session. Persist the latest successful context locator for dispatcher restart, and block instead of starting fresh when that context cannot be resumed. Block that task after the 10th unchecked-item retry remains incomplete, and continue draining independent work. + +After an AGY/Gemini worker exits `0`, apply the same `CODE_REVIEW_PATH` implementation-checklist regex before accepting worker completion. If it is incomplete, run a fresh quota probe: only an `exhausted` target becomes `provider-quota` and enters the existing selector failover/promotion chain; `available` or `unknown` remains a completion-evidence recovery on Gemini. + +For Pi worker recovery attempts, pass only `Read {PLAN_PATH}. Continue.` without a locator explanation. Pi self-check recovery must preserve the current full-pass or unchecked-item role and use its concise prompt. For other CLI escalation attempts, pass `Continue from {LOCATOR_PATH}. Check the saved context and current workspace. Keep artifact content in English. Final in Korean.` Preserve the collaboration prohibition and next-state-materialization sentence in official-review escalation and recovery prompts. Do not ask the model to write a separate handoff summary. + +When recovering a KST-night `local-G07`โ€“`local-G08` Laguna locator or a terminal `session-stall` locator left by an earlier dispatcher, first require the locator and native session to belong to the current physical workspace. Do not create a fresh session ID for an owned locator. Resume its native session file with `pi --session` and the existing `--session-dir`. For worker recovery pass `Think in English. Keep artifact content in English. Final in Korean. Continue this session and complete the current task.` For interrupted full self-check recovery pass `Think in English. Final in Korean. Continue. Keep files in English.` For an unchecked-item retry, pass its normal concise prompt while resuming the existing native session. After a dispatcher restart, find the owned locator and resume the same session. Count this same-session restart toward the same stage's 10-consecutive-failure limit. + +## Work-Log Contract + +- Keep exactly one `agent-task/{task_group}/WORK_LOG.md` per task group. Do not create one in a split-subtask directory. +- Allow only the dispatcher to modify this file. Worker/self-check/review models need not read or update it, and success must not depend on its prose. +- Append chronological `START`/`FINISH` rows with time, task, loop, role, attempt, model, result, and locator. In `task`, record the active role artifact relative to `agent-task/`: the PLAN path for a worker and the CODE_REVIEW path for self-check/review. In `loop`, record the PLAN identity's zero-based `plan` number (`0` is the initial plan). Record time in KST (`UTC+09:00`) as `YY-MM-DD HH:MM:SS`, for example `26-07-26 07:40:15`. Use this single timeline to inspect parallel execution order. +- Do not require the common code-review skill to preserve `WORK_LOG.md`. For split work the group log normally remains in the parent because review moves only the selected subtask. For a single task review may move the log with the task archive; after review exits, resolve exactly one source from the active group path or verified completed archive and normalize it to `work_log_N.log`. +- After every observed task in a task group has a verified complete archive and no active/running task remains, append the final `FINISH` and move the generated `WORK_LOG.md` under the final completed archive's group root as `work_log_N.log`. If an archive exists after restart but the last `START` lacks `FINISH`, do not terminate or archive while any PID/start token, per-attempt process marker, or pidless stream/native evidence remains live. Track it until execution evidence has ended and the complete archive is verified, then append `FINISH` with `reconciled:verified-complete-archive` and move the log. Use `agent-task/archive/YYYY/MM/{task_group}/` for split tasks and the actual suffix-bearing archive destination for a single task. Set `N` to one more than the maximum suffix for the same task group across all months, starting at `0`. +- If `WORK_LOG.md` archiving fails or multiple active/archive sources exist, drain other independent work and return non-terminal exit `3` for retry. Return successful exit `0` only after a completed group that generated a log has no active `WORK_LOG.md` and its `work_log_N.log` is verified. Keep an incomplete group's `WORK_LOG.md` active for blocker or exit `3` recovery. +- Split each attempt locator into `stream.log` for model stdout/stderr and `heartbeat.log` for dispatcher state. Determine health only from the newest progress in `stream.log` and native session events; never use heartbeat mtime as progress evidence. Do not copy either log into `WORK_LOG.md`. +- Keep child stdout/stderr, normalized model output, and periodic heartbeat records in locator-owned logs only. The dispatcher's user-visible stdout is an event stream and must never mirror model stream lines or heartbeat ticks. +- If locator refresh temporarily fails after an attempt starts, do not terminate a live model process or start a duplicate task. Record a warning, keep monitoring, and preserve error evidence at the next successful refresh. +- After verifying a PASS archive's `complete.log` and confirming no live execution evidence for that task, delete all of its attempt directories, including locators, native sessions, `stream.log`, `heartbeat.log`, and CLI auxiliary logs. Do not delete them while a model process or conservatively active pidless stream/native evidence remains. Treat transient deletion failure as non-terminal exit `3` for the next reconciliation without blocking the completed task or other tasks; do not return successful exit `0` while any attempt directory remains. Preserve failed or blocked attempt logs as recovery evidence. +- Record log-creation or append failure in the locator as `work-log-setup` or `work-log-runtime-write` and block the task. +- Exclude dispatcher-authored `WORK_LOG.md` changes from official-review progress/stagnation signatures. Count only real changes in PLAN/CODE_REVIEW, review logs, and the write-set. + +## Caller Lifecycle and Status Display + +- **ABSOLUTE RULE โ€” Do not stop the whole task group when a task-local blocker appears.** Delay only the blocked task and consumers that require its incomplete result as a predecessor. Keep the caller turn active until every independent ready/running task finishes. +- **ABSOLUTE RULE โ€” Scan the complete new-task candidate set only on initial dispatcher entry and immediately after creating a verified `complete.log`.** After a worker/self-check/review attempt ends or a task changes stage, reclassify only that task. After `complete.log` is created, immediately start every runnable task except currently running tasks in the same pass. Another task's execution, wait, dependency, review, or recovery state must not block a candidate. If no candidate or running task remains and only blockers and their dependent waits remain, exit with code `2`. +- Treat the dispatcher as the execution lifecycle and observation owner. It performs deterministic health checks, recovery, retries, routing, and state transitions without caller-LLM supervision. The caller owns only launch authorization, intervention after an attention event, and the `final` gate. +- Keep the caller turn suspended and launch the dispatcher as one persistent foreground execution. Use execution-layer event waiting or direct stdout streaming; never use an LLM-generated polling turn as a keepalive. Never start a duplicate dispatcher while the child is live. +- Never wrap the dispatcher in `timeout`, a short `wait_for`, or an arbitrary cancel/terminate wrapper. Tool yield or expiration of a response window is not process termination. Resume the same execution-layer wait without commentary, analysis, or inspection. +- **ABSOLUTE RULE โ€” The caller never monitors.** During normal execution or event silence, do not run a timer loop, periodically poll through the model, or inspect `ps`, dispatcher `--dry-run`, `state.json`, locator files, `stream.log`, `heartbeat.log`, or `WORK_LOG.md`. A tool yield, empty wait, routine lifecycle event, or response-window expiration does not permit caller-LLM involvement. +- Stream routine lifecycle banners directly from dispatcher stdout to the user without routing them through the caller LLM. Routine events include starts, deterministic retries/recovery, waits, per-task review results, per-task completion while other work remains, and any event for which the dispatcher has already selected the next action. +- Wake the caller LLM only for an attention event that the dispatcher cannot resolve autonomously: a verified `USER_REVIEW` decision, an exhausted terminal blocker, an unrecoverable state/log contract error, loss of the execution handle that requires targeted recovery, or terminal dispatcher exit. A warning or automatic retry is not an attention event merely because it reports an error. +- No dispatcher output, an empty wait, or a wait-window expiration is normal event silence. It never permits `final`, caller termination, a duplicate dispatcher, a state inspection, or a model wake-up. Keep the execution-layer wait attached with the longest supported window. +- A lost session/cell exists only when the execution layer reports the tracked identifier unavailable or aborted, or reports the child process exited; a normal wait return alone is insufficient. Then perform exactly one reinspection of active tasks, locators, PIDs, and state. If that snapshot proves a live dispatcher owner, do not inspect it again until an attention event is observed. Resume event waiting from the same session/cell when available; otherwise subscribe from EOF to only newly appended START/FINISH rows in the task-group WORK_LOG.md. If the fallback observer itself ends without an event while the dispatcher remains live, reattach the same EOF-only observer without reading any prior row or inspecting state. A routine START/FINISH row or direct output only confirms the subscription and does not permit model wake-up or state inspection. Only a dispatcher exit, explicit attention event, fallback-observer error, or explicit user request permits the next targeted inspection. Exit code `0` is successful terminal state. Exit code `2` is a drained blocker or explicit persistent-state-error terminal state. Exit code `3` is a non-terminal tracking state, including another dispatcher workspace lock, a live external agent, or an unexpected dispatcher interruption; inspect PID, locator, and state only after that event. +- On a scheduler/control-plane exception or unexpected exception in an individual agent coroutine, do not immediately freeze it as a task blocker or let the dispatcher event loop cancel other running agents and child processes. Monitor every independent running agent until natural completion, return non-terminal exit `3`, and let the next dispatcher reconcile file and state results. Even when the original exception is a persistent-state error, do not convert it to exit `2` if any agent was running. +- In drained-blocker terminal state, persist the orchestration group as `blocked`, directly blocked tasks as `blocked`, consumers waiting on their predecessors as `waiting`, and verified independent completed tasks as `complete` in `.git/agent-task-dispatcher/state.json`. On re-entry, set incomplete observed tasks back to orchestration state `active`, then reevaluate actual task-local blockers and dependencies. +- Persist observed tasks and the complete same-name archive baseline present at startup, regardless of `complete.log`, in `.git/agent-task-dispatcher/state.json`. If an active task disappears after child restart, recover completion only when exactly one new `complete.log` archive absent from the baseline exists; block when none or multiple exist. Do not count a late `complete.log` added to an incomplete archive that existed before execution as current-run completion. +- If existing `state.json` cannot be read or validated as a JSON object, block the dispatcher. Never replace it with empty state or reset the 10-attempt budget. Repair or explicitly handle it before rerunning. +- When a new user turn arrives, continue tracking the same overall request unless it explicitly cancels the previous request. +- Let the execution layer display `์ž‘์—…์‹œ์ž‘`, `์ž๊ฐ€๊ฒ€์ฆ์‹œ์ž‘`, `๋ฆฌ๋ทฐ์‹œ์ž‘`, `๋ฆฌ๋ทฐ์žฌ์‹œ๋„`, `Pi๋ณต๊ตฌ์žฌ์‹œ๋„`, `์„ธ์…˜์‘๋‹ต๋ณต๊ตฌ์žฌ์‹œ๋„`, `์„ธ์…˜์—ฐ๊ฒฐ์žฌ์‹œ๋„`, `๋ฆฌ๋ทฐ๊ฒฐ๊ณผ`, `์ž‘์—…๋Œ€๊ธฐ`, `์ž‘์—…์ฐจ๋‹จ`, `๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ`, and `์ž‘์—…์™„๋ฃŒ` directly from dispatcher stdout. Never duplicate them in model-authored `commentary`. Use `commentary` only when an attention event actually requires caller reasoning or a user decision. Event silence never grants `final`; only the two permissions in the absolute-priority section do. +- Determine every CLI's health/progress primarily from actual stdout/stderr in `stream.log`, plus native session events when available. Before accepting PID, marker, native-session, or stream evidence, require the locator path and recorded workspace identity to belong to the current physical workspace; accept an identity-less legacy locator only under the current store's `runs` root. Never use heartbeat mtime as progress evidence. Record workspace id, dispatcher PID, agent PID, each process start token, and the per-attempt process environment marker in the locator; namespace that marker by workspace. Another dispatcher must not start a duplicate attempt merely because the stream is quiet when the PID/start token or marker shows the same process is alive. For a locator without an agent PID, never infer stale state or duplicate recovery from elapsed time while any stream/native progress evidence exists; use only an actual terminal error or confirmed process exit as recovery evidence for every model. Run Pi with `--mode json` so `thinking_delta`, `text_delta`, and tool streams reach stdout. End an **exact** Pi toolCall-to-all-toolResult interval only when every `toolCall.id` in the preceding assistant event matches a later `toolResult.toolCallId`; never terminate the process on a time limit. If the locator lacks an agent PID during this interval, never classify it as stale or duplicate recovery based on log age; require recorded process evidence to show termination. Do not infer tool execution from `starting`, `unknown`, model reasoning, or post-toolResult state. Outside this interval, use only `stream.log` updates for Pi liveness; toolResult alone does not reset the model-response silence clock. If the stream stops for three minutes outside tool execution, store the final stream excerpt as `pi_silence_inspection` for Pi or `stream_silence_inspection` for another CLI, emit `๋ชจ๋ธ์‘๋‹ต์ ๊ฒ€`, and do not terminate the model process. Recover only from an actual terminal error or process exit. +- Detect a local-model `repetition-loop` only when the same normalized chunk repeats three consecutive times with no new tool event or file/state change. Do not infer it from similarity or semantic duplication in `thinking_delta`/`text_delta`. This signal alone must not terminate the process, block the task, trigger recovery/retry, or escalate the model; keep observing for substantive progress or an actual terminal error. +- Keep `provider-connection`, `provider-stream-disconnect`, `session-stall`, `generic-error`, `process-terminated`, context/quota/model errors, and review-control violations distinct, but make them share a budget of 10 consecutive automatic recovery failures for the same task stage. On the 10th failure, block that task and do not auto-resume after cooldown. Reset the stage counter after success. +- Record an explicit terminal blocker when the initial Pi full self-check plus 10 same-context unchecked-item retries leave the implementation checklist incomplete, or official review makes no change 10 consecutive times. +- While one task recovers or becomes blocked, continue every ready/running task that neither requires it as a predecessor nor collides with its retained workspace claim. Internal recovery or blocking must not trigger an arbitrary complete-candidate rescan. +- If review shared-state preflight fails, block only ready review tasks and still start every worker/self-check with a disjoint claim in the same pass. The complete scan after `complete.log` must preserve the existing snapshot rather than reread already running task directories, avoiding races with parallel archive moves that could stop another process. +- For KST-night `local-G07`โ€“`local-G08` Laguna locator `context-limit`/`session-stall`, prefer the Prompt Contract's same-session resume and display `Pi์„ธ์…˜์—ฐ์†์žฌ์‹œ์ž‘`. Use a fresh session and `์„ธ์…˜์‘๋‹ต๋ณต๊ตฌ์žฌ์‹œ๋„` only for other legacy Pi `session-stall` recovery. +- Do not stop for user review based on filename alone. Recognize a `user-review` terminal blocker only when the active task's `USER_REVIEW.md` contains `์ƒํƒœ: USER_REVIEW`, exactly one supported type, a concrete target, non-`์—†์Œ`/`๋ฏธ์ •` blocker rationale, unresolved user actions or decisions, and resume conditions that prevent the next safe implementation step. For `milestone-lock`, require a real `agent-roadmap/**/milestones/*.md` target. For `external-execution`, require an exact runner/device/service/access target and evidence that no authorized automatic executor can perform the required verification. If the form is incomplete or conflicts with active PLAN/CODE_REVIEW, block it as a task-state contract error instead. +- Recognize `## Code Review Result` (with `Overall Verdict: PASS|WARN|FAIL`) or legacy `## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ` (with `์ข…ํ•ฉ ํŒ์ •: PASS|WARN|FAIL`) as the review verdict. If both canonical and legacy headings are present in the same file, fail closed. Never parse the same string in implementation evidence, command output, or example text as the runtime verdict. +- Locator/raw logs under `.git/agent-task-dispatcher/runs/` are internal recovery state and may not appear in the normal project tree. Include the `locator=` path emitted when the dispatcher starts an attempt and the task-group `WORK_LOG.md` path in status updates. +- If a specified `task_group` has neither an observed active task nor a persisted completed task, return state error `unobserved-task-group` with exit code `2`; never treat it as empty completion. +- If child failure is recoverable inside the repository, continue within the 10-attempt budget. After draining independent work, report a blocker that the caller cannot clear in the current turnโ€”such as exhausted budget, required user decision, or external permissionโ€”with its path, evidence, and resume condition. + +## Failure Classification and Reporting Contract + +- Record dispatcher PID, actual agent PID, import time, source path, import-time SHA-256, attempt-start current SHA-256, and `dispatcher_source_matches_loaded` in every attempt locator. Every failure banner and subsequent status must present the locator's exact `failure_class`, `failure_source`, `provider_transport_failure_confirmed`, `dispatcher_pid`, `agent_pid`, `dispatcher_source_sha256`, source-match state, and `locator`; never summarize them into a broader cause. +- A running Python dispatcher does not hot-reload source edits. If `dispatcher_source_matches_loaded=false`, do not claim that new rules are active. Report the loaded/current hashes and execution-version difference until the process-owning session can safely exit and restart. +- Use `provider-connection` or `provider-stream-disconnect` only when original CLI terminal diagnostics contain a strong provider pattern in provider/backend/SSE context. Do not infer provider failure from `connection refused`, `dial tcp`, or `curl` peer failure in ordinary tool/test stderr. For a confirmed attempt, preserve `failure_source=provider-terminal-diagnostic`, `provider_transport_failure_confirmed=true`, `failure_evidence_source`, and `failure_evidence_excerpt` in the locator. +- Treat legacy locator `session-stall` as a record of an earlier dispatcher timeout policy, not as provider failure. During recovery, report `failure_source=dispatcher-timeout`, `provider_transport_failure_confirmed=false`, `termination_initiator=dispatcher`, and the original timeout phase/seconds. Never let the current dispatcher create a new silence timeout. +- Record a SIGTERM-family termination not initiated by the dispatcher as `process-terminated`, with `failure_source=process-termination` and `termination_initiator=unknown`. Never classify exit code `143` as provider failure without actual provider terminal evidence. +- Do not generalize one `pi -p` fresh/isolated session attempt to a Pi TUI or system-wide provider outage. Describe a system-level provider outage only with additional controlled reproduction using the same command, model, and prompt, or backend-health evidence. +- Count `process-terminated` in the same per-stage consecutive-failure budget as other automatic-recovery classes. On the 10th consecutive failure, block that task; never reset the budget after cooldown or auto-resume. A shared budget does not imply common causation or establish provider-failure evidence. + +## Procedure + +1. **Inspect state.** + - Print active tasks, routes, stages, and dependencies: + + ```bash + python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --dry-run + ``` + + - Treat `NN_...` as immediately eligible. Treat `NN+PP[,QQ...]_...` as eligible only after each predecessor's `complete.log` is found once in the active or narrow archive lookup for the same task group and predecessor execution evidence has ended. + - Never infer an implicit dependency from numeric order alone. + +2. **Run the dispatcher.** + - Run all active tasks with the default physical-workspace cap of `3`: + + ```bash + python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py + ``` + + - Run one task group: + + ```bash + python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --task-group + ``` + + - Cap total concurrent attempts across the physical workspace: + + ```bash + python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --max-parallel 2 + ``` + + - Explicitly disable the cap: + + ```bash + python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --max-parallel 0 + ``` + + - Preview classification without launching CLIs under the same cap: + + ```bash + python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --dry-run --max-parallel 2 + ``` + + - If a worker/self-check/review future ends without `complete.log`, reread only that task and run its next stage. Do not rescan the complete candidate set. + - Persist `active_stage` for a running task. After dispatcher restart, exclude that task from candidates, restore or conservatively adopt its workspace write claim, and immediately dispatch every other dependency-ready task whose claim does not collide. + - **ABSOLUTE RULE:** Scan the complete candidate set only at initial entry and immediately after creating a verified `complete.log`. In that scan, exclude tasks shown as running by current-workspace state and native session/locator evidence, then atomically admit every dependency-ready task with a non-colliding write claim. An unmet dependency or write collision excludes only that task. Exit instead of polling when no candidate remains. + - Persist Pi worker success, Pi self-check success, and official review as separate stages. If restart state is `worker_done=true` and `selfcheck_done=false`, resume on the same Pi model, not with worker or review. Run the full pass when `selfcheck_incomplete=0`; otherwise resume the persisted successful self-check context locator with an unchecked-item retry. Never replace a missing or invalid persisted context with a fresh session. + - Key persistent state to the `task/plan/tag` generation at the start of PLAN. Checklist/body edits to the same PLAN do not reset the stage; a new plan number in a follow-up PLAN does. + - Send an already completed review stub with no dispatcher execution record to review. Never send dispatcher-recorded Pi worker success to review before self-check completes. + - Start official review and worker/self-check together when they belong to different dependency-ready tasks with disjoint workspace claims. Wait for a claim owner to reach verified completion before admitting a colliding task. + - Let the dispatcher record every worker/self-check/review attempt start and finish in the task-group `WORK_LOG.md`. + - Archive `WORK_LOG.md` as `work_log_N.log` only after the final task review process exits, the dispatcher appends `FINISH`, and a complete scan finds no active/running task in that group. Accept the log at either the active group path or the verified completed single-task archive; do not impose either location contract on common plan/code-review. + +3. **Escalate and recover context.** + - Escalate `agy -> Claude -> Codex` or `Claude -> Codex` only on terminal provider error events or stderr evidence of context/output limits, provider quota/rate limits, unavailable models, or confirmed provider transport errors. For AGY, accept top-level `error`, `fatal`, `request.failed`, or `turn.failed` events; failed/rejected status with a top-level error/code; stderr; or strong `RESOURCE_EXHAUSTED`, HTTP 429, quota, or rate-limit evidence in `agy-cli.log`. For Claude, classify a `rate_limit_event` with `rate_limit_info.status=rejected`, an error `result` with `api_error_status=429` or `error=rate_limit`, or a `You've hit your session limit ยท resets ...` terminal diagnostic as `provider-quota`. Never escalate from an assistant message, source text, tool/test output, or a plain quota-configuration string in an AGY log. + - Target Codex `gpt-5.6-terra` with reasoning `high` when escalating from Claude to Codex. + - If Codex returns the same error, retry in a fresh Codex session using the locator while preserving the previous Codex model/reasoning and sharing the same stage's 10-consecutive-failure limit. Continue dispatching other tasks during recovery. + - When current source reads a locator blocked 10 times as `generic-error` by older dispatcher source, collapse those 10 failures into one terminal error and clear only that task's blocker only if all 10 terminal-evidence records for the same task/plan/role/source/execution target reclassify to the same escalatable error. Include `stream.log` and the attempt's `agy-cli.log` for AGY. Do not adjust automatically when any history is missing or mixed, or when the locator dispatcher source hash equals the current source hash. Dry-run must display this escalation recovery and next model without writing state. Live execution must choose the higher target from the locator's actual failed target, not the initial PLAN route, inherit locator context, and restore the same escalation target and locator from persisted reclassification metadata after immediate restart. + - Recover timeout, crash, process termination, permission, and ordinary implementation errors on the same target within the same stage's 10-consecutive-failure limit, preserving the actual failure class and locator. At exhaustion, block only that task and keep dispatching independent work. + - On success after escalation, record `worker_cli` and `worker_model` from the successful locator's actual target, not the initial PLAN route. + - Never escalate Pi to a cloud model. + - Use attempt identity `__p____aNN` and namespace the process marker with the physical workspace id. Record canonical workspace root/id, CLI/model/reasoning effort, PLAN/review, `WORK_LOG.md`, session ID, native session path, and raw output log in the locator. + - Store locators under repository `.git/agent-task-dispatcher/runs/`. Fall back to `${XDG_STATE_HOME}/agent-task-dispatcher//runs/` only when `.git` state is unwritable. + +4. **Converge review.** + - Run every official review in an independent Codex one-shot session with no separate numeric limit. Dispatch all ready reviews with disjoint workspace claims in parallel. + - For finalization recovery without an active PLAN, recover the review target and write claim from the archived plan log for the same task/plan/tag. Keep the claim until the completed archive is verified. + - Forbid collaboration/sub-agent tools in official review and finish inside the current one-shot session. If such a tool call appears, clean up that attempt's independent subprocess group and retry in a fresh review session. Count the failure toward the same stage's 10-consecutive-failure limit. + - Delegate PASS archive, WARN/FAIL follow-up pairs, and review-finalization recovery to the `code-review` file contract. + - Reclassify any remaining active pair and send it to worker or review. + - Declare stagnation only when the plan write-set source snapshot and review/finding artifacts are all unchanged. Display `๋ฃจํ”„์ •์ฒด๊ฒฝ๊ณ ` and retry with backoff; on the 10th unchanged attempt, block that task as `review-no-progress-limit`. + - Record a verified `USER_REVIEW.md`, dependency ambiguity, 10 repeated failures, or work-log setup/runtime-write failure only as that task's blocker. Delay only the blocker and consumers that depend on it; continue every independent ready/running task. Return drained terminal blocker exit code `2` only when no independent work remains. + +## Verification Checklist + +- [ ] Scan the complete candidate set only on initial entry and immediately after verified `complete.log`; atomically claim and start every non-running, dependency-ready, non-colliding candidate in the same pass. +- [ ] Confirm the actual CLI/model for each route matches the routing table. +- [ ] Run exactly one full fresh-session self-check only for Pi work, followed by at most 10 unchecked-item retries in that same Pi native session context when its checklist remains incomplete. +- [ ] Run every official review with Codex `gpt-5.6-sol` xhigh and dispatch dependency-ready reviews with disjoint workspace claims in parallel, subject to the global `--max-parallel` cap (no separate review-only limit). +- [ ] Locate the native session and output log for every attempt locator. +- [ ] Record every worker/self-check/review attempt `START`/`FINISH` in one task-group `WORK_LOG.md`. +- [ ] For every completed task group that generated `WORK_LOG.md`, archive a `work_log_N.log` containing the final review `FINISH` and leave no active `WORK_LOG.md`. +- [ ] Verify that a PASS task is archived and each newly released dependent task starts. +- [ ] For success, verify every task's `complete.log`. For blocker exit, verify that no ready/running task remains and only task-local blockers and their dependent waits remain. +- [ ] Verify dispatcher stdout contains lifecycle/attention events only; raw child output and heartbeat ticks remain in locator-owned logs and never require caller-LLM relay. +- [ ] On blocking, output the task, reason, and locator. +- If verification fails, stop the dispatcher and report only the cause without manually moving or overwriting active PLAN/CODE_REVIEW files. + +## Output Format + +```text +------------------------------------------ +์ž‘์—…์‹œ์ž‘: 03+01_event_contract_unit_tests +------------------------------------------ +model=pi/iop/ornith:35b +plan=/absolute/path/PLAN-local-G05.md +work_log=/absolute/path/WORK_LOG.md + +------------------------------------------ +๋ฆฌ๋ทฐ์‹œ์ž‘: 03+01_event_contract_unit_tests +------------------------------------------ +model=codex/gpt-5.6-sol xhigh +review=/absolute/path/CODE_REVIEW-local-G05.md +``` + +Use the same separator format for `์ž‘์—…๋Œ€๊ธฐ`, `์ž‘์—…์ˆ˜ํ–‰์ค‘`, `์ž๊ฐ€๊ฒ€์ฆ์‹œ์ž‘`, `๋กœ๊ทธ๋ณด์™„์žฌ์‹œ๋„`, `๋ชจ๋ธ์Šน๊ฒฉ`, `๋ฆฌ๋ทฐ๊ฒฐ๊ณผ`, `๋ฃจํ”„์ •์ฒด๊ฒฝ๊ณ `, `์ž‘์—…์ฐจ๋‹จ`, `์ž‘์—…๋กœ๊ทธ์•„์นด์ด๋ธŒ`, and `์ž‘์—…์™„๋ฃŒ`. + +## Prohibitions + +- Never print periodic heartbeat ticks or child model stdout/stderr to dispatcher stdout. Preserve them only in locator-owned logs. +- Never reevaluate PLAN/CODE_REVIEW lane or G in the dispatcher or rename those files. +- Never infer dependency from numeric order when no predecessor index is present. +- Never scan the complete archive or read archive files outside dependency candidates. +- Never ask a worker to perform official review, archive work, or create `complete.log`. +- Never treat Pi self-check as official review. +- Never depend on a model-authored handoff summary for context recovery. +- Never treat a generic failure as token/quota failure and escalate it to a higher model. +- Never resolve `USER_REVIEW.md` automatically or guess a user decision. diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/agents/openai.yaml b/agent-ops/skills/common/orchestrate-agent-task-loop/agents/openai.yaml new file mode 100644 index 0000000..9611fbf --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Agent Task Loop Orchestrator" + short_description: "Orchestrate PLAN execution and Codex review loops" + default_prompt: "Use $orchestrate-agent-task-loop to execute the active agent-task workflow." diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py new file mode 100644 index 0000000..a748032 --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py @@ -0,0 +1,7074 @@ +#!/usr/bin/env python3 +"""Dispatch every independently-ready agent-task pair until the task group completes.""" + +from __future__ import annotations + +import argparse +import asyncio +import fcntl +import importlib.util +import hashlib +import json +import os +import re +import signal +import shutil +import subprocess +import sys +import uuid +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + + +_OBSERVATION_MODULE_NAME = "agent_task_dispatcher_observation" + + +def load_sibling_observation_module(): + loaded = sys.modules.get(_OBSERVATION_MODULE_NAME) + if loaded is not None: + return loaded + spec = importlib.util.spec_from_file_location( + _OBSERVATION_MODULE_NAME, + Path(__file__).with_name("dispatcher_observation.py"), + ) + if spec is None or spec.loader is None: + raise RuntimeError("failed to load dispatcher observation module") + module = importlib.util.module_from_spec(spec) + sys.modules[_OBSERVATION_MODULE_NAME] = module + try: + spec.loader.exec_module(module) + except BaseException: + sys.modules.pop(_OBSERVATION_MODULE_NAME, None) + raise + return module + + +observation = load_sibling_observation_module() +SEP = observation.SEP +banner = observation.banner +attempt_event = observation.attempt_event +validation_claim = observation.validation_claim + +PLAN_RE = re.compile(r"^PLAN-(local|cloud)-G(0[1-9]|10)\.md$") +REVIEW_RE = re.compile(r"^CODE_REVIEW-(local|cloud)-G(0[1-9]|10)\.md$") +PLAN_LOG_RE = re.compile( + r"^plan_(local|cloud)_G(0[1-9]|10)_(0|[1-9][0-9]*)\.log$" +) +REVIEW_LOG_RE = re.compile( + r"^code_review_(local|cloud)_G(0[1-9]|10)_(0|[1-9][0-9]*)\.log$" +) +SUBTASK_RE = re.compile(r"^(?P\d{2})(?:\+(?P\d{2}(?:,\d{2})*))?_[a-z0-9_]+$") +MODIFIED_FILES_HEADINGS = ("Modified Files Summary", "์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ") +MODIFIED_FILES_HEADER_CELLS = frozenset({"file", "files", "path", "paths", "ํŒŒ์ผ", "๊ฒฝ๋กœ"}) +PLACEHOLDER_PATH_RE = re.compile( + r"(?:[<>{}]|\.\.\.|(?:^|[/_.-])(?:tbd|todo|placeholder)(?:$|[/_.-]))", + re.IGNORECASE, +) +IMPLEMENTATION_CHECKLIST_HEADINGS = ("Implementation Checklist", "๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ") +# The canonical English and legacy Korean verdict contracts are paired: a +# heading only accepts the verdict label of its own schema. Mixed pairs are not +# a documented schema and must fail closed. +CODE_REVIEW_RESULT_SCHEMAS = ( + ("Code Review Result", "Overall Verdict"), + ("์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ", "์ข…ํ•ฉ ํŒ์ •"), +) +USER_REVIEW_SCHEMAS = ( + { + "status_heading": "Status", + "reason_heading": "Reason", + "type_label": "Type", + "target_label": "Target", + "evidence_heading": "Blocking Evidence", + "evidence_label": "Blocking rationale", + "decision_headings": ("Required User Action",), + "resume_heading": "Resume Condition", + }, + { + "status_heading": "์ƒํƒœ", + "reason_heading": "์‚ฌ์œ ", + "type_label": "์œ ํ˜•", + "target_label": "์—ฐ๊ฒฐ ๋Œ€์ƒ", + "evidence_heading": "์ฐจ๋‹จ ๊ทผ๊ฑฐ", + "evidence_label": "์ฐจ๋‹จ ํŒ๋‹จ ๊ทผ๊ฑฐ", + "decision_headings": ("์‚ฌ์šฉ์ž ์กฐ์น˜ ๋˜๋Š” ๊ฒฐ์ •", "์—ฐ๊ฒฐ ๊ฒฐ์ • ํ•„์š”"), + "resume_heading": "์žฌ๊ฐœ ์กฐ๊ฑด", + }, +) +VERDICT_SCHEMA_MATCHERS = tuple( + ( + re.compile(rf"^##\s*{re.escape(heading)}[ \t]*$", re.MULTILINE), + re.compile( + rf"^(?:-\s*)?(?:\*\*)?{re.escape(label)}(?:\*\*)?\s*:\s*(PASS|WARN|FAIL)[ \t]*$", + re.MULTILINE, + ), + re.compile( + rf"^###\s+{re.escape(label)}[ \t]*$\s*^(?:\*\*)?(PASS|WARN|FAIL)(?:\*\*)?[ \t]*$", + re.MULTILINE, + ), + ) + for heading, label in CODE_REVIEW_RESULT_SCHEMAS +) +PLAN_IDENTITY_RE = re.compile( + r"" +) +IMPLEMENTATION_CHECKBOX_RE = re.compile( + r"^-\s+\[([^\]\r\n]*)\]", re.MULTILINE +) +WORK_LOG_NAME = "WORK_LOG.md" +WORK_LOG_ARCHIVE_RE = re.compile(r"^work_log_(\d+)\.log$") +WORK_LOG_HEADER = ( + "| seq | time | event | task | loop | role | attempt | model | result | locator |" +) +WORK_LOG_SEPARATOR = "|---:|---|---|---|---:|---|---:|---|---|---|" +LEGACY_WORK_LOG_HEADER = ( + "| seq | time | event | task | role | attempt | model | result | locator |" +) +LEGACY_WORK_LOG_SEPARATOR = "|---:|---|---|---|---|---:|---|---|---|" +WORK_LOG_EXECUTION_LOOP_RE = re.compile( + r"__p(?P\d+)__(?:worker|selfcheck|review)__a\d+(?=$|[/\\])" +) +AGENT_PROCESS_MARKER_ENV = "AGENT_TASK_EXECUTION_ID" +DISPATCHER_CHILD_BOUNDARY_PROMPT = ( + "You are a child agent already launched by the dispatcher, not the " + "orchestration caller. Execute only the assigned role directly. Do not " + "start, monitor, or wait for orchestration through dispatch.py or " + "orchestrate-agent-task-loop. You may run dispatch.py --validate-plan only " + "when required by plan or code-review finalization because that mode " + "validates one candidate PLAN without starting or monitoring orchestration." +) +SELF_CHECK_PROMPT_PREFIX = "Think in English. Final in Korean." +KST = timezone(timedelta(hours=9), name="KST") +DEFAULT_MAX_PARALLEL = 3 + + +def validated_max_parallel(value: int) -> int: + """Validate and return a non-negative integer for --max-parallel. + + Rejects negative values and non-integer types. Used both for CLI + argument parsing and for programmatic callers that may pass arbitrary + namespaces. + """ + if not isinstance(value, int) or isinstance(value, bool): + raise ValueError( + f"--max-parallel must be an integer >= 0, got {value!r}" + ) + if value < 0: + raise ValueError( + f"--max-parallel must be >= 0, got {value}" + ) + return value + + +STREAM_HEARTBEAT_SECONDS = 30 +PI_MODEL_RESPONSE_STALL_SECONDS = 3 * 60 +PI_SESSION_SCHEMA_VERSION = 3 +RECOVERY_FAILURE_LIMIT = 10 +SELF_CHECK_UNCHECKED_RETRY_LIMIT = 10 +REVIEW_NO_PROGRESS_LIMIT = 10 +PROVIDER_TRANSPORT_FAILURES = frozenset( + {"provider-connection", "provider-stream-disconnect"} +) +FAILURE_EVIDENCE_LIMIT = 2000 +# Used only to reject a stale locator whose dispatcher and agent PIDs are both +# gone. A live process is inspected after silence; it is never killed solely by +# this fallback clock. +CODEX_STREAM_STALL_SECONDS = 5 * 60 +PROMOTABLE_PATTERNS = { + "context-limit": [ + r"context (?:length|window)", r"maximum context", r"prompt is too long", + r"too many tokens", r"token limit", r"exceeded.{0,40}token", + r"output (?:token )?limit", r"maximum output", r"\bmax_tokens\b", + r"response (?:is )?too long", + ], + "provider-quota": [ + r"rate.?limit", r"\bquota\b", r"resource[_ ]?exhausted", r"\b429\b", + r"too many requests", r"usage limit", r"capacity limit", + r"\bsession limit\b", + ], + "model-unavailable": [ + r"model.{0,40}(?:not found|unavailable)", r"overloaded", + r"temporarily unavailable", + ], + "provider-connection": [ + r"\bprovider[_ -]?tunnel[_ -]?error\b", + ( + r"(?:provider|backend|/v1/chat/completions|/v1/responses)" + r".{0,160}(?:connection refused|dial tcp)" + ), + ], + "provider-stream-disconnect": [ + r"backend connection failed during streaming request", + r"sse stream before done", + r"llama-server was unresponsive", + r"backend watchdog", + r"model will be reloaded automatically on retry", + ( + r"(?:provider|backend|sse).{0,160}" + r"curl error: failure when receiving data from the peer" + ), + ], +} +PROMOTABLE_FAILURES = frozenset( + {"context-limit", "provider-quota", "model-unavailable"} +) +CLOUD_PROMOTION_FAILURES = PROMOTABLE_FAILURES | PROVIDER_TRANSPORT_FAILURES +QUALIFIED_FAILOVER_FAILURES = frozenset( + {"provider-quota", "context-limit", "model-unavailable", "provider-stream-disconnect"} +) + + +class DispatcherAlreadyRunning(RuntimeError): + """A live dispatcher owns the workspace; this is non-terminal tracking state.""" + + +class DispatcherTerminalStateError(RuntimeError): + """Persistent workspace state prevents safe dispatch before work can start.""" + + +class DispatcherInterruptedWithActiveWork(RuntimeError): + """A control-plane error occurred after one or more agent tasks had started.""" + + +class ExecutionDecisionError(RuntimeError): + """A selector decision is invalid for this task and must fail closed.""" + + +def now_iso() -> str: + return datetime.now(timezone.utc).isoformat() + + +def work_log_now_kst() -> str: + return datetime.now(KST).strftime("%y-%m-%d %H:%M:%S") + + +def sha256_file(path: Path | None) -> str: + if path is None or not path.exists(): + return "none" + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(65536), b""): + digest.update(chunk) + return digest.hexdigest() + + +DISPATCHER_SOURCE_PATH = Path(__file__).resolve() +DISPATCHER_SOURCE_SHA256 = sha256_file(DISPATCHER_SOURCE_PATH) +DISPATCHER_PROCESS_STARTED_AT = now_iso() + + +def dispatcher_source_provenance() -> dict[str, Any]: + current_sha256 = sha256_file(DISPATCHER_SOURCE_PATH) + return { + "dispatcher_pid": os.getpid(), + "dispatcher_process_start_token": process_start_token(os.getpid()), + "dispatcher_process_started_at": DISPATCHER_PROCESS_STARTED_AT, + "dispatcher_source_path": str(DISPATCHER_SOURCE_PATH), + "dispatcher_source_sha256": DISPATCHER_SOURCE_SHA256, + "dispatcher_source_current_sha256": current_sha256, + "dispatcher_source_matches_loaded": current_sha256 == DISPATCHER_SOURCE_SHA256, + } + + +def plan_identity(path: Path | None) -> str: + if path is None or not path.exists(): + return "none" + text = path.read_text(encoding="utf-8", errors="replace")[:1024] + match = PLAN_IDENTITY_RE.search(text) + if not match: + return sha256_file(path) + identity = "\0".join(match.group(name) for name in ("task", "plan", "tag")) + return "meta:" + hashlib.sha256(identity.encode()).hexdigest() + + +def write_json(path: Path, value: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(path.suffix + ".tmp") + temporary.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + temporary.replace(path) + + +def milestone_work_log_path(task: Task) -> Path: + return ( + task.directory.parent / WORK_LOG_NAME + if "/" in task.name + else task.directory / WORK_LOG_NAME + ) + + +def work_log_task_name(task: Task, role: str) -> str: + """Return the role-specific active artifact shown in the task column.""" + artifact = task.plan if role == "worker" else task.review + if artifact is None: + return task.name + return f"{task.name}/{artifact.name}" + + +def work_log_loop_number(task: Task, execution_id: str) -> int: + """Keep one loop identity even when a reviewer archives the active PLAN.""" + match = WORK_LOG_EXECUTION_LOOP_RE.search(execution_id) + return int(match.group("loop")) if match else plan_number(task) + + +def append_work_log_event( + path: Path, + *, + task_name: str, + loop: int, + event: str, + execution_id: str, + role: str, + attempt: int, + model: str, + result: str, + locator: Path, +) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("a+", encoding="utf-8") as stream: + fcntl.flock(stream.fileno(), fcntl.LOCK_EX) + try: + stream.seek(0) + text = stream.read() + if not text: + stream.write( + "# Milestone Work Log\n\n" + "> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n" + f"{WORK_LOG_HEADER}\n" + f"{WORK_LOG_SEPARATOR}\n" + ) + sequence = 1 + else: + stream.seek(0, os.SEEK_END) + if WORK_LOG_HEADER not in text: + if not text.endswith("\n"): + stream.write("\n") + stream.write( + "\n## Dispatcher Timeline\n\n" + "> Dispatcher-owned. Workers and reviewers do not edit this section.\n\n" + f"{WORK_LOG_HEADER}\n" + f"{WORK_LOG_SEPARATOR}\n" + ) + sequence = 1 + max( + ( + int(match.group(1)) + for match in re.finditer(r"^\|\s*(\d+)\s*\|", text, re.MULTILINE) + ), + default=0, + ) + if not text.endswith("\n"): + stream.write("\n") + + def cell(value: Any) -> str: + return str(value).replace("|", r"\|").replace("\n", " ") + + stream.write( + f"| {sequence} | {work_log_now_kst()} | {cell(event)} | " + f"{cell(task_name)} | " + f"{loop} | {cell(role)} | {attempt} | {cell(model)} | {cell(result)} | " + f"{cell(locator.resolve())} |\n" + ) + stream.flush() + finally: + fcntl.flock(stream.fileno(), fcntl.LOCK_UN) + return path + + +def append_milestone_event( + task: Task, + *, + event: str, + execution_id: str, + role: str, + attempt: int, + model: str, + result: str, + locator: Path, +) -> Path: + return append_work_log_event( + milestone_work_log_path(task), + task_name=work_log_task_name(task, role), + loop=work_log_loop_number(task, execution_id), + event=event, + execution_id=execution_id, + role=role, + attempt=attempt, + model=model, + result=result, + locator=locator, + ) + + +def safe_name(value: str) -> str: + return re.sub(r"[^A-Za-z0-9_.-]+", "__", value).strip("_") or "task" + + +@dataclass(frozen=True) +class AgentSpec: + cli: str + model: str + display: str + local_pi: bool = False + reasoning_effort: str | None = None + + +def effective_reasoning_effort(spec: AgentSpec) -> str | None: + if spec.cli in {"codex", "claude"}: + return spec.reasoning_effort or "xhigh" + return None + + +def agent_spec_from_record(record: dict[str, Any]) -> AgentSpec | None: + cli = str(record.get("cli") or "") + model = str(record.get("model") or "") + if not cli or not model: + return None + reasoning_effort = record.get("reasoning_effort") + if reasoning_effort is not None: + reasoning_effort = str(reasoning_effort) + local_pi = cli == "pi" + if cli in {"codex", "claude"}: + effort = reasoning_effort or "xhigh" + display = f"{cli}/{model} {effort}" + elif cli == "pi": + display = f"pi/iop/{model}" + else: + display = f"{cli}/{model}" + return AgentSpec( + cli, + model, + display, + local_pi=local_pi, + reasoning_effort=reasoning_effort, + ) + + +def agent_spec_from_locator(locator: Path | None) -> AgentSpec | None: + if locator is None: + return None + try: + record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(record, dict): + return None + return agent_spec_from_record(record) + + +@dataclass(frozen=True) +class PiSessionState: + phase: str + expected_tool_call_ids: tuple[str, ...] = () + completed_tool_call_ids: tuple[str, ...] = () + pending_tool_call_ids: tuple[str, ...] = () + reason: str = "" + + +@dataclass(frozen=True) +class LegacyPromotionRecovery: + locator: Path + role: str + failure_class: str + evidence: str + evidence_source: str + prior_dispatcher_sha256: str + failed_cli: str + failed_model: str + failed_reasoning_effort: str | None + + +def failed_spec_from_recovery( + recovery: LegacyPromotionRecovery, +) -> AgentSpec: + record = { + "cli": recovery.failed_cli, + "model": recovery.failed_model, + "reasoning_effort": recovery.failed_reasoning_effort, + } + spec = agent_spec_from_record(record) + if spec is None: + raise ValueError("legacy promotion recovery์— failed agent identity๊ฐ€ ์—†๋‹ค") + return spec + + +@dataclass +class Task: + name: str + directory: Path + plan: Path | None + review: Path | None + user_review: Path | None + recovery: bool + errors: list[str] = field(default_factory=list) + index: int = 0 + deps: tuple[str, ...] = () + write_set: set[str] = field(default_factory=set) + write_set_known: bool = False + plan_hash: str = "none" + lane: str | None = None + grade: int | None = None + + +def task_target_files(task: Task) -> list[str]: + """Return the canonical plan-declared file targets for dispatcher output.""" + return sorted(str(Path(path).resolve()) for path in task.write_set) + + +def task_observation_lines(task: Task) -> list[str]: + """Render the task directory and declared file targets for operator logs.""" + lines = [f"task_dir={task.directory.resolve()}"] + targets = task_target_files(task) + if targets: + lines.extend(f"target_file={path}" for path in targets) + else: + lines.append( + "target_file=unavailable (Modified Files Summary has no valid file claim)" + ) + return lines + + +def next_execution_identity( + store: StateStore, + task: Task, + role: str, +) -> tuple[int, str]: + attempt = store.next_attempt(task, role) + identity = ( + f"{safe_name(task.name)}__p{plan_number(task)}__{role}__a{attempt:02d}" + ) + return attempt, identity + + +class StateStore: + def __init__(self, workspace: Path): + self.workspace = workspace.resolve() + self.workspace_id = hashlib.sha256( + str(self.workspace).encode() + ).hexdigest()[:16] + git_marker = self.workspace / ".git" + git_directory: Path | None = None + if git_marker.is_dir(): + git_directory = git_marker + elif git_marker.is_file(): + marker = git_marker.read_text(encoding="utf-8", errors="replace").strip() + if marker.startswith("gitdir:"): + candidate = Path(marker.split(":", 1)[1].strip()) + git_directory = ( + candidate + if candidate.is_absolute() + else (self.workspace / candidate).resolve() + ) + candidates = [] + if git_directory is not None: + candidates.append(git_directory / "agent-task-dispatcher") + state_base = Path(os.environ.get("XDG_STATE_HOME", str(Path.home() / ".local" / "state"))) + candidates.append(state_base / "agent-task-dispatcher" / self.workspace_id) + self.root = candidates[-1] + last_error: OSError | None = None + for candidate in candidates: + try: + candidate.mkdir(parents=True, exist_ok=True) + self.root = candidate + last_error = None + break + except OSError as exc: + last_error = exc + if last_error is not None: + raise DispatcherTerminalStateError( + f"dispatcher state ๋””๋ ‰ํ„ฐ๋ฆฌ๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์—†๋‹ค: {candidates}" + ) from last_error + self.path = self.root / "state.json" + self.runs = self.root / "runs" + self.runs.mkdir(exist_ok=True) + self.lock_stream = (self.root / "dispatcher.lock").open("a+", encoding="utf-8") + try: + fcntl.flock(self.lock_stream.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError as exc: + self.lock_stream.seek(0) + owner = self.lock_stream.read().strip() or "owner metadata unavailable" + self.lock_stream.close() + raise DispatcherAlreadyRunning( + f"๊ฐ™์€ workspace์˜ dispatcher๊ฐ€ ์ด๋ฏธ ์‹คํ–‰ ์ค‘์ด๋‹ค: " + f"{self.root}; owner={owner}" + ) from exc + try: + self.lock_stream.seek(0) + self.lock_stream.truncate() + self.lock_stream.write( + json.dumps(dispatcher_source_provenance(), ensure_ascii=False) + "\n" + ) + self.lock_stream.flush() + except OSError as exc: + self.lock_stream.close() + raise DispatcherTerminalStateError( + f"dispatcher lock owner metadata๋ฅผ ๊ธฐ๋กํ•  ์ˆ˜ ์—†๋‹ค: {self.root}" + ) from exc + if self.path.exists(): + try: + self.data = json.loads(self.path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError) as exc: + self.lock_stream.close() + raise DispatcherTerminalStateError( + f"dispatcher state๋ฅผ ์ฝ์„ ์ˆ˜ ์—†๋‹ค: {self.path}" + ) from exc + if not isinstance(self.data, dict): + self.lock_stream.close() + raise DispatcherTerminalStateError( + f"dispatcher state๊ฐ€ object๊ฐ€ ์•„๋‹ˆ๋‹ค: {self.path}" + ) + else: + self.data = {"tasks": {}, "attempt_counters": {}} + try: + self._bind_workspace_identity() + self.write_claim_snapshot() + except DispatcherTerminalStateError: + self.lock_stream.close() + raise + + def _bind_workspace_identity(self) -> None: + expected = { + "id": self.workspace_id, + "root": str(self.workspace), + } + current = self.data.get("workspace_identity") + if current is None: + self.data["workspace_identity"] = expected + return + if not isinstance(current, dict): + raise DispatcherTerminalStateError( + f"dispatcher workspace identity๊ฐ€ object๊ฐ€ ์•„๋‹ˆ๋‹ค: {self.path}" + ) + if ( + current.get("id") != expected["id"] + or current.get("root") != expected["root"] + ): + raise DispatcherTerminalStateError( + "dispatcher state์˜ workspace identity๊ฐ€ ํ˜„์žฌ checkout๊ณผ ๋‹ค๋ฅด๋‹ค: " + f"state={current} current={expected}" + ) + + def write_claim_snapshot(self) -> dict[str, dict[str, Any]]: + raw = self.data.setdefault("write_claims", {}) + if not isinstance(raw, dict): + raise DispatcherTerminalStateError( + f"dispatcher write_claims๊ฐ€ object๊ฐ€ ์•„๋‹ˆ๋‹ค: {self.path}" + ) + snapshot: dict[str, dict[str, Any]] = {} + for owner, value in raw.items(): + if not isinstance(owner, str) or not owner or not isinstance(value, dict): + raise DispatcherTerminalStateError( + f"dispatcher write claim ํ˜•์‹์ด ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: owner={owner!r}" + ) + paths = value.get("paths") + exclusive = value.get("exclusive", False) + if not isinstance(paths, list) or not isinstance(exclusive, bool): + raise DispatcherTerminalStateError( + f"dispatcher write claim ๊ฒฝ๋กœ ํ˜•์‹์ด ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: owner={owner}" + ) + if value.get("workspace_id") != self.workspace_id: + raise DispatcherTerminalStateError( + "dispatcher write claim์˜ workspace identity๊ฐ€ ๋‹ค๋ฅด๋‹ค: " + f"owner={owner}" + ) + canonical: list[str] = [] + for raw_path in paths: + if not isinstance(raw_path, str) or not raw_path: + raise DispatcherTerminalStateError( + f"dispatcher write claim ๊ฒฝ๋กœ๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: owner={owner}" + ) + path = Path(raw_path) + resolved = path.resolve() + try: + resolved.relative_to(self.workspace) + except ValueError as exc: + raise DispatcherTerminalStateError( + "dispatcher write claim์ด workspace ๋ฐ–์„ ๊ฐ€๋ฆฌํ‚จ๋‹ค: " + f"owner={owner} path={raw_path}" + ) from exc + if not path.is_absolute() or str(resolved) != raw_path or resolved == self.workspace: + raise DispatcherTerminalStateError( + "dispatcher write claim ๊ฒฝ๋กœ๊ฐ€ canonical file์ด ์•„๋‹ˆ๋‹ค: " + f"owner={owner} path={raw_path}" + ) + canonical.append(raw_path) + if (not canonical and not exclusive) or len(canonical) != len(set(canonical)): + raise DispatcherTerminalStateError( + f"dispatcher write claim ๊ฒฝ๋กœ ์ง‘ํ•ฉ์ด ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: owner={owner}" + ) + record = dict(value) + record["paths"] = sorted(canonical) + snapshot[owner] = record + return snapshot + + def replace_write_claims( + self, + claims: dict[str, dict[str, Any]], + *, + persist: bool, + ) -> None: + previous = self.data.get("write_claims", {}) + self.data["write_claims"] = claims + try: + self.write_claim_snapshot() + if persist and previous != claims: + self.save() + except BaseException: + self.data["write_claims"] = previous + raise + + def adopt_active_write_claim(self, task: Task) -> None: + claims = self.write_claim_snapshot() + if task.name in claims: + return + timestamp = now_iso() + claims[task.name] = { + "task": task.name, + "plan_hash": task.plan_hash, + "paths": sorted(task.write_set) if task.write_set_known else [], + "exclusive": not task.write_set_known, + "workspace_id": self.workspace_id, + "acquired_at": timestamp, + "updated_at": timestamp, + "source": "active-recovery", + } + self.replace_write_claims(claims, persist=True) + + def release_write_claim(self, task_name: str, *, persist: bool = True) -> bool: + claims = self.write_claim_snapshot() + if task_name not in claims: + return False + del claims[task_name] + self.replace_write_claims(claims, persist=persist) + return True + + def save(self) -> None: + write_json(self.path, self.data) + + def close(self) -> None: + if not self.lock_stream.closed: + self.lock_stream.close() + + def task_state(self, task: Task) -> dict[str, Any]: + tasks = self.data.setdefault("tasks", {}) + current = tasks.get(task.name) + if not current or current.get("plan_hash") != task.plan_hash: + current = { + "plan_hash": task.plan_hash, + "worker_done": False, + "worker_cli": None, + "worker_model": None, + "selfcheck_done": False, + "blocked": None, + "active_stage": None, + "active_locator": None, + "review_no_progress": 0, + "selfcheck_incomplete": 0, + "selfcheck_context_locator": None, + "recovery_failures": {}, + "execution_decisions": {}, + "route_transition_history": [], + "stage_failure_budgets": {}, + "retry_quota_refresh_pending": False, + "retry_quota_refresh_context": None, + "blocker_evidence": None, + } + tasks[task.name] = current + self.save() + return current + + def peek_task_state(self, task: Task) -> dict[str, Any]: + current = self.data.get("tasks", {}).get(task.name) + if current and current.get("plan_hash") == task.plan_hash: + return dict(current) + return { + "plan_hash": task.plan_hash, + "worker_done": False, + "worker_cli": None, + "worker_model": None, + "selfcheck_done": False, + "blocked": None, + "active_stage": None, + "active_locator": None, + "review_no_progress": 0, + "selfcheck_incomplete": 0, + "selfcheck_context_locator": None, + "recovery_failures": {}, + "execution_decisions": {}, + "route_transition_history": [], + "retry_quota_refresh_pending": False, + "retry_quota_refresh_context": None, + "blocker_evidence": None, + } + + def update_task(self, task: Task, **values: Any) -> None: + state = self.task_state(task) + state.update(values) + self.save() + + def mark_active(self, task: Task, stage: str, locator: Path | None = None) -> None: + self.update_task( + task, + active_stage=stage, + active_locator=str(locator) if locator else None, + active_started_at=now_iso(), + ) + + def clear_active(self, task: Task) -> None: + self.update_task( + task, + active_stage=None, + active_locator=None, + active_started_at=None, + ) + + def consume_matching_retry_handoff(self, task: Task, locator_path: str) -> bool: + """Atomically consume a pending retry handoff when a matching locator exists. + + When a worker writes its locator and sets active_locator, the pending + retry_quota_refresh state must be cleared in the same transaction. + This prevents a crash window where a restart sees the pending handoff + and creates a duplicate invocation. + + Returns True if the pending handoff was consumed, False if no matching + locator was found (active_locator is None or differs from locator_path). + """ + state = self.task_state(task) + active = state.get("active_locator") + if active != locator_path: + return False + pending = state.get("retry_quota_refresh_pending") + if not pending: + return False + context = state.get("retry_quota_refresh_context") + if not isinstance(context, dict): + return False + context_locator = context.get("locator") + if context_locator != locator_path: + return False + # Snapshot current state to restore on save failure. This ensures the + # crash window is not widened by a partial consume: if the save fails, + # the pending handoff remains intact both in-memory and on-disk. + pre_state = dict(state) + pre_keys = set(state.keys()) + pre_values = {k: state.get(k) for k in ["retry_quota_refresh_pending", "retry_quota_refresh_context"]} + try: + self.update_task( + task, + retry_quota_refresh_pending=False, + retry_quota_refresh_context=None, + ) + except Exception: + # Restore the pre-consume state on any failure. + for k, v in pre_values.items(): + state[k] = v + # Restore key existence: if a key existed before, restore its value; + # if a key did not exist before, ensure it is not present. + for k in list(state.keys()): + if k not in pre_keys: + del state[k] + for k, v in pre_values.items(): + if k not in state: + state[k] = v + raise + return True + + def commit_retry_handoff_locator( + self, task: Task, handoff_id: str, locator_path: str, + ) -> bool: + """Atomically commit a new locator and consume a matching pending retry handoff. + + This is the durable one-save transition for retry handoff. It matches + the pending handoff by stable handoff_id (not by locator path, which + changes on each attempt) and atomically updates active_locator, clears + the pending flag, and clears the context in a single save. + + On save failure the pre-state is fully restored both in-memory and on + disk so the crash window is not widened. + + Returns True if a matching pending handoff was consumed, False if no + pending handoff with the given handoff_id was found. + """ + state = self.task_state(task) + pending = state.get("retry_quota_refresh_pending") + if not pending: + return False + context = state.get("retry_quota_refresh_context") + if not isinstance(context, dict): + return False + if context.get("handoff_id") != handoff_id: + return False + # Snapshot current state to restore on save failure. + pre_state = dict(state) + pre_keys = set(state.keys()) + pre_values = { + k: state.get(k) + for k in [ + "retry_quota_refresh_pending", + "retry_quota_refresh_context", + "active_locator", + ] + } + try: + self.update_task( + task, + active_locator=locator_path, + retry_quota_refresh_pending=False, + retry_quota_refresh_context=None, + ) + except Exception: + for k, v in pre_values.items(): + state[k] = v + for k in list(state.keys()): + if k not in pre_keys: + del state[k] + for k, v in pre_values.items(): + if k not in state: + state[k] = v + raise + return True + + def next_attempt(self, task: Task, role: str) -> int: + key = f"{task.name}|{task.plan_hash}|{role}" + counters = self.data.setdefault("attempt_counters", {}) + number = int(counters.get(key, 0)) + counters[key] = number + 1 + self.save() + return number + + def clear_blocked(self, task_group: str | None = None) -> None: + prefix = f"{task_group}/" if task_group else None + for task_name, value in self.data.get("tasks", {}).items(): + if ( + task_group is not None + and task_name != task_group + and not task_name.startswith(prefix) + ): + continue + value["blocked"] = None + value["review_no_progress"] = 0 + value["selfcheck_incomplete"] = 0 + value["selfcheck_context_locator"] = None + value["recovery_failures"] = {} + value["stage_failure_budgets"] = {} + value["retry_quota_refresh_pending"] = False + self.save() + + def mark_retry_quota_refresh(self, task_group: str | None = None, workspace: Path | None = None) -> None: + prefix = f"{task_group}/" if task_group else None + for task_name, value in self.data.get("tasks", {}).items(): + if ( + task_group is not None + and task_name != task_group + and not task_name.startswith(prefix) + ): + continue + if not value.get("blocked"): + continue + blocker_evidence = value.get("blocker_evidence") if isinstance(value.get("blocker_evidence"), dict) else {} + decisions = value.get("execution_decisions", {}) + worker_decision = decisions.get("worker") if isinstance(decisions, dict) else None + role = blocker_evidence.get("role") + failure_class = blocker_evidence.get("failure_class") + locator = blocker_evidence.get("locator") + selected = blocker_evidence.get("selected") + work_unit_id = blocker_evidence.get("work_unit_id") + qualified = ( + role == "worker" + and failure_class in QUALIFIED_FAILOVER_FAILURES + and isinstance(locator, str) + and locator.strip() + and isinstance(selected, dict) + and isinstance(work_unit_id, str) + and isinstance(worker_decision, dict) + and worker_decision.get("work_unit_id") == work_unit_id + ) + handoff_id = str(uuid.uuid4()) + retry_context = ({ + "role": role, + "failure_class": failure_class, + "locator": locator, + "selected": selected, + "work_unit_id": work_unit_id, + "handoff_id": handoff_id, + } if qualified else None) + + value["blocked"] = None + value["review_no_progress"] = 0 + value["selfcheck_incomplete"] = 0 + value["selfcheck_context_locator"] = None + value["recovery_failures"] = {} + value["stage_failure_budgets"] = {} + value["retry_quota_refresh_pending"] = qualified + value["retry_quota_refresh_context"] = retry_context + value["blocker_evidence"] = None + self.save() + + + def prepare_orchestration( + self, + scope: str, + tasks: list[Task], + workspace: Path, + ) -> None: + orchestrations = self.data.setdefault("orchestrations", {}) + current = orchestrations.get(scope) + if current is None or (current.get("status") == "complete" and tasks): + current = {"status": "running", "tasks": {}} + orchestrations[scope] = current + changed = False + tracked = current.setdefault("tasks", {}) + for task in tasks: + record = tracked.get(task.name) + if record is None: + tracked[task.name] = { + "status": "active", + "archive": None, + "archive_baseline": [ + str(path.resolve()) + for path in matching_archive_directories_by_name( + workspace, + task.name, + require_complete=False, + ) + ], + } + changed = True + continue + if record.get("status") != "complete" and ( + record.get("status") != "active" or "reason" in record + ): + record["status"] = "active" + record.pop("reason", None) + changed = True + if changed or current.get("status") != "running": + current["status"] = "running" + self.save() + + def mark_orchestration_task_complete( + self, + scope: str, + task_name: str, + archive: str | Path, + ) -> None: + archive_path = Path(archive).resolve() + if not archive_path.is_dir() or not (archive_path / "complete.log").is_file(): + raise RuntimeError( + f"์™„๋ฃŒ archive์— complete.log๊ฐ€ ์—†๋‹ค: task={task_name} archive={archive_path}" + ) + current = self.data.setdefault("orchestrations", {}).setdefault( + scope, {"status": "running", "tasks": {}} + ) + tracked = current.setdefault("tasks", {}) + record = tracked.setdefault( + task_name, + {"status": "active", "archive": None, "archive_baseline": []}, + ) + record.update(status="complete", archive=str(archive_path)) + record.pop("reason", None) + self.release_write_claim(task_name, persist=False) + self.save() + cleanup_completed_task_attempt_logs(self.runs, task_name) + + def mark_orchestration_blocked( + self, + scope: str, + outcomes: dict[str, tuple[str, str]], + ) -> None: + current = self.data.setdefault("orchestrations", {}).setdefault( + scope, {"status": "running", "tasks": {}} + ) + current["status"] = "blocked" + tracked = current.setdefault("tasks", {}) + for task_name, (status, reason) in outcomes.items(): + record = tracked.setdefault( + task_name, + { + "status": "active", + "archive": None, + "archive_baseline": [], + }, + ) + if record.get("status") == "complete": + continue + record.update(status=status, reason=reason) + self.save() + + def reconcile_orchestration( + self, + scope: str, + workspace: Path, + active_or_running: set[str], + ) -> tuple[dict[str, str], dict[str, str]]: + current = self.data.setdefault("orchestrations", {}).setdefault( + scope, {"status": "running", "tasks": {}} + ) + completed: dict[str, str] = {} + errors: dict[str, str] = {} + changed = False + for task_name, record in current.setdefault("tasks", {}).items(): + if record.get("status") == "complete": + archive = str(record.get("archive") or "") + if archive and (Path(archive) / "complete.log").is_file(): + completed[task_name] = archive + if task_name not in active_or_running: + changed = ( + self.release_write_claim(task_name, persist=False) + or changed + ) + else: + errors[task_name] = "persisted complete archive๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค" + continue + if task_name in active_or_running: + continue + baseline = set(str(path) for path in record.get("archive_baseline", [])) + candidates = [ + path + for path in matching_archive_directories_by_name(workspace, task_name) + if str(path.resolve()) not in baseline + ] + if len(candidates) == 1: + archive = str(candidates[0].resolve()) + record.update(status="complete", archive=archive) + completed[task_name] = archive + changed = self.release_write_claim(task_name, persist=False) or changed + changed = True + elif not candidates: + errors[task_name] = ( + "๊ด€์ฐฐ๋œ task๊ฐ€ active์™€ ์ƒˆ complete.log archive ๋ชจ๋‘์—์„œ ์‚ฌ๋ผ์กŒ๋‹ค" + ) + else: + errors[task_name] = ( + "์ƒˆ complete.log archive๊ฐ€ ์—ฌ๋Ÿฌ ๊ฐœ๋ผ ์™„๋ฃŒ ๊ฒฝ๋กœ๋ฅผ ํ™•์ •ํ•  ์ˆ˜ ์—†๋‹ค: " + + ",".join(str(path) for path in candidates) + ) + if changed: + self.save() + for task_name in completed: + if task_name not in active_or_running: + cleanup_completed_task_attempt_logs(self.runs, task_name) + return completed, errors + + def orchestration_tasks(self, scope: str) -> set[str]: + current = self.data.get("orchestrations", {}).get(scope, {}) + return set(current.get("tasks", {})) + + def mark_orchestration_complete(self, scope: str) -> None: + current = self.data.setdefault("orchestrations", {}).setdefault( + scope, {"status": "running", "tasks": {}} + ) + current["status"] = "complete" + self.save() + + +def orchestration_live_agent_processes( + store: StateStore, + scope: str, +) -> dict[str, str]: + """Return observed tasks with live or conservatively active evidence.""" + task_states = store.data.get("tasks", {}) + live: dict[str, str] = {} + for task_name in store.orchestration_tasks(scope): + state = task_states.get(task_name) + if not isinstance(state, dict): + continue + is_live, detail = external_active_is_live( + state, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if is_live: + live[task_name] = detail + return live + + +def workspace_live_agent_processes( + store: StateStore, +) -> dict[str, str]: + """Return observed tasks across the entire physical workspace with live or conservatively active evidence.""" + task_states = store.data.get("tasks", {}) + live: dict[str, str] = {} + if isinstance(task_states, dict): + for task_name, state in task_states.items(): + if not isinstance(state, dict): + continue + is_live, detail = external_active_is_live( + state, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if is_live: + live[task_name] = detail + return live + + +def parse_route(plan: Path | None) -> tuple[str | None, int | None]: + if plan is None: + return None, None + match = PLAN_RE.match(plan.name) + if not match: + return None, None + return match.group(1), int(match.group(2)) + + +def parse_task_name(task_root: Path, directory: Path) -> str: + return directory.relative_to(task_root).as_posix() + + +def inspect_write_set( + plan: Path | None, + workspace: Path, +) -> tuple[set[str], list[str]]: + if plan is None: + return set(), ["PLAN ๊ฒฝ๋กœ๊ฐ€ ์—†๋‹ค"] + if not plan.is_file(): + return set(), [f"PLAN ํŒŒ์ผ์ด ์—†๋‹ค: {plan}"] + workspace = workspace.resolve() + try: + text = plan.read_text(encoding="utf-8", errors="replace") + except OSError as exc: + return set(), [f"PLAN ํŒŒ์ผ์„ ์ฝ์„ ์ˆ˜ ์—†๋‹ค: {plan}: {exc}"] + matches = [] + for heading in MODIFIED_FILES_HEADINGS: + pattern = rf"^##\s*{re.escape(heading)}[ \t]*$([\s\S]*?)(?=^##\s|\Z)" + for m in re.finditer(pattern, text, re.MULTILINE): + matches.append(m) + if not matches: + return set(), ["Modified Files Summary ์„น์…˜์ด ์—†๋‹ค"] + if len(matches) != 1: + return set(), [ + f"Modified Files Summary ์„น์…˜์€ ์ •ํ™•ํžˆ 1๊ฐœ์—ฌ์•ผ ํ•œ๋‹ค: count={len(matches)}" + ] + match = matches[0] + result: set[str] = set() + diagnostics: list[str] = [] + for line in match.group(1).splitlines(): + if not line.lstrip().startswith("|"): + continue + cells = [cell.strip() for cell in line.strip().strip("|").split("|")] + if not cells: + continue + if all(set(cell) <= {":", "-"} for cell in cells): + continue + if cells[0].casefold() in MODIFIED_FILES_HEADER_CELLS: + continue + claims = re.findall(r"`([^`]+)`", cells[0]) + if not claims: + diagnostics.append( + f"์ •ํ™•ํ•œ backtick workspace ํŒŒ์ผ ๊ฒฝ๋กœ๊ฐ€ ์—†๋Š” claim ํ–‰: {cells[0]}" + ) + continue + for value in claims: + normalized = re.sub(r":\d+(?::\d+)?$", "", value.strip()) + if not normalized: + diagnostics.append("๋นˆ ๊ฒฝ๋กœ claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค") + continue + if PLACEHOLDER_PATH_RE.search(normalized): + diagnostics.append( + f"placeholder ๋˜๋Š” malformed path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}" + ) + continue + if normalized.startswith(("http://", "https://")): + diagnostics.append(f"URL claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}") + continue + if "\\" in normalized: + diagnostics.append( + f"malformed path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}" + ) + continue + if any(character in normalized for character in "*?[]"): + diagnostics.append( + f"glob ๋˜๋Š” broad path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}" + ) + continue + if normalized.endswith(("/", "\\")): + diagnostics.append( + f"๋””๋ ‰ํ„ฐ๋ฆฌ claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}" + ) + continue + candidate = Path(normalized) + try: + resolved = ( + candidate.resolve() + if candidate.is_absolute() + else (workspace / candidate).resolve() + ) + except (OSError, RuntimeError) as exc: + diagnostics.append( + f"๊ฒฝ๋กœ๋ฅผ canonicalizeํ•  ์ˆ˜ ์—†๋‹ค: {normalized}: {exc}" + ) + continue + try: + resolved.relative_to(workspace) + except ValueError: + diagnostics.append( + f"workspace ๋ฐ– claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}" + ) + continue + if resolved == workspace: + diagnostics.append("workspace root claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค") + continue + if resolved.is_dir(): + diagnostics.append( + f"๋””๋ ‰ํ„ฐ๋ฆฌ claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: {normalized}" + ) + continue + result.add(str(resolved)) + if not result: + diagnostics.append("์ •ํ™•ํ•œ workspace ํŒŒ์ผ claim์ด ํ•˜๋‚˜ ์ด์ƒ ํ•„์š”ํ•˜๋‹ค") + return result, diagnostics + + +def extract_write_set(plan: Path | None, workspace: Path) -> tuple[set[str], bool]: + write_set, diagnostics = inspect_write_set(plan, workspace) + if diagnostics: + return set(), False + return write_set, True + + +def latest_verdict_log(directory: Path) -> Path | None: + candidates: list[tuple[int, Path]] = [] + for path in directory.glob("code_review_*.log"): + match = REVIEW_LOG_RE.fullmatch(path.name) + if match is None or not path.is_file() or read_verdict(path) is None: + continue + candidates.append((int(match.group(3)), path)) + if not candidates: + return None + return max( + candidates, + key=lambda candidate: (candidate[0], candidate[1].name), + )[1] + + +def matching_plan_log(directory: Path, review_log: Path | None) -> Path | None: + if review_log is None: + return None + review_identity = plan_identity(review_log) + matches = [ + path + for path in directory.glob("plan_*.log") + if PLAN_LOG_RE.fullmatch(path.name) is not None + and path.is_file() + and plan_identity(path) == review_identity + ] + return max(matches, key=lambda path: path.stat().st_mtime_ns) if matches else None + + +def read_task_directory(workspace: Path, directory: Path) -> Task | None: + """Read one already-known task directory without scanning the task group.""" + task_root = workspace / "agent-task" + if not directory.is_dir(): + return None + plans = sorted(p for p in directory.iterdir() if p.is_file() and PLAN_RE.match(p.name)) + reviews = sorted(p for p in directory.iterdir() if p.is_file() and REVIEW_RE.match(p.name)) + users = sorted(directory.glob("USER_REVIEW.md")) + complete = directory / "complete.log" + recovery_log = latest_verdict_log(directory) + if not plans and not reviews and not users and not complete.exists() and recovery_log is None: + return None + name = parse_task_name(task_root, directory) + errors: list[str] = [] + if len(plans) > 1: + errors.append(f"active PLAN์ด {len(plans)}๊ฐœ๋‹ค") + if len(reviews) > 1: + errors.append(f"active CODE_REVIEW๊ฐ€ {len(reviews)}๊ฐœ๋‹ค") + if len(users) > 1: + errors.append(f"USER_REVIEW๊ฐ€ {len(users)}๊ฐœ๋‹ค") + if users and (plans or reviews): + errors.append("USER_REVIEW stop state์™€ active PLAN/CODE_REVIEW๊ฐ€ ๊ณต์กดํ•œ๋‹ค") + plan = plans[0] if len(plans) == 1 else None + review = reviews[0] if len(reviews) == 1 else None + recovery = complete.exists() or recovery_log is not None + if bool(plan) != bool(review) and not recovery: + errors.append("active PLAN/CODE_REVIEW pair๊ฐ€ ๋ถˆ์™„์ „ํ•˜๋‹ค") + relative = directory.relative_to(task_root) + subtask = relative.parts[1] if len(relative.parts) == 2 else None + index = 0 + deps: tuple[str, ...] = () + if subtask: + match = SUBTASK_RE.match(subtask) + if match: + index = int(match.group("index")) + deps = tuple((match.group("deps") or "").split(",")) if match.group("deps") else () + else: + errors.append(f"split subtask ์ด๋ฆ„์ด ๊ณ„์•ฝ๊ณผ ๋‹ค๋ฅด๋‹ค: {subtask}") + lane, grade = parse_route(plan) + recovery_plan = matching_plan_log(directory, recovery_log) + write_set_source = plan or recovery_plan + write_set: set[str] = set() + write_set_known = False + if recovery_log is not None and recovery_plan is None: + errors.append( + "PLAN Modified Files Summary๋ฅผ ๋ณต๊ตฌํ•  matching PLAN log๊ฐ€ ์—†๋‹ค" + ) + elif write_set_source is not None: + write_set, write_set_diagnostics = inspect_write_set( + write_set_source, + workspace, + ) + write_set_known = bool(write_set) and not write_set_diagnostics + errors.extend( + f"PLAN Modified Files Summary๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: {diagnostic}" + for diagnostic in write_set_diagnostics + ) + if plan is not None: + metadata = PLAN_IDENTITY_RE.search( + plan.read_text(encoding="utf-8", errors="replace")[:1024] + ) + if metadata is None: + errors.append("PLAN task/plan/tag metadata๋ฅผ ํŒ๋ณ„ํ•  ์ˆ˜ ์—†๋‹ค") + elif metadata.group("task") != name: + errors.append( + f"PLAN task metadata๊ฐ€ ๋””๋ ‰ํ„ฐ๋ฆฌ์™€ ๋‹ค๋ฅด๋‹ค: {metadata.group('task')}" + ) + return Task( + name=name, + directory=directory, + plan=plan, + review=review, + user_review=users[0] if len(users) == 1 else None, + recovery=recovery, + errors=errors, + index=index, + deps=deps, + write_set=write_set, + write_set_known=write_set_known, + plan_hash=( + plan_identity(plan) + if plan + else sha256_file( + recovery_log + or (users[0] if len(users) == 1 else complete) + ) + ), + lane=lane, + grade=grade, + ) + + + + +@dataclass +class StageFailureBudget: + """Persistent failure counter shared by one work unit and stage.""" + + store: StateStore + task: Task + work_unit_id: str + stage: str + + + @classmethod + def from_decision(cls, store: StateStore, task: Task, decision: dict[str, Any]) -> "StageFailureBudget": + work_unit_id = decision.get("work_unit_id") + stage = decision.get("stage") + if not isinstance(work_unit_id, str) or not work_unit_id or not isinstance(stage, str) or not stage: + raise ExecutionDecisionError("stage failure budget identity๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + return cls(store, task, work_unit_id, stage) + + + @property + def key(self) -> str: + return f"{self.work_unit_id}|{self.stage}" + + def _budgets(self) -> dict[str, Any]: + state = self.store.task_state(self.task) + budgets = state.get("stage_failure_budgets", {}) + if not isinstance(budgets, dict): + raise ExecutionDecisionError("persisted stage failure budgets schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + return dict(budgets) + + def count(self) -> int: + entry = self._budgets().get(self.key, {}) + if not isinstance(entry, dict): + raise ExecutionDecisionError("persisted stage failure budget entry๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + return int(entry.get("count", 0)) + + def record_failure(self, *, target: dict[str, Any], transition: str) -> int: + budgets = self._budgets() + entry = dict(budgets.get(self.key, {})) + count = int(entry.get("count", 0)) + 1 + entry.update( + work_unit_id=self.work_unit_id, stage=self.stage, count=count, + last_target={"adapter": target.get("adapter"), "target": target.get("target")}, + last_transition=transition, + ) + budgets[self.key] = entry + self.store.update_task(self.task, stage_failure_budgets=budgets) + return count + + def reset_on_success(self) -> None: + budgets = self._budgets() + budgets.pop(self.key, None) + self.store.update_task(self.task, stage_failure_budgets=budgets) + + +def scan_tasks( + workspace: Path, + task_group: str | None, + *, + exclude_names: set[str] | None = None, +) -> list[Task]: + task_root = workspace / "agent-task" + if not task_root.is_dir(): + raise DispatcherTerminalStateError( + f"agent-task ๋””๋ ‰ํ„ฐ๋ฆฌ๊ฐ€ ์—†๋‹ค: {task_root}" + ) + directories: list[Path] = [] + try: + groups = [task_root / task_group] if task_group else sorted( + p for p in task_root.iterdir() if p.is_dir() and p.name != "archive" + ) + except FileNotFoundError: + return [] + for group in groups: + if not group.is_dir(): + continue + directories.append(group) + try: + directories.extend(sorted(p for p in group.iterdir() if p.is_dir())) + except FileNotFoundError: + continue + tasks = [ + task + for directory in directories + if ( + exclude_names is None + or parse_task_name(task_root, directory) not in exclude_names + ) + if (task := read_task_directory(workspace, directory)) is not None + ] + return sorted(tasks, key=lambda task: (task.index, task.name)) + + +def dependency_candidates(workspace: Path, task: Task, predecessor: str) -> list[Path]: + parts = task.name.split("/") + if len(parts) != 2: + return [] + group = parts[0] + task_root = workspace / "agent-task" + found: list[Path] = [] + active_group = task_root / group + for pattern in (f"{predecessor}_*/complete.log", f"{predecessor}+*/complete.log"): + found.extend(active_group.glob(pattern)) + archive = task_root / "archive" + if archive.is_dir(): + try: + years = list(archive.iterdir()) + except FileNotFoundError: + years = [] + for year in years: + if not year.is_dir(): + continue + try: + months = list(year.iterdir()) + except FileNotFoundError: + continue + for month in months: + archived_group = month / group + if not archived_group.is_dir(): + continue + for pattern in (f"{predecessor}_*/complete.log", f"{predecessor}+*/complete.log"): + found.extend(archived_group.glob(pattern)) + return sorted(set(path.resolve() for path in found)) + + +def dependency_state(workspace: Path, task: Task) -> tuple[bool, str]: + missing: list[str] = [] + ambiguous: list[str] = [] + for predecessor in task.deps: + candidates = dependency_candidates(workspace, task, predecessor) + if not candidates: + missing.append(predecessor) + elif len(candidates) > 1: + ambiguous.append(f"{predecessor}={','.join(str(p) for p in candidates)}") + if ambiguous: + return False, "dependency ambiguity: " + "; ".join(ambiguous) + if missing: + return False, "predecessor complete.log ๋Œ€๊ธฐ: " + ",".join(missing) + return True, "ready" + + +def live_predecessors( + task: Task, + active_task_names: set[str], +) -> list[str]: + parts = task.name.split("/") + if len(parts) != 2 or not task.deps: + return [] + group = parts[0] + live: list[str] = [] + for predecessor in task.deps: + prefix = re.compile(rf"^{re.escape(predecessor)}(?:[+_])") + if any( + name.startswith(f"{group}/") + and prefix.match(name.split("/", 1)[1]) + for name in active_task_names + ): + live.append(predecessor) + return live + + +def _selector_module(): + if "agent_task_execution_target_selector" in sys.modules: + return sys.modules["agent_task_execution_target_selector"] + path = Path(__file__).resolve().parent / "select_execution_target.py" + spec = importlib.util.spec_from_file_location("agent_task_execution_target_selector", path) + if spec is None or spec.loader is None: + raise ExecutionDecisionError(f"selector load ์‹คํŒจ: {path}") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def _decision_file(task: Task, stage: str) -> Path: + path = task.plan if stage == "worker" else task.review + if path is None or not path.is_file(): + raise ExecutionDecisionError(f"{stage} selector ์ž…๋ ฅ ํŒŒ์ผ์ด ์—†๋‹ค") + return path + + +def agent_spec_from_decision(decision: dict[str, Any]) -> AgentSpec: + selected = decision.get("selected") + if not isinstance(selected, dict): + raise ExecutionDecisionError("selector selected๊ฐ€ object๊ฐ€ ์•„๋‹ˆ๋‹ค") + adapter, target = selected.get("adapter"), selected.get("target") + local_pi = selected.get("selfcheck_required") + execution_class = selected.get("execution_class") + if (not isinstance(adapter, str) or not isinstance(target, str) or not target + or not isinstance(local_pi, bool) + or execution_class not in {"local_model", "cloud_model"}): + raise ExecutionDecisionError("selector selected schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + try: + selector = _selector_module() + selector._validate_prior_decision(decision) + decision_info = decision["decision"] + evaluated_at = selector.datetime.fromisoformat(decision_info["evaluated_at"]) + policy_targets = selector.policy.select_policy( + stage=decision["stage"], lane=decision["lane"], grade=decision["grade"], + evaluated_at=evaluated_at, + ).candidates + selector._validate_prior_candidate_identity( + decision, + stage=decision["stage"], + lane=decision["lane"], + grade=decision["grade"], + ) + canonical = selector.policy.canonical_target(adapter, target) + except Exception as exc: + raise ExecutionDecisionError(f"selector policy validation ์‹คํŒจ: {exc}") from exc + if canonical is None or ( + canonical.execution_class != execution_class + or canonical.selfcheck_required != local_pi + ): + raise ExecutionDecisionError("selector selected๊ฐ€ canonical policy target์ด ์•„๋‹ˆ๋‹ค") + initial_keys = {(item.adapter, item.target) for item in policy_targets} + if (adapter, target) not in initial_keys: + promotion_path = decision.get("promotion_path") + if not isinstance(promotion_path, list) or len(promotion_path) < 2: + raise ExecutionDecisionError("selector promotion path๊ฐ€ ์—†๋‹ค") + resolved_path = [] + for index, entry in enumerate(promotion_path): + if not isinstance(entry, dict): + raise ExecutionDecisionError( + f"selector promotion path[{index}]๊ฐ€ object๊ฐ€ ์•„๋‹ˆ๋‹ค" + ) + resolved = selector.policy.canonical_target( + entry.get("adapter"), entry.get("target") + ) + if resolved is None: + raise ExecutionDecisionError( + f"selector promotion path[{index}] target์ด canonical์ด ์•„๋‹ˆ๋‹ค" + ) + resolved_path.append(resolved) + if (resolved_path[0].adapter, resolved_path[0].target) not in initial_keys: + raise ExecutionDecisionError("selector promotion path ์‹œ์ž‘ target์ด ์ž˜๋ชป๋๋‹ค") + if any( + selector.policy.promotion_target(previous) != current + for previous, current in zip(resolved_path, resolved_path[1:]) + ): + raise ExecutionDecisionError("selector promotion path ์ˆœ์„œ๊ฐ€ ์ž˜๋ชป๋๋‹ค") + if resolved_path[-1] != canonical: + raise ExecutionDecisionError("selector promotion path tail์ด selected์™€ ๋‹ค๋ฅด๋‹ค") + if adapter == "pi": + if not target.startswith("iop/") or not local_pi: + raise ExecutionDecisionError("Pi selector target/schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + return AgentSpec("pi", target.removeprefix("iop/"), f"pi/{target}", local_pi=True) + if adapter not in {"agy", "claude", "codex"} or local_pi: + raise ExecutionDecisionError(f"selector adapter/schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: {adapter!r}") + reasoning_effort = "high" if canonical == selector.policy.CODEX_TERRA_HIGH else None + suffix = " high" if reasoning_effort == "high" else ( + " xhigh" if adapter in {"claude", "codex"} else "" + ) + display = f"{adapter}/{target}{suffix}" + if reasoning_effort is not None: + return AgentSpec( + adapter, + target, + display, + reasoning_effort=reasoning_effort, + ) + return AgentSpec(adapter, target, display) + + +def _spec_from_completing_decision(decision: dict[str, Any]) -> AgentSpec: + """Lightweight AgentSpec extraction from a persisted completing decision. + + Unlike `agent_spec_from_decision`, this does not re-validate against the + selector policy. The completing decision is already authoritative evidence + of the target that succeeded, so re-running policy is unnecessary and would + defeat the purpose of pinning the selfcheck target. + """ + selected = decision.get("selected") + if not isinstance(selected, dict): + raise ExecutionDecisionError( + "completing decision selected schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค" + ) + adapter = selected.get("adapter") + target = selected.get("target") + execution_class = selected.get("execution_class") + selfcheck_required = selected.get("selfcheck_required") + if not all(isinstance(value, str) and value for value in (adapter, target, execution_class)): + raise ExecutionDecisionError( + "completing decision selected์˜ adapter/target/execution_class๋Š” ๋นˆ ๋ฌธ์ž์—ด์ด ์•„๋‹Œ string์ด์–ด์•ผ ํ•œ๋‹ค" + ) + if execution_class not in {"local_model", "cloud_model"}: + raise ExecutionDecisionError( + f"completing decision execution_class์ด ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: {execution_class}" + ) + if not isinstance(selfcheck_required, bool): + raise ExecutionDecisionError( + "completing decision selected.selfcheck_required must be a boolean" + ) + if adapter == "pi": + if not target.startswith("iop/"): + raise ExecutionDecisionError( + f"Pi completing decision target์ด iop/ prefix๊ฐ€ ์•„๋‹ˆ๋‹ค: {target}" + ) + if execution_class != "local_model": + raise ExecutionDecisionError( + f"Pi completing decision execution_class์ด local_model์ด ์•„๋‹ˆ๋‹ค: {execution_class}" + ) + if not selfcheck_required: + raise ExecutionDecisionError( + "Pi completing decision selfcheck_required๊ฐ€ False์ด๋‹ค" + ) + model = target.removeprefix("iop/") + display = f"pi/{target}" + return AgentSpec(adapter, model, display, local_pi=True) + if adapter not in {"agy", "claude", "codex"}: + raise ExecutionDecisionError( + f"completing decision adapter๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: {adapter!r}" + ) + if execution_class != "cloud_model": + raise ExecutionDecisionError( + f"cloud completing decision execution_class์ด cloud_model์ด ์•„๋‹ˆ๋‹ค: {adapter}/{execution_class}" + ) + if selfcheck_required: + raise ExecutionDecisionError( + f"cloud completing decision selfcheck_required๊ฐ€ True์ด๋‹ค: {adapter}/{target}" + ) + display = f"{adapter}/{target}" + return AgentSpec(adapter, target, display, local_pi=False) + + +def select_execution_decision( + task: Task, *, stage: str, prior_decision: dict[str, Any] | None = None, + quota_snapshot: dict[str, Any] | None = None, + evaluated_at: datetime | None = None, + transition: str | None = None, + failure_class: str | None = None, +) -> dict[str, Any]: + try: + selector = _selector_module() + except Exception as exc: + code = getattr(exc, "code", exc.__class__.__name__) + raise ExecutionDecisionError( + f"{stage} selector load ์‹คํŒจ [{code}]: {exc}" + ) from exc + try: + if transition is None: + if prior_decision is not None and stage == "worker" and task.plan and task.plan.is_file(): + current_id = work_unit_id_from_file(task.plan) + prior_id = prior_decision.get("work_unit_id") if isinstance(prior_decision, dict) else None + if current_id and isinstance(prior_id, str) and prior_id and prior_id != current_id: + prior_decision = None + transition = "resume" if prior_decision is not None else "initial" + return selector.select_execution_target( + _decision_file(task, stage), stage=stage, + evaluated_at=evaluated_at or datetime.now(KST), + transition=transition, + prior_decision=prior_decision, + quota_snapshot=quota_snapshot, + failure_class=failure_class, + ) + except (OSError, ValueError, selector.SelectorInputError) as exc: + code = getattr(exc, "code", exc.__class__.__name__) + raise ExecutionDecisionError( + f"{stage} selector decision ์‹คํŒจ [{code}]: {exc}" + ) from exc + + +def work_unit_id_from_file(path: Path) -> str | None: + if not path.is_file(): + return None + try: + head = path.read_text(encoding="utf-8", errors="replace")[:1024] + match = PLAN_IDENTITY_RE.search(head) + if match: + return f"{match.group('task')}::plan-{match.group('plan')}::tag-{match.group('tag')}" + except Exception: + pass + return None + + +def official_review_plan_source(task: Task) -> Path: + """Resolve the authoritative PLAN generation for an official review.""" + if task.plan is not None or task.review is not None: + if task.plan is None or task.review is None: + raise ExecutionDecisionError( + "official review active PLAN/CODE_REVIEW pair๊ฐ€ ๋ถˆ์™„์ „ํ•˜๋‹ค" + ) + return task.plan + recovery_log = latest_verdict_log(task.directory) + recovery_plan = matching_plan_log(task.directory, recovery_log) + if recovery_log is None or recovery_plan is None: + raise ExecutionDecisionError( + "official review recovery์˜ matching archived PLAN identity๋ฅผ ๋ณต๊ตฌํ•  ์ˆ˜ ์—†๋‹ค" + ) + return recovery_plan + + +def official_review_source_identity(task: Task) -> tuple[str, int, str]: + source = official_review_plan_source(task) + route_match = PLAN_RE.match(source.name) or PLAN_LOG_RE.match(source.name) + if route_match is None: + raise ExecutionDecisionError( + f"official review PLAN route๋ฅผ ๋ณต๊ตฌํ•  ์ˆ˜ ์—†๋‹ค: {source.name}" + ) + try: + head = source.read_text(encoding="utf-8", errors="replace")[:1024] + except OSError as exc: + raise ExecutionDecisionError( + f"official review PLAN source๋ฅผ ์ฝ์„ ์ˆ˜ ์—†๋‹ค: {source}" + ) from exc + metadata = PLAN_IDENTITY_RE.search(head) + if metadata is None or metadata.group("task") != task.name: + raise ExecutionDecisionError( + f"official review PLAN work-unit identity๋ฅผ ๋ณต๊ตฌํ•  ์ˆ˜ ์—†๋‹ค: {source}" + ) + work_unit_id = ( + f"{metadata.group('task')}::plan-{metadata.group('plan')}::" + f"tag-{metadata.group('tag')}" + ) + return route_match.group(1), int(route_match.group(2)), work_unit_id + + +def synthesized_official_review_decision( + task: Task, *, evaluated_at: datetime | None = None +) -> dict[str, Any]: + lane, grade, work_unit_id = official_review_source_identity(task) + evaluated = evaluated_at or datetime.now(KST) + if evaluated.tzinfo is None or evaluated.utcoffset() is None: + raise ExecutionDecisionError( + "official review evaluated_at์ด timezone-aware๊ฐ€ ์•„๋‹ˆ๋‹ค" + ) + recovery_from_archive = task.plan is None and task.review is None + selector = _selector_module() + policy_decision = selector.policy.select_policy( + stage="review", lane=lane, grade=grade, evaluated_at=evaluated + ) + selected_target = policy_decision.candidates[0] + target_ref = { + "adapter": selected_target.adapter, + "target": selected_target.target, + } + candidate = { + "candidate_rank": 1, + "adapter": selected_target.adapter, + "target": selected_target.target, + "execution_class": selected_target.execution_class, + "selfcheck_required": selected_target.selfcheck_required, + "quota_mode": "bounded", + "quota_status": "unknown", + "eligibility": "eligible", + "rejection_reason": None, + } + return { + "schema_version": selector.SCHEMA_VERSION, + "work_unit_id": work_unit_id, + "stage": "review", + "lane": lane, + "grade": grade, + "selected": { + "adapter": selected_target.adapter, + "target": selected_target.target, + "execution_class": selected_target.execution_class, + "selfcheck_required": selected_target.selfcheck_required, + }, + "candidates": [candidate], + "decision": { + "rule_id": policy_decision.rule_id, + "policy_priority": policy_decision.policy_priority, + "reason_codes": list(policy_decision.reason_codes), + "evaluated_at": evaluated.astimezone(KST).isoformat(), + "timezone": selector.TIMEZONE_NAME, + "time_window": policy_decision.time_window, + "pinned": recovery_from_archive, + }, + "quota": { + "snapshot_id": None, + "mode": "bounded", + "status": "unknown", + "source": "official_review_fixed_policy", + "checked_at": None, + "targets": [], + }, + "transition": { + "previous_target": dict(target_ref) if recovery_from_archive else None, + "next_target": dict(target_ref) if recovery_from_archive else None, + "trigger": "resume" if recovery_from_archive else "initial", + "context_transfer": "none", + }, + } + + +def read_or_preview_stage_decision( + task: Task, + state: dict[str, Any], + *, + stage: str, + dry_run: bool = False, + evaluated_at: datetime | None = None, + quota_snapshot: dict[str, Any] | None = None, +) -> dict[str, Any]: + decisions = state.get("execution_decisions", {}) if isinstance(state, dict) else {} + prior = decisions.get(stage) if isinstance(decisions, dict) else None + + if stage == "review": + lane, grade, work_unit_id = official_review_source_identity(task) + if ( + isinstance(prior, dict) + and isinstance(prior.get("decision"), dict) + and isinstance(prior.get("quota"), dict) + ): + if ( + prior.get("work_unit_id") != work_unit_id + or prior.get("stage") != "review" + or prior.get("lane") != lane + or prior.get("grade") != grade + ): + raise ExecutionDecisionError( + "persisted official review decision์ด recovery source identity/route์™€ ๋‹ค๋ฅด๋‹ค" + ) + agent_spec_from_decision(prior) + return prior + return synthesized_official_review_decision( + task, evaluated_at=evaluated_at + ) + + if isinstance(prior, dict) and task.plan and task.plan.is_file(): + work_unit_id = work_unit_id_from_file(task.plan) + if work_unit_id and prior.get("work_unit_id") == work_unit_id: + return prior + + if quota_snapshot is None: + quota_snapshot = state.get("quota_snapshot") if isinstance(state, dict) else None + return select_execution_decision( + task, + stage=stage, + prior_decision=prior, + quota_snapshot=quota_snapshot, + evaluated_at=evaluated_at, + ) + + +def selector_evidence_lines(decision: dict[str, Any] | None) -> list[str]: + if not isinstance(decision, dict): + return [] + selected = decision.get("selected", {}) + if not isinstance(selected, dict): + return [] + work_unit = decision.get("work_unit_id", "none") + decision_info = decision.get("decision", {}) + if not isinstance(decision_info, dict): + decision_info = {} + rule_id = decision_info.get("rule_id", decision.get("rule_id", "none")) + priority = decision_info.get( + "policy_priority", decision.get("priority", "none") + ) + transition = decision.get("transition", {}) + trigger = transition.get("trigger", "none") if isinstance(transition, dict) else "none" + quota = decision.get("quota", decision.get("quota_snapshot", {})) + quota_status = quota.get("status", "none") if isinstance(quota, dict) else "none" + + candidates = decision.get("candidates", []) + cand_strs = [] + if isinstance(candidates, list): + for c in candidates: + if isinstance(c, dict): + rank = c.get("candidate_rank", "?") + adapter = c.get("adapter", "?") + target = c.get("target", "?") + elig = c.get("eligibility", "?") + cand_strs.append(f"#{rank}:{adapter}/{target}({elig})") + + reasons = decision_info.get( + "reason_codes", selected.get("reason_codes", []) + ) + reason_str = ",".join(reasons) if isinstance(reasons, list) else str(reasons) + + lines = [ + f"work_unit_id={work_unit}", + f"rule_id={rule_id}", + f"priority={priority}", + f"transition={trigger}", + f"quota_status={quota_status}", + ] + if cand_strs: + lines.append(f"candidates={';'.join(cand_strs)}") + if reason_str: + lines.append(f"reason_codes={reason_str}") + return lines + + +def selector_runtime_evidence(decision: dict[str, Any]) -> dict[str, Any]: + """Return canonical selector fields persisted in runtime audit records.""" + return { + "work_unit_id": decision.get("work_unit_id"), + "candidates": decision.get("candidates"), + "selected": decision.get("selected"), + "decision": decision.get("decision"), + "quota": decision.get("quota"), + "transition": decision.get("transition"), + } + + +def commit_execution_decision( + store: StateStore, task: Task, stage: str, decision: dict[str, Any], + quota_snapshot: dict[str, Any] | None = None, +) -> None: + state = store.task_state(task) + decisions, history = state.get("execution_decisions", {}), state.get("route_transition_history", []) + if not isinstance(decisions, dict) or not isinstance(history, list): + raise ExecutionDecisionError("persisted selector state schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + decisions = dict(decisions) + decisions[stage] = decision + + stage_budget_count = 0 + try: + stage_budget_count = StageFailureBudget.from_decision(store, task, decision).count() + except Exception: + pass + + selected = decision.get("selected", {}) + history_entry = { + "stage": stage, + "transition": decision.get("transition", {}).get("trigger") if isinstance(decision.get("transition"), dict) else None, + "work_unit_id": decision.get("work_unit_id"), + "candidates": decision.get("candidates"), + "selected": selected, + "decision": decision.get("decision"), + "reason_codes": decision.get("decision", {}).get("reason_codes", []) + if isinstance(decision.get("decision"), dict) + else [], + "quota": decision.get("quota"), + "stage_budget": stage_budget_count, + } + history = [*history, history_entry] + # Preserve retry handoff state whenever a pending retry is in flight. + # The retry handoff (stable handoff_id, pending flag, context) must survive + # the decision commit so that the subsequent production invoke() can read + # it, embed the handoff_id in the new locator record, and atomically + # consume the pending handoff via commit_retry_handoff_locator(). + # Clearing it here would force invoke() to fall back to a generic + # active-locator update and lose the crash-safe handoff identity. + # invoke() handles consumption regardless of failover or resume transition. + is_retry_in_flight = bool(state.get("retry_quota_refresh_pending")) + update_kwargs = { + "execution_decisions": decisions, + "route_transition_history": history, + "blocked": None, + "blocker_evidence": None, + } + if not is_retry_in_flight: + update_kwargs["retry_quota_refresh_pending"] = False + update_kwargs["retry_quota_refresh_context"] = None + if quota_snapshot is not None: + update_kwargs["quota_snapshot"] = quota_snapshot + store.update_task(task, **update_kwargs) + + +def persisted_execution_decision( + store: StateStore, task: Task, *, stage: str, + transition: str | None = None, + failure_class: str | None = None, + evaluated_at: datetime | None = None, + quota_snapshot: dict[str, Any] | None = None, +) -> tuple[dict[str, Any], AgentSpec]: + state = store.task_state(task) + decisions = state.get("execution_decisions", {}) + if not isinstance(decisions, dict): + raise ExecutionDecisionError("persisted selector state schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + is_retry = retry_quota_refresh_pending(state) and stage == "worker" + if quota_snapshot is None and not is_retry: + quota_snapshot = state.get("quota_snapshot") + if quota_snapshot is not None and not isinstance(quota_snapshot, dict): + raise ExecutionDecisionError("persisted quota snapshot schema๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค") + prior_decision = decisions.get(stage) + + retry_ctx = state.get("retry_quota_refresh_context") if isinstance(state.get("retry_quota_refresh_context"), dict) else {} + + if stage == "review": + decision = read_or_preview_stage_decision( + task, state, stage=stage, evaluated_at=evaluated_at, quota_snapshot=quota_snapshot + ) + else: + if transition is None: + if is_retry: + transition = "failover" + failure_class = failure_class or retry_ctx.get("failure_class") or "provider-quota" + elif prior_decision is not None and stage == "worker" and task.plan and task.plan.is_file(): + current_id = work_unit_id_from_file(task.plan) + prior_id = prior_decision.get("work_unit_id") if isinstance(prior_decision, dict) else None + if current_id and isinstance(prior_id, str) and prior_id and prior_id != current_id: + prior_decision = None + transition = "resume" if prior_decision is not None else "initial" + else: + transition = "resume" if prior_decision is not None else "initial" + + try: + decision = select_execution_decision( + task, stage=stage, prior_decision=prior_decision, + quota_snapshot=quota_snapshot, + transition=transition, + failure_class=failure_class, + evaluated_at=evaluated_at, + ) + except ExecutionDecisionError as exc: + # No persisted unused quota target is an explicit resume case. A + # failed qualified failover with a fresh snapshot must not consume + # the retry intent before its decision can commit successfully. + if is_retry and transition == "failover" and quota_snapshot is None: + decision = select_execution_decision( + task, stage=stage, prior_decision=prior_decision, + quota_snapshot=quota_snapshot, + transition="resume", + evaluated_at=evaluated_at, + ) + else: + raise + + spec = agent_spec_from_decision(decision) + commit_execution_decision(store, task, stage, decision, quota_snapshot=quota_snapshot) + return decision, spec + + +def has_persisted_worker_decision(state: dict[str, Any], task: Task | None = None) -> bool: + decisions = state.get("execution_decisions", {}) + if not isinstance(decisions, dict): + return False + prior = decisions.get("worker") + if prior is None: + return False + if task is not None and task.plan and task.plan.is_file(): + current_id = work_unit_id_from_file(task.plan) + if current_id and prior.get("work_unit_id") != current_id: + return False + return True + + +def retry_quota_refresh_pending(state: dict[str, Any]) -> bool: + return bool(state.get("retry_quota_refresh_pending")) + + +def derive_work_unit_quota_evidence( + decision: dict[str, Any] | None, + status: str = "exhausted", + reason: str = "confirmed_runtime_provider_quota", +) -> dict[str, Any]: + selector = _selector_module() + func = getattr(selector, "derive_work_unit_quota_evidence", None) + if func is not None: + return func(decision, status=status, reason=reason) + return { + "schema_version": "1.0", + "snapshot_id": None, + "source": "derived_work_unit_quota", + "checked_at": datetime.now(KST).isoformat(), + "targets": [], + "required_caps": [], + "reason_codes": [reason], + } + + +def build_admission_batch_snapshot( + store: StateStore, + ready_items: list[tuple[Task, str]], + admission_time: datetime, + quota_probe_command: str = "iop-node quota-probe", +) -> dict[str, Any] | None: + selector = _selector_module() + policy_mod = selector.policy + + unique_keys = [] + seen_keys = set() + + for task, stage in ready_items: + if stage != "worker": + continue + state = store.peek_task_state(task) + if has_persisted_worker_decision(state, task) and not retry_quota_refresh_pending(state): + continue + + is_retry = retry_quota_refresh_pending(state) + if is_retry: + decisions = state.get("execution_decisions", {}) + prior = decisions.get("worker") if isinstance(decisions, dict) else None + candidates = prior.get("candidates", []) if isinstance(prior, dict) else [] + used = prior.get("used_candidates", []) if isinstance(prior, dict) else [] + selected = prior.get("selected") if isinstance(prior, dict) else None + used_keys = { + (entry.get("adapter"), entry.get("target")) + for entry in used + if isinstance(entry, dict) + } + if isinstance(selected, dict): + used_keys.add((selected.get("adapter"), selected.get("target"))) + candidates_to_probe = [ + type("Target", (), candidate)() + for candidate in candidates + if isinstance(candidate, dict) + and candidate.get("execution_class") != "local_model" + and (candidate.get("adapter"), candidate.get("target")) not in used_keys + ] + else: + lane, grade = task.lane, task.grade + if not lane or not grade: + continue + try: + pol_dec = policy_mod.select_policy( + stage="worker", lane=lane, grade=grade, evaluated_at=admission_time + ) + except ValueError: + continue + candidates_to_probe = pol_dec.candidates + + for cand in candidates_to_probe: + if cand.execution_class == "local_model" and not is_retry: + break + spec = policy_mod.quota_probe_spec(cand) + if spec is not None: + key = (cand.adapter, cand.target, spec.command, tuple(spec.required_caps)) + if key not in seen_keys: + seen_keys.add(key) + unique_keys.append(key) + + if not unique_keys: + return None + + batch_id = f"batch-quota-{uuid.uuid4().hex[:12]}" + batch_provider_cls = getattr(selector, "QuotaBatchProvider", None) + if batch_provider_cls is None: + return None + batch_provider = batch_provider_cls(quota_probe_command=quota_probe_command) + return batch_provider.aggregate( + snapshot_id=batch_id, + checked_at=admission_time, + keys=unique_keys, + ) + + +def plan_number(task: Task) -> int: + if task.plan and task.plan.exists(): + match = PLAN_IDENTITY_RE.search( + task.plan.read_text(encoding="utf-8", errors="replace")[:1024] + ) + if match: + return int(match.group("plan")) + return 0 + + +def completing_decision_requires_selfcheck(state: dict[str, Any]) -> bool: + completing = state.get("completing_decision") + if not isinstance(completing, dict): + return False + selected = completing.get("selected") + if not isinstance(selected, dict): + return False + return selected.get("execution_class") == "local_model" + + +def _validated_completing_decision( + task: Task, decision: dict[str, Any] +) -> tuple[dict[str, Any], AgentSpec]: + """Strictly validate a completing decision against the task contract. + + Enforces that the decision's stage is "worker", its work_unit_id matches + the task's PLAN identity, and its selected fields pass the canonical + adapter/class/selfcheck normalization through `_spec_from_completing_decision`. + + Returns the validated decision and its normalized AgentSpec. + Raises ExecutionDecisionError on any contract violation so that callers + can fail closed rather than advancing to an inconsistent stage. + """ + if not isinstance(decision, dict): + raise ExecutionDecisionError( + "completing decision์ด dict๊ฐ€ ์•„๋‹ˆ๋‹ค" + ) + if decision.get("stage") != "worker": + raise ExecutionDecisionError( + f"completing decision stage๊ฐ€ worker๊ฐ€ ์•„๋‹ˆ๋‹ค: {decision.get('stage')!r}" + ) + expected_work_unit_id = work_unit_id_from_file(task.plan) + if decision.get("work_unit_id") != expected_work_unit_id: + raise ExecutionDecisionError( + f"completing decision work_unit_id ๋ถˆ์ผ์น˜: " + f"persisted={decision.get('work_unit_id')!r} " + f"plan={expected_work_unit_id!r}" + ) + spec = _spec_from_completing_decision(decision) + return decision, spec + + +def _completing_decision_is_valid( + task: Task, state: dict[str, Any] +) -> bool: + """Check whether the persisted completing decision satisfies the task contract. + + Validates stage, work_unit_id, and selected adapter/class/selfcheck + combination. Used by task_stage to prevent a worker_done state with no + authoritative completing decision from advancing to review. + """ + completing = state.get("completing_decision") + if not isinstance(completing, dict): + return False + try: + _validated_completing_decision(task, completing) + except ExecutionDecisionError: + return False + return True + + +def concrete_user_review_value(value: str) -> bool: + normalized = value.strip().strip("`").strip() + normalized = re.sub(r"^-\s*", "", normalized).strip() + if not normalized or re.search(r"\{[^}]+\}|<[^>]+>", normalized): + return False + return normalized.casefold() not in { + "-", + "n/a", + "na", + "none", + "unknown", + "๋ฏธ์ •", + "์—†์Œ", + "ํ•ด๋‹น ์—†์Œ", + } + + +def user_review_blocker_state(path: Path) -> tuple[bool, str]: + if not path.is_file(): + return False, "ํŒŒ์ผ์ด ์—†๋‹ค" + try: + text = path.read_text(encoding="utf-8", errors="replace") + except OSError as exc: + return False, f"ํŒŒ์ผ์„ ์ฝ์„ ์ˆ˜ ์—†๋‹ค: {exc}" + matched_schemas = [ + schema + for schema in USER_REVIEW_SCHEMAS + if re.search( + rf"^##\s*{re.escape(schema['status_heading'])}[ \t]*$", + text, + re.MULTILINE, + ) + ] + if len(matched_schemas) != 1: + return False, "์ง€์›ํ•˜๋Š” USER_REVIEW schema๊ฐ€ ์ •ํ™•ํžˆ ํ•˜๋‚˜๊ฐ€ ์•„๋‹ˆ๋‹ค" + schema = matched_schemas[0] + status = markdown_section(text, schema["status_heading"]).strip().strip("`") + if status != "USER_REVIEW": + return False, "์ƒํƒœ๊ฐ€ USER_REVIEW๊ฐ€ ์•„๋‹ˆ๋‹ค" + reason = markdown_section(text, schema["reason_heading"]) + gate_type_matches = re.findall( + rf"(?m)^-\s*{re.escape(schema['type_label'])}:\s*" + r"(milestone-lock|external-execution)\s*$", + reason, + ) + if len(gate_type_matches) != 1: + return False, "์ง€์›ํ•˜๋Š” user-review ์œ ํ˜•์ด ์ •ํ™•ํžˆ ํ•˜๋‚˜๊ฐ€ ์•„๋‹ˆ๋‹ค" + gate_type = gate_type_matches[0] + target = re.search( + rf"(?m)^-\s*{re.escape(schema['target_label'])}:\s*(.+?)\s*$", + reason, + ) + target_value = target.group(1) if target else "" + if not concrete_user_review_value(target_value): + return False, "๊ตฌ์ฒด์ ์ธ ์—ฐ๊ฒฐ ๋Œ€์ƒ์ด ์—†๋‹ค" + if gate_type == "milestone-lock" and ( + "agent-roadmap/" not in target_value + or "/milestones/" not in target_value + or ".md" not in target_value + ): + return False, "๊ตฌ์ฒด์ ์ธ Milestone ์—ฐ๊ฒฐ ๋Œ€์ƒ์ด ์—†๋‹ค" + evidence = markdown_section(text, schema["evidence_heading"]) + evidence_line = re.search( + rf"(?m)^-\s*{re.escape(schema['evidence_label'])}:\s*(.+?)\s*$", + evidence, + ) + if evidence_line is None or not concrete_user_review_value( + evidence_line.group(1) + ): + return False, "๊ตฌ์ฒด์ ์ธ ์ฐจ๋‹จ ํŒ๋‹จ ๊ทผ๊ฑฐ๊ฐ€ ์—†๋‹ค" + decision = markdown_section(text, schema["decision_headings"]) + unresolved = [ + value + for value in re.findall(r"(?m)^-\s*\[\s\]\s+(.+?)\s*$", decision) + if concrete_user_review_value(value) + ] + if not unresolved: + return False, "๋ฏธํ•ด๊ฒฐ ์‚ฌ์šฉ์ž ์กฐ์น˜ ๋˜๋Š” ๊ฒฐ์ • ํ•ญ๋ชฉ์ด ์—†๋‹ค" + resume = markdown_section(text, schema["resume_heading"]) + resume_conditions = [ + line + for line in resume.splitlines() + if concrete_user_review_value(line) + ] + if not resume_conditions: + return False, "๊ตฌ์ฒด์ ์ธ ์žฌ๊ฐœ ์กฐ๊ฑด์ด ์—†๋‹ค" + return True, f"unresolved {gate_type} user action or decision" + + +def task_stage(task: Task, state: dict[str, Any]) -> str: + if task.errors: + return "blocked" + if task.user_review: + if task.plan is not None or task.review is not None: + return "blocked" + blocking, _ = user_review_blocker_state(task.user_review) + return "user-review" if blocking else "blocked" + if task.recovery and (task.plan is None or task.review is None): + return "review" + if task.review and task.review.exists(): + text = task.review.read_text(encoding="utf-8", errors="replace") + if verdict_from_text(text): + return "review" + if state.get("worker_done"): + if not _completing_decision_is_valid(task, state): + return "blocked" + if completing_decision_requires_selfcheck(state) and not state.get("selfcheck_done"): + return "selfcheck" + return "review" + return "worker" + + +def markdown_section(text: str, heading: str | tuple[str, ...]) -> str: + headings = (heading,) if isinstance(heading, str) else heading + matches = [] + for h in headings: + for m in re.finditer(rf"^##\s*{re.escape(h)}[ \t]*$", text, re.MULTILINE): + matches.append(m) + if len(matches) != 1: + return "" + match = matches[0] + next_heading = re.search(r"^##\s+", text[match.end():], re.MULTILINE) + end = match.end() + next_heading.start() if next_heading else len(text) + return text[match.end():end].strip() + + +def implementation_review_errors(task: Task) -> list[str]: + if task.review is None or not task.review.is_file(): + return ["CODE_REVIEW ํŒŒ์ผ ์—†์Œ"] + text = task.review.read_text(encoding="utf-8", errors="replace") + checklist = markdown_section(text, IMPLEMENTATION_CHECKLIST_HEADINGS) + checkbox_values = IMPLEMENTATION_CHECKBOX_RE.findall(checklist) + if not checkbox_values or any(not value.strip() for value in checkbox_values): + return ["๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ ๋ฏธ์™„๋ฃŒ"] + return [] + + +def classify_failure_with_evidence(output: str) -> tuple[str, str | None]: + lines = output.splitlines() + for category, patterns in PROMOTABLE_PATTERNS.items(): + for line in reversed(lines): + lowered = line.lower() + if any(re.search(pattern, lowered, re.DOTALL) for pattern in patterns): + return category, line + return "generic-error", None + + +def classify_failure(output: str) -> str: + return classify_failure_with_evidence(output)[0] + + +def termination_signal(return_code: int) -> tuple[str, bool] | None: + signal_number: int | None = None + inferred = False + if return_code < 0: + signal_number = -return_code + elif return_code > 128: + signal_number = return_code - 128 + inferred = True + if signal_number is None: + return None + try: + return signal.Signals(signal_number).name, inferred + except ValueError: + return None + + +def failure_report_lines(failure: str, locator: Path) -> list[str]: + record: dict[str, Any] = {} + try: + record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + pass + failure_class = str(record.get("failure_class") or failure) + source = str(record.get("failure_source") or "unverified") + provider_confirmed = bool( + record.get("provider_transport_failure_confirmed", False) + ) + lines = [ + f"failure_class={failure_class}", + f"failure_source={source}", + "provider_transport_failure_confirmed=" + f"{str(provider_confirmed).lower()}", + ] + if record.get("dispatcher_pid") is not None: + lines.append(f"dispatcher_pid={record['dispatcher_pid']}") + if record.get("agent_pid") is not None: + lines.append(f"agent_pid={record['agent_pid']}") + if record.get("dispatcher_source_sha256"): + lines.append( + f"dispatcher_source_sha256={record['dispatcher_source_sha256']}" + ) + source_matches_loaded = record.get("dispatcher_source_matches_loaded") + if source_matches_loaded is not None: + lines.append( + "dispatcher_source_matches_loaded=" + f"{str(bool(source_matches_loaded)).lower()}" + ) + if ( + source_matches_loaded is False + and record.get("dispatcher_source_current_sha256") + ): + lines.append( + "dispatcher_source_current_sha256=" + f"{record['dispatcher_source_current_sha256']}" + ) + if provider_confirmed: + evidence_source = record.get("failure_evidence_source") + evidence = record.get("failure_evidence_excerpt") + if evidence_source: + lines.append(f"provider_evidence_source={evidence_source}") + if evidence: + rendered = str(evidence).replace("\r", r"\r").replace("\n", r"\n") + lines.append(f"provider_evidence={rendered}") + if failure_class == "session-stall": + lines.extend( + [ + f"timeout_phase={record.get('pi_session_phase') or 'unknown'}", + f"timeout_seconds={record.get('session_stall_seconds') or 'unknown'}", + "termination_initiator=" + f"{record.get('termination_initiator') or 'dispatcher'}", + ] + ) + elif failure_class == "process-terminated": + lines.extend( + [ + f"termination_signal={record.get('termination_signal') or 'unknown'}", + "termination_initiator=" + f"{record.get('termination_initiator') or 'unknown'}", + ] + ) + lines.append(f"locator={locator}") + return lines + + +def terminal_diagnostic(cli: str, channel: str, line: str) -> str | None: + if channel == "stderr": + return line + try: + value = json.loads(line) + except json.JSONDecodeError: + if cli == "agy" and re.match( + r"^\s*(?:error|fatal|provider error|model error)\b", line, re.IGNORECASE + ): + return line + return None + event_type = str(value.get("type", "")) + if cli == "codex" and event_type in {"turn.failed", "error"}: + return json.dumps(value.get("error", value), ensure_ascii=False) + if cli == "agy": + severity = str(value.get("severity") or value.get("level") or "") + status = str(value.get("status") or "") + non_terminal_event = event_type.lower() in { + "assistant", + "message", + "tool", + "tool.result", + "tool_result", + } + if ( + not non_terminal_event + and ( + event_type.lower() in { + "error", + "fatal", + "request.failed", + "turn.failed", + } + or severity.lower() in {"error", "fatal"} + or ( + status.lower() in {"failed", "rejected"} + and any( + field in value + for field in ( + "code", + "error", + "error_code", + "status_code", + ) + ) + ) + ) + ): + return json.dumps(value, ensure_ascii=False) + if cli == "claude": + subtype = str(value.get("subtype", "")) + if event_type == "rate_limit_event": + rate_limit_info = value.get("rate_limit_info") + if isinstance(rate_limit_info, dict) and str( + rate_limit_info.get("status", "") + ).lower() == "rejected": + return json.dumps(value, ensure_ascii=False) + if event_type == "result" and ( + value.get("is_error") or subtype.startswith("error") + ): + # Preserve typed terminal fields such as api_error_status=429 and + # error=rate_limit. The human-readable result alone is not the + # failure contract and may change between Claude CLI releases. + return json.dumps(value, ensure_ascii=False) + if event_type == "system" and subtype.startswith("error"): + return json.dumps(value, ensure_ascii=False) + return None + + +def legacy_promotion_recovery( + runs_root: Path, + task: Task, + state: dict[str, Any], +) -> LegacyPromotionRecovery | None: + """Reclassify only an older dispatcher's exhausted generic terminal failure.""" + blocked = str(state.get("blocked") or "") + recovery_failures = state.get("recovery_failures") + if not blocked or not isinstance(recovery_failures, dict): + return None + locator_match = re.search(r"(?:^|\s)locator=(.+?)\s*$", blocked) + if locator_match is None: + return None + locator = Path(locator_match.group(1)) + try: + locator = locator.resolve(strict=True) + runs_root = runs_root.resolve(strict=True) + if not locator.is_relative_to(runs_root) or locator.name != "locator.json": + return None + latest_record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + role = str(latest_record.get("role") or "") + try: + failure_count = int(recovery_failures.get(role, 0)) + except (TypeError, ValueError): + return None + prior_sha256 = str(latest_record.get("dispatcher_source_sha256") or "") + failed_spec = agent_spec_from_record(latest_record) + if ( + latest_record.get("task") != task.name + or latest_record.get("status") != "failed" + or latest_record.get("failure_class") != "generic-error" + or role not in {"worker", "selfcheck", "review"} + or failure_count < RECOVERY_FAILURE_LIMIT + or not prior_sha256 + or prior_sha256 == DISPATCHER_SOURCE_SHA256 + or failed_spec is None + or promoted_spec(failed_spec, 0) is None + ): + return None + + plan = latest_record.get("plan_number") + latest_attempt = latest_record.get("attempt") + if not isinstance(latest_attempt, int): + return None + classified_attempts: list[ + tuple[int, Path, str, str, str] + ] = [] + for attempt_directory in runs_root.iterdir(): + candidate = attempt_directory / "locator.json" + if not candidate.is_file(): + continue + try: + record = json.loads(candidate.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + if ( + record.get("task") != task.name + or record.get("role") != role + or record.get("plan_number") != plan + or record.get("status") != "failed" + or record.get("failure_class") != "generic-error" + or record.get("dispatcher_source_sha256") != prior_sha256 + or record.get("cli") != failed_spec.cli + or record.get("model") != failed_spec.model + or effective_reasoning_effort( + agent_spec_from_record(record) or failed_spec + ) + != effective_reasoning_effort(failed_spec) + or not isinstance(record.get("attempt"), int) + ): + continue + diagnostics = attempt_terminal_diagnostics( + attempt_directory, + record, + ) + if not diagnostics: + continue + failure_class, evidence = classify_failure_with_evidence( + "\n".join(diagnostic for _, diagnostic in diagnostics[-50:]) + ) + if failure_class not in CLOUD_PROMOTION_FAILURES or evidence is None: + continue + evidence_source = next( + ( + source + for source, diagnostic in reversed(diagnostics) + if diagnostic == evidence + ), + f"{failed_spec.cli}:terminal", + ) + classified_attempts.append( + ( + int(record["attempt"]), + candidate.resolve(), + failure_class, + evidence, + evidence_source, + ) + ) + classified_attempts.sort(key=lambda item: item[0]) + expected_attempts = list( + range(latest_attempt - failure_count + 1, latest_attempt + 1) + ) + matching_attempts = [ + item + for item in classified_attempts + if item[0] in expected_attempts + ] + if ( + [item[0] for item in matching_attempts] != expected_attempts + or matching_attempts[-1][1] != locator + or len({item[2] for item in matching_attempts}) != 1 + ): + # Do not collapse a mixed or incomplete failure history to one retry. + return None + _, _, failure_class, evidence, evidence_source = matching_attempts[-1] + return LegacyPromotionRecovery( + locator=locator, + role=role, + failure_class=failure_class, + evidence=evidence, + evidence_source=evidence_source, + prior_dispatcher_sha256=prior_sha256, + failed_cli=failed_spec.cli, + failed_model=failed_spec.model, + failed_reasoning_effort=failed_spec.reasoning_effort, + ) + + +def persisted_legacy_promotion_recovery( + task: Task, + state: dict[str, Any], + locator: Path, + role: str, +) -> LegacyPromotionRecovery | None: + metadata = state.get("legacy_terminal_reclassification") + if not isinstance(metadata, dict): + return None + try: + recorded_locator = Path(str(metadata["locator"])).resolve(strict=True) + record = json.loads(recorded_locator.read_text(encoding="utf-8")) + except (KeyError, OSError): + return None + except json.JSONDecodeError: + return None + if not isinstance(record, dict): + return None + failure_class = str(metadata.get("failure_class") or "") + failed_spec = agent_spec_from_record(record) + recorded_cli = str(metadata.get("failed_cli") or "") + recorded_model = str(metadata.get("failed_model") or "") + if ( + recorded_locator != locator.resolve() + or record.get("task") != task.name + or record.get("role") != role + or record.get("status") != "failed" + or record.get("failure_class") != "generic-error" + or failure_class not in CLOUD_PROMOTION_FAILURES + or str(metadata.get("current_dispatcher_sha256") or "") + != DISPATCHER_SOURCE_SHA256 + or failed_spec is None + or promoted_spec(failed_spec, 0) is None + or (recorded_cli and recorded_cli != failed_spec.cli) + or (recorded_model and recorded_model != failed_spec.model) + ): + return None + return LegacyPromotionRecovery( + locator=recorded_locator, + role=role, + failure_class=failure_class, + evidence=failure_class, + evidence_source=str(metadata.get("evidence_source") or "terminal"), + prior_dispatcher_sha256=str( + metadata.get("prior_dispatcher_sha256") or "unknown" + ), + failed_cli=failed_spec.cli, + failed_model=failed_spec.model, + failed_reasoning_effort=( + str(metadata["failed_reasoning_effort"]) + if metadata.get("failed_reasoning_effort") is not None + else failed_spec.reasoning_effort + ), + ) + + +def pending_persisted_legacy_promotion_recovery( + task: Task, + state: dict[str, Any], +) -> LegacyPromotionRecovery | None: + metadata = state.get("legacy_terminal_reclassification") + recovery_failures = state.get("recovery_failures") + if not isinstance(metadata, dict) or not isinstance( + recovery_failures, dict + ): + return None + role = str(metadata.get("role") or "") + if not role: + pending_roles = [ + str(candidate) + for candidate, count in recovery_failures.items() + if count + ] + if len(pending_roles) != 1: + return None + role = pending_roles[0] + try: + failure_count = int(recovery_failures.get(role, 0)) + locator = Path(str(metadata["locator"])) + except (KeyError, TypeError, ValueError): + return None + if not 0 < failure_count < RECOVERY_FAILURE_LIMIT: + return None + return persisted_legacy_promotion_recovery( + task, + state, + locator, + role, + ) + + +def codex_collaboration_tool(line: str) -> str | None: + try: + value = json.loads(line) + except json.JSONDecodeError: + return None + item = value.get("item") or {} + if ( + value.get("type") == "item.started" + and item.get("type") == "collab_tool_call" + and item.get("tool") + ): + return str(item["tool"]) + return None + + +async def terminate_process_group( + process: asyncio.subprocess.Process, + grace_seconds: float = 5, +) -> None: + """Terminate the exact subprocess group and escalate if descendants remain.""" + try: + os.killpg(process.pid, signal.SIGTERM) + except ProcessLookupError: + if process.returncode is None: + await process.wait() + return + + if process.returncode is None: + try: + await asyncio.wait_for(process.wait(), timeout=grace_seconds) + except TimeoutError: + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + await process.wait() + return + + try: + os.killpg(process.pid, 0) + except ProcessLookupError: + return + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + + +def agy_log_diagnostics(path: Path) -> list[str]: + if not path.exists(): + return [] + diagnostics: list[str] = [] + for line in path.read_text(encoding="utf-8", errors="replace").splitlines()[-200:]: + failure_class, evidence = classify_failure_with_evidence(line) + if ( + failure_class not in CLOUD_PROMOTION_FAILURES + or evidence is None + ): + continue + if failure_class == "provider-quota" and not re.search( + ( + r"RESOURCE[_ ]?EXHAUSTED" + r"|\b(?:HTTP|status(?: code)?)\s*[:=]?\s*429\b" + r"|\btoo many requests\b" + r"|(?:rate.?limit|quota|capacity).{0,40}" + r"(?:exceed|exhaust|reached|reject)" + r"|(?:exceed|exhaust|reached|reject).{0,40}" + r"(?:rate.?limit|quota|capacity)" + r"|(?:rate.?limit|quota).{0,40}retry after" + ), + line, + re.IGNORECASE, + ): + continue + diagnostics.append(line) + return diagnostics + + +def attempt_terminal_diagnostics( + attempt_directory: Path, + record: dict[str, Any], +) -> list[tuple[str, str]]: + spec = agent_spec_from_record(record) + if spec is None: + return [] + try: + stream_lines = (attempt_directory / "stream.log").read_text( + encoding="utf-8", + errors="replace", + ).splitlines() + except OSError: + stream_lines = [] + diagnostics: list[tuple[str, str]] = [] + for stream_line in stream_lines: + match = re.match(r"^\[(stdout|stderr)\]\s?(.*)$", stream_line) + if match is None: + continue + channel, payload = match.groups() + diagnostic = terminal_diagnostic(spec.cli, channel, payload) + if diagnostic: + diagnostics.append((f"{spec.cli}:{channel}", diagnostic)) + if spec.cli == "agy": + diagnostics.extend( + ("agy:cli-log", diagnostic) + for diagnostic in agy_log_diagnostics( + attempt_directory / "agy-cli.log" + ) + ) + return diagnostics + + +def promoted_spec(spec: AgentSpec, recovery_count: int) -> AgentSpec | None: + if spec.cli == "agy": + return AgentSpec("claude", "claude-opus-4-8", "claude/claude-opus-4-8 xhigh") + if spec.cli == "claude": + return AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + if spec.cli == "codex" and recovery_count < 1: + return spec + return None + + +def render_json_line(cli: str, line: str) -> tuple[list[str], str | None]: + try: + value = json.loads(line) + except json.JSONDecodeError: + return [line.rstrip()], None + session_id = value.get("thread_id") or value.get("session_id") + rendered: list[str] = [] + if cli == "codex": + if value.get("type") == "thread.started" and session_id: + rendered.append(f"session={session_id}") + item = value.get("item") or {} + item_type = item.get("type") + if item_type == "agent_message" and item.get("text"): + rendered.extend(str(item["text"]).splitlines()) + elif item_type == "command_execution": + rendered.append(f"$ {item.get('command', '')} (exit={item.get('exit_code', '?')})") + elif item_type in {"mcp_tool_call", "web_search"}: + rendered.append(f"{item_type}: {item.get('server', '')} {item.get('tool', item.get('query', ''))}") + elif value.get("type") == "turn.failed": + rendered.append(str(value.get("error", value))) + elif cli == "claude": + message = value.get("message") or {} + for block in message.get("content") or []: + if block.get("type") == "text": + rendered.extend(str(block.get("text", "")).splitlines()) + elif block.get("type") == "tool_use": + rendered.append(f"tool={block.get('name', '')}") + if value.get("type") == "result" and value.get("result"): + rendered.extend(str(value["result"]).splitlines()) + session_id = session_id or value.get("session_id") + return rendered, str(session_id) if session_id else None + + +def native_session_path(cli: str, workspace: Path, session_id: str | None, attempt_dir: Path) -> str | None: + if cli == "claude" and session_id: + encoded = str(workspace).replace("/", "-") + return str(Path.home() / ".claude" / "projects" / encoded / f"{session_id}.jsonl") + if cli == "pi" and session_id: + matches = list((attempt_dir / "pi-sessions").glob(f"*{session_id}*.jsonl")) + return str(matches[0]) if matches else str(attempt_dir / "pi-sessions") + if cli == "codex" and session_id: + matches = list((Path.home() / ".codex" / "sessions").glob(f"**/*{session_id}*.jsonl")) + return str(matches[0]) if matches else str(Path.home() / ".codex" / "sessions") + return None + + +def native_session_mtime_ns(path: str | None) -> int | None: + if not path: + return None + candidate = Path(path) + return candidate.stat().st_mtime_ns if candidate.is_file() else None + + +def reverse_jsonl_lines(path: Path): + with path.open("rb") as stream: + stream.seek(0, os.SEEK_END) + position = stream.tell() + buffer = b"" + while position > 0: + read_size = min(8192, position) + position -= read_size + stream.seek(position) + buffer = stream.read(read_size) + buffer + lines = buffer.split(b"\n") + buffer = lines[0] + for line in reversed(lines[1:]): + if line.strip(): + yield line + if buffer.strip(): + yield buffer + + +def pi_session_header_version(path: Path) -> int | None: + with path.open("rb") as stream: + first_line = stream.readline() + if not first_line.strip(): + return None + header = json.loads(first_line) + if not isinstance(header, dict) or header.get("type") != "session": + return None + version = header.get("version") + return version if isinstance(version, int) else None + + +def pi_native_session_state(path: str | None) -> PiSessionState: + if not path: + return PiSessionState("starting", reason="native-session-path-missing") + candidate = Path(path) + if not candidate.is_file(): + return PiSessionState("starting", reason="native-session-file-missing") + completed_ids: list[str] = [] + expected_entry_id: str | None = None + active_leaf_found = False + try: + version = pi_session_header_version(candidate) + if version != PI_SESSION_SCHEMA_VERSION: + return PiSessionState( + "unknown", + reason=( + f"unsupported-session-version:{version}" + if version is not None + else "session-header-invalid" + ), + ) + for raw_line in reverse_jsonl_lines(candidate): + value = json.loads(raw_line) + if not isinstance(value, dict): + return PiSessionState("unknown", reason="invalid-entry-schema") + if value.get("type") == "session": + break + entry_id = value.get("id") + parent_id = value.get("parentId") + if ( + not isinstance(entry_id, str) + or not entry_id + or "parentId" not in value + or (parent_id is not None and not isinstance(parent_id, str)) + ): + return PiSessionState("unknown", reason="invalid-entry-identity") + if active_leaf_found and entry_id != expected_entry_id: + continue + active_leaf_found = True + expected_entry_id = parent_id + if value.get("type") != "message": + continue + message = value.get("message") + if not isinstance(message, dict): + return PiSessionState("unknown", reason="invalid-message-schema") + role = message.get("role") + if role == "toolResult": + tool_call_id = message.get("toolCallId") + if not isinstance(tool_call_id, str) or not tool_call_id: + return PiSessionState( + "unknown", reason="tool-result-id-missing" + ) + if tool_call_id in completed_ids: + return PiSessionState( + "unknown", reason="duplicate-tool-result-id" + ) + completed_ids.append(tool_call_id) + continue + if role == "user": + if completed_ids: + return PiSessionState( + "unknown", reason="tool-results-without-assistant" + ) + return PiSessionState("awaiting-model", reason="user-message") + if role != "assistant": + return PiSessionState( + "unknown", reason=f"unsupported-message-role:{role}" + ) + + content = message.get("content") + if not isinstance(content, list): + return PiSessionState( + "unknown", reason="assistant-content-not-list" + ) + if any( + not isinstance(block, dict) + or block.get("type") not in {"text", "thinking", "toolCall"} + for block in content + ): + return PiSessionState( + "unknown", reason="unsupported-assistant-content" + ) + tool_calls = [ + block + for block in content + if isinstance(block, dict) and block.get("type") == "toolCall" + ] + if not tool_calls: + if completed_ids: + return PiSessionState( + "unknown", reason="tool-results-without-tool-calls" + ) + return PiSessionState("finishing", reason="assistant-final") + + expected_ids: list[str] = [] + for tool_call in tool_calls: + tool_call_id = tool_call.get("id") + if not isinstance(tool_call_id, str) or not tool_call_id: + return PiSessionState( + "unknown", reason="tool-call-id-missing" + ) + if tool_call_id in expected_ids: + return PiSessionState( + "unknown", reason="duplicate-tool-call-id" + ) + expected_ids.append(tool_call_id) + + unexpected_ids = [ + tool_call_id + for tool_call_id in completed_ids + if tool_call_id not in expected_ids + ] + if unexpected_ids: + return PiSessionState( + "unknown", reason="tool-result-id-not-in-latest-batch" + ) + completed_set = set(completed_ids) + completed = tuple( + tool_call_id + for tool_call_id in expected_ids + if tool_call_id in completed_set + ) + pending = tuple( + tool_call_id + for tool_call_id in expected_ids + if tool_call_id not in completed_set + ) + return PiSessionState( + "tool-running" if pending else "awaiting-model", + expected_tool_call_ids=tuple(expected_ids), + completed_tool_call_ids=completed, + pending_tool_call_ids=pending, + reason=( + "pending-tool-results" + if pending + else "all-tool-results-recorded" + ), + ) + if completed_ids: + return PiSessionState( + "unknown", reason="tool-results-without-assistant" + ) + if active_leaf_found and expected_entry_id is not None: + return PiSessionState("unknown", reason="active-branch-parent-missing") + except (OSError, UnicodeDecodeError, json.JSONDecodeError): + return PiSessionState("unknown", reason="unreadable-jsonl") + return PiSessionState("starting", reason="no-message-events") + + +def pi_native_session_phase(path: str | None) -> str: + return pi_native_session_state(path).phase + + +def log_tail_excerpt(path: Path, *, byte_limit: int = 8192, char_limit: int = 2000) -> str: + """Return a bounded recent log excerpt without loading a long reasoning stream.""" + try: + with path.open("rb") as stream: + stream.seek(max(0, path.stat().st_size - byte_limit)) + text = stream.read().decode("utf-8", errors="replace") + except OSError as exc: + return f"" + return text[-char_limit:] + + +def process_start_token(value: Any) -> str | None: + """Read Linux process start ticks so PID reuse is not treated as liveness.""" + try: + pid = int(value) + text = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8") + close = text.rfind(")") + fields = text[close + 2 :].split() + return fields[19] if close >= 0 and len(fields) > 19 else None + except (TypeError, ValueError, OSError): + return None + + +def process_is_alive(value: Any, expected_start_token: Any = None) -> bool: + """Return whether the same attempt/dispatcher process still exists.""" + try: + pid = int(value) + if pid <= 0: + return False + os.kill(pid, 0) + except (TypeError, ValueError, OSError): + return False + current_token = process_start_token(pid) + if ( + expected_start_token is not None + and current_token is not None + and str(expected_start_token) != current_token + ): + return False + return True + + +def marked_agent_process_pids(marker: str) -> list[int]: + """Find live processes carrying the per-attempt environment marker.""" + expected = f"{AGENT_PROCESS_MARKER_ENV}={marker}".encode() + matches: list[int] = [] + for environ in Path("/proc").glob("[0-9]*/environ"): + try: + values = environ.read_bytes().split(b"\0") + pid = int(environ.parent.name) + except (OSError, ValueError): + continue + if expected in values: + matches.append(pid) + return sorted(matches) + + +def locator_workspace_ownership( + locator_path: Path, + locator: dict[str, Any], + *, + expected_workspace: Path | None = None, + expected_workspace_id: str | None = None, + expected_runs_root: Path | None = None, +) -> tuple[bool, str]: + if expected_workspace is None and expected_workspace_id is None: + return True, "" + try: + expected_root = ( + expected_workspace.resolve() if expected_workspace is not None else None + ) + expected_id = expected_workspace_id + if expected_id is None and expected_root is not None: + expected_id = hashlib.sha256(str(expected_root).encode()).hexdigest()[:16] + if expected_runs_root is None: + return False, "ํ˜„์žฌ workspace์˜ locator runs root๊ฐ€ ์—†๋‹ค" + resolved_runs = expected_runs_root.resolve() + resolved_locator = locator_path.resolve() + resolved_locator.relative_to(resolved_runs) + except (OSError, RuntimeError, ValueError): + return ( + False, + "foreign workspace locator path: " + f"locator={locator_path} expected_runs={expected_runs_root}", + ) + + recorded_workspace = locator.get("workspace") + recorded_workspace_id = locator.get("workspace_id") + if recorded_workspace_id not in (None, "") and ( + str(recorded_workspace_id) != str(expected_id) + ): + return ( + False, + "foreign workspace locator id: " + f"recorded={recorded_workspace_id} expected={expected_id}", + ) + if recorded_workspace not in (None, ""): + try: + recorded_root = Path(str(recorded_workspace)).resolve() + except (OSError, RuntimeError): + return False, "locator workspace ๊ฒฝ๋กœ๋ฅผ canonicalizeํ•  ์ˆ˜ ์—†๋‹ค" + if expected_root is not None and recorded_root != expected_root: + return ( + False, + "foreign workspace locator root: " + f"recorded={recorded_root} expected={expected_root}", + ) + evidence_fields = ["stream_log"] + if locator.get("cli") == "pi": + evidence_fields.append("native_session_path") + for field in evidence_fields: + raw_evidence = locator.get(field) + if raw_evidence in (None, ""): + continue + try: + Path(str(raw_evidence)).resolve().relative_to(resolved_runs) + except (OSError, RuntimeError, ValueError): + return ( + False, + "foreign workspace locator evidence: " + f"field={field} path={raw_evidence}", + ) + # An identity-less legacy locator is accepted only because physical + # containment under the current store's runs root was already proved. + return True, "" + + +def external_active_is_live( + state: dict[str, Any], + *, + expected_workspace: Path | None = None, + expected_workspace_id: str | None = None, + expected_runs_root: Path | None = None, +) -> tuple[bool, str]: + raw_locator = state.get("active_locator") + if not raw_locator: + return False, "active locator ์—†์Œ" + target = Path(str(raw_locator)) + locator_path = target if target.name == "locator.json" else target / "locator.json" + path_owned, ownership_detail = locator_workspace_ownership( + locator_path, + {}, + expected_workspace=expected_workspace, + expected_workspace_id=expected_workspace_id, + expected_runs_root=expected_runs_root, + ) + if not path_owned: + return False, ownership_detail + locator: dict[str, Any] = {} + if locator_path.is_file(): + try: + locator = json.loads(locator_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return False, f"locator ํŒ๋… ์‹คํŒจ: {locator_path}" + if not isinstance(locator, dict): + return False, f"locator object ํ˜•์‹์ด ์•„๋‹ˆ๋‹ค: {locator_path}" + + owned, ownership_detail = locator_workspace_ownership( + locator_path, + locator, + expected_workspace=expected_workspace, + expected_workspace_id=expected_workspace_id, + expected_runs_root=expected_runs_root, + ) + if not owned: + return False, ownership_detail + + if locator: + status = str(locator.get("status") or "") + if status and status != "running": + return False, f"locator status={status}" + + # The stream may legitimately remain quiet during long reasoning. A live + # process is stronger evidence than a locator or dispatcher heartbeat, and + # prevents a second dispatcher from duplicating an active attempt. + agent_pid_recorded = locator.get("agent_pid") not in (None, "") + for field, token_field in ( + ("agent_pid", "agent_process_start_token"), + ("dispatcher_pid", "dispatcher_process_start_token"), + ): + if process_is_alive(locator.get(field), locator.get(token_field)): + return True, f"{field}={locator[field]} alive; output stream is monitored" + process_marker = str(locator.get("agent_process_marker") or "") + if process_marker: + marker_pids = marked_agent_process_pids(process_marker) + if marker_pids: + return ( + True, + "agent process marker alive: " + + ",".join(str(pid) for pid in marker_pids), + ) + return ( + False, + "agent process marker is absent from the process table", + ) + if agent_pid_recorded: + return ( + False, + "recorded agent process identity is no longer alive", + ) + + native_raw = locator.get("native_session_path") + native = Path(str(native_raw)) if native_raw else None + if native and native.is_dir(): + sessions = list(native.glob("*.jsonl")) + native = max(sessions, key=lambda path: path.stat().st_mtime_ns) if sessions else None + if native is None or not native.is_file(): + roots = [target] if target.is_dir() else [target.parent] + sessions = [ + path + for root in roots + for path in (*root.glob("*.jsonl"), *root.glob("pi-sessions/*.jsonl")) + ] + native = max(sessions, key=lambda path: path.stat().st_mtime_ns) if sessions else None + now = datetime.now(timezone.utc).timestamp() + cli = str(locator.get("cli") or "") + stream_progress_at: float | None = None + stream_raw = locator.get("stream_log") + stream = Path(str(stream_raw)) if stream_raw else None + if stream and stream.is_file(): + stream_progress_at = stream.stat().st_mtime + if native and native.is_file(): + native_progress_at = native.stat().st_mtime + progress_at = max(native_progress_at, stream_progress_at or 0.0) + inactive = max(0.0, now - progress_at) + if cli == "pi": + phase = pi_native_session_phase(str(native)) + # Only an exact incomplete toolCall -> toolResult batch is a tool + # execution interval. Unknown/starting/model-reasoning states + # must never be treated as a stalled tool merely because their + # native event file is quiet. + if phase == "tool-running": + return ( + True, + "phase=tool-running with no agent PID evidence; " + "time-based duplicate recovery is disabled", + ) + return ( + True, + f"phase={phase} native+stream inactive={inactive:.1f}s " + "with no agent PID evidence; time-based duplicate recovery is disabled", + ) + return ( + True, + "native+stream inactive=" + f"{inactive:.1f}s with no agent PID evidence; " + "time-based duplicate recovery is disabled", + ) + + if stream_progress_at is not None: + inactive = max(0.0, now - stream_progress_at) + return ( + True, + f"stream inactive={inactive:.1f}s with no agent PID evidence; " + "time-based duplicate recovery is disabled", + ) + + return False, f"active ์ฆ๊ฑฐ ์—†์Œ: {raw_locator}" + + +def laguna_resume_locator( + state: dict[str, Any], + *, + expected_workspace: Path | None = None, + expected_workspace_id: str | None = None, + expected_runs_root: Path | None = None, +) -> Path | None: + raw_locator = state.get("active_locator") + if not raw_locator: + return None + target = Path(str(raw_locator)) + locator = target if target.name == "locator.json" else target / "locator.json" + path_owned, _ = locator_workspace_ownership( + locator, + {}, + expected_workspace=expected_workspace, + expected_workspace_id=expected_workspace_id, + expected_runs_root=expected_runs_root, + ) + if not path_owned: + return None + try: + record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(record, dict): + return None + owned, _ = locator_workspace_ownership( + locator, + record, + expected_workspace=expected_workspace, + expected_workspace_id=expected_workspace_id, + expected_runs_root=expected_runs_root, + ) + if not owned: + return None + if ( + record.get("cli") != "pi" + or not str(record.get("model", "")).startswith("laguna-s") + or record.get("failure_class") not in {"context-limit", "session-stall"} + or record.get("status") != "failed" + ): + return None + native_raw = record.get("native_session_path") + native = Path(str(native_raw)) if native_raw else None + if native is None or not native.exists(): + return None + if expected_runs_root is not None: + try: + native.resolve().relative_to(expected_runs_root.resolve()) + except (OSError, RuntimeError, ValueError): + return None + return locator + + +def selfcheck_context_resume_locator( + state: dict[str, Any], + task: Task, + *, + expected_workspace: Path, + expected_workspace_id: str, + expected_runs_root: Path, +) -> tuple[Path | None, str]: + raw_locator = state.get("selfcheck_context_locator") + if not isinstance(raw_locator, str) or not raw_locator: + return None, "persisted selfcheck context locator๊ฐ€ ์—†๋‹ค" + target = Path(raw_locator) + locator = target if target.name == "locator.json" else target / "locator.json" + path_owned, detail = locator_workspace_ownership( + locator, + {}, + expected_workspace=expected_workspace, + expected_workspace_id=expected_workspace_id, + expected_runs_root=expected_runs_root, + ) + if not path_owned: + return None, detail + try: + record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None, "persisted selfcheck context locator๋ฅผ ์ฝ์„ ์ˆ˜ ์—†๋‹ค" + if not isinstance(record, dict): + return None, "persisted selfcheck context locator ํ˜•์‹์ด ์ž˜๋ชป๋๋‹ค" + owned, detail = locator_workspace_ownership( + locator, + record, + expected_workspace=expected_workspace, + expected_workspace_id=expected_workspace_id, + expected_runs_root=expected_runs_root, + ) + if not owned: + return None, detail + if ( + record.get("task") != task.name + or record.get("role") != "selfcheck" + or record.get("cli") != "pi" + or record.get("status") != "succeeded" + ): + return None, "persisted selfcheck context locator identity๊ฐ€ ์ผ์น˜ํ•˜์ง€ ์•Š๋Š”๋‹ค" + native_raw = record.get("native_session_path") + native = Path(str(native_raw)) if native_raw else None + if native is not None and native.is_dir(): + sessions = list(native.glob("*.jsonl")) + native = max(sessions, key=lambda path: path.stat().st_mtime_ns) if sessions else None + if native is None or not native.is_file(): + return None, "persisted selfcheck native session์ด ์—†๋‹ค" + try: + native.resolve().relative_to(expected_runs_root.resolve()) + except (OSError, RuntimeError, ValueError): + return None, "persisted selfcheck native session์ด workspace runs ๋ฐ–์— ์žˆ๋‹ค" + return locator, "" + + +def agy_conversations() -> dict[Path, int]: + root = Path.home() / ".gemini" / "antigravity-cli" / "conversations" + if not root.is_dir(): + return {} + return {path: path.stat().st_mtime_ns for path in root.glob("*.db")} + + +def build_command( + spec: AgentSpec, + prompt: str, + workspace: Path, + session_id: str, + attempt_dir: Path, + pi_resume_session: Path | None = None, +) -> list[str]: + if spec.cli == "codex": + return [ + "codex", "exec", "--json", "-C", str(workspace), "-m", spec.model, + "-c", f'model_reasoning_effort="{effective_reasoning_effort(spec)}"', + "--dangerously-bypass-approvals-and-sandbox", prompt, + ] + if spec.cli == "claude": + return [ + "claude", "-p", "--output-format", "stream-json", "--verbose", + "--session-id", session_id, "--model", spec.model, + "--effort", str(effective_reasoning_effort(spec)), + "--dangerously-skip-permissions", prompt, + ] + if spec.cli == "agy": + return [ + # `--print` consumes its immediately following argument as the prompt. + # Keeping the timeout first makes Gemini answer the literal flag instead. + "agy", "--print", prompt, "--print-timeout", "8h", "--model", spec.model, + "--dangerously-skip-permissions", "--log-file", str(attempt_dir / "agy-cli.log"), + ] + if spec.cli == "pi": + command = [ + "pi", "-p", "--mode", "json", "--approve", "--provider", "iop", "--model", spec.model, + "--thinking", "high", + ] + if pi_resume_session is not None: + command.extend( + [ + "--session", str(pi_resume_session), + "--session-dir", str(pi_resume_session.parent), + ] + ) + else: + command.extend( + [ + "--session-id", session_id, + "--session-dir", str(attempt_dir / "pi-sessions"), + ] + ) + command.append(prompt) + return command + raise RuntimeError(f"์ง€์›ํ•˜์ง€ ์•Š๋Š” CLI: {spec.cli}") + + +async def invoke( + workspace: Path, + store: StateStore, + task: Task, + role: str, + spec: AgentSpec, + prompt: str, + resume_locator: Path | None = None, +) -> tuple[int, str | None, Path]: + attempt, identity = next_execution_identity(store, task, role) + attempt_dir = store.runs / f"{datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')}__{identity}" + attempt_dir.mkdir(parents=True, exist_ok=False) + locator_path = attempt_dir / "locator.json" + stream_path = attempt_dir / "stream.log" + normalized_output_path = attempt_dir / "normalized-output.log" + heartbeat_path = attempt_dir / "heartbeat.log" + stream_path.touch() + normalized_output_path.touch() + heartbeat_path.touch() + session_id = str(uuid.uuid4()) + process_marker = f"w{store.workspace_id}__{identity}__{uuid.uuid4()}" + pi_resume_session: Path | None = None + if spec.local_pi and resume_locator and resume_locator.is_file(): + resume_locator_path = ( + resume_locator + if resume_locator.name == "locator.json" + else resume_locator / "locator.json" + ) + path_owned, _ = locator_workspace_ownership( + resume_locator_path, + {}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if path_owned: + try: + prior = json.loads(resume_locator_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + prior = {} + else: + prior = {} + owned, _ = locator_workspace_ownership( + resume_locator_path, + prior if isinstance(prior, dict) else {}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if owned and isinstance(prior, dict): + prior_native = prior.get("native_session_path") + candidate = Path(str(prior_native)) if prior_native else None + if candidate and candidate.is_dir(): + sessions = list(candidate.glob("*.jsonl")) + candidate = ( + max(sessions, key=lambda path: path.stat().st_mtime_ns) + if sessions + else None + ) + if candidate and candidate.is_file(): + try: + candidate.resolve().relative_to(store.runs.resolve()) + except (OSError, RuntimeError, ValueError): + candidate = None + if candidate and candidate.is_file(): + pi_resume_session = candidate + resume_locator = resume_locator_path + session_id = str(prior.get("session_id") or candidate.stem) + started_at = now_iso() + work_log_path = milestone_work_log_path(task) + record: dict[str, Any] = { + "execution_id": identity, + "task": task.name, + "task_directory": str(task.directory.resolve()), + "target_files": task_target_files(task), + "target_files_known": task.write_set_known, + "plan_number": plan_number(task), + "role": role, + "attempt": attempt, + "workspace": str(store.workspace), + "workspace_id": store.workspace_id, + **dispatcher_source_provenance(), + "cli": spec.cli, + "model": spec.model, + "reasoning_effort": effective_reasoning_effort(spec), + "agent_process_marker": process_marker, + "plan_path": str(task.plan) if task.plan else None, + "review_path": str(task.review) if task.review else None, + "session_id": session_id if spec.cli in {"claude", "pi"} else None, + "native_session_path": ( + str(pi_resume_session) + if pi_resume_session is not None + else native_session_path(spec.cli, workspace, session_id, attempt_dir) + ), + "output_log": str(stream_path), + "stream_log": str(stream_path), + "normalized_output_log": str(normalized_output_path), + "heartbeat_log": str(heartbeat_path), + "cli_log": str(attempt_dir / "agy-cli.log") if spec.cli == "agy" else None, + "work_log": str(work_log_path.resolve()), + "started_at": started_at, + "status": "running", + "resumed_from_locator": str(resume_locator) if pi_resume_session else None, + } + stage_decision = None + if isinstance(store, StateStore): + decisions = store.task_state(task).get("execution_decisions", {}) + if isinstance(decisions, dict): + stage_decision = decisions.get(role) + if isinstance(stage_decision, dict): + record.update(selector_runtime_evidence(stage_decision)) + try: + record["stage_budget"] = StageFailureBudget.from_decision(store, task, stage_decision).count() + except Exception: + record["stage_budget"] = 0 + # Resolve the retry handoff identity that was assigned when the pending + # quota refresh was created before the first durable locator write, so + # the first record on disk already carries the stable handoff ID a + # crash/restart can match against (the locator path changes on every + # attempt). + retry_handoff_id: str | None = None + if isinstance(store, StateStore): + retry_ctx = store.task_state(task).get("retry_quota_refresh_context") + if isinstance(retry_ctx, dict): + retry_handoff_id = retry_ctx.get("handoff_id") + if retry_handoff_id: + record["retry_handoff_id"] = retry_handoff_id + write_json(locator_path, record) + if isinstance(store, StateStore): + if retry_handoff_id: + # One-save transition: update active_locator, clear pending flag, + # and clear context together. Restore pre-state on save failure. + # A mismatch (False) or a save fault (raises) must stop before + # the provider process seam so we never launch a duplicate + # invocation against a retry intent we failed to commit. + if not store.commit_retry_handoff_locator(task, retry_handoff_id, str(locator_path)): + raise ExecutionDecisionError("retry handoff commit mismatch") + else: + store.update_task(task, active_locator=str(locator_path)) + prefix = f"[{task.directory.name}][{role}][a{attempt:02d}]" + + def persist_locator_record() -> None: + """Do not abort a live model solely because a locator refresh failed.""" + try: + write_json(locator_path, record) + except OSError as exc: + record["locator_write_error"] = str(exc) + attempt_event( + prefix, + f"locator ๊ธฐ๋ก ๊ฒฝ๊ณ : locator={locator_path} error={exc}", + ) + + for line in task_observation_lines(task): + attempt_event(prefix, line) + attempt_event(prefix, f"locator={locator_path}") + try: + append_milestone_event( + task, + event="START", + execution_id=identity, + role=role, + attempt=attempt, + model=spec.display, + result="running", + locator=locator_path, + ) + except OSError as exc: + line = f"milestone work log setup failed: {exc}" + heartbeat_path.write_text(line + "\n", encoding="utf-8") + record.update( + status="failed", + finished_at=now_iso(), + exit_code=1, + failure_class="work-log-setup", + failure_source="work-log", + provider_transport_failure_confirmed=False, + work_log_error=str(exc), + ) + persist_locator_record() + attempt_event(prefix, line) + return 1, "work-log-setup", locator_path + command = build_command( + spec, + prompt, + workspace, + session_id, + attempt_dir, + pi_resume_session=pi_resume_session, + ) + before_agy = agy_conversations() if spec.cli == "agy" else {} + diagnostics: list[str] = [] + diagnostic_origins: list[str] = [] + control_violation: str | None = None + try: + process = await asyncio.create_subprocess_exec( + *command, + cwd=workspace, + env={ + **os.environ, + AGENT_PROCESS_MARKER_ENV: process_marker, + }, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + limit=10 * 1024 * 1024, + start_new_session=True, + ) + # Keep the child PID in the locator before monitoring output. If this + # dispatcher is interrupted, a later dispatcher can distinguish a + # genuinely live, silent model from a stale locator and must not launch + # a duplicate continuation. + record["agent_pid"] = process.pid + record["agent_process_start_token"] = process_start_token(process.pid) + persist_locator_record() + except FileNotFoundError: + line = f"command not found: {command[0]}" + heartbeat_path.write_text(line + "\n", encoding="utf-8") + failure_class = "generic-error" + try: + append_milestone_event( + task, + event="FINISH", + execution_id=identity, + role=role, + attempt=attempt, + model=spec.display, + result=f"failed:{failure_class}:127", + locator=locator_path, + ) + except OSError as exc: + record["work_log_runtime_error"] = str(exc) + failure_class = "work-log-runtime-write" + record.update( + status="failed", + finished_at=now_iso(), + exit_code=127, + failure_class=failure_class, + failure_source=( + "work-log" if failure_class == "work-log-runtime-write" else "cli-launch" + ), + provider_transport_failure_confirmed=False, + ) + persist_locator_record() + attempt_event(prefix, line) + return 127, failure_class, locator_path + + readers: list[asyncio.Task[None]] = [] + try: + assert process.stdout is not None and process.stderr is not None + queue: asyncio.Queue[tuple[str, bytes | None]] = asyncio.Queue() + + async def pump(channel: str, stream: asyncio.StreamReader) -> None: + try: + while True: + raw = await stream.readline() + if not raw: + break + await queue.put((channel, raw)) + finally: + await queue.put((channel, None)) + + readers = [ + asyncio.create_task(pump("stdout", process.stdout)), + asyncio.create_task(pump("stderr", process.stderr)), + ] + finished_streams = 0 + loop = asyncio.get_running_loop() + last_native_mtime: int | None = None + last_stream_mtime: int | None = None + last_native_progress_at = loop.time() + last_stream_progress_at = loop.time() + with ( + stream_path.open("w", encoding="utf-8") as stream_log, + normalized_output_path.open("w", encoding="utf-8") as normalized_output_log, + heartbeat_path.open("a", encoding="utf-8") as heartbeat_log, + ): + while finished_streams < len(readers): + try: + channel, raw = await asyncio.wait_for( + queue.get(), timeout=STREAM_HEARTBEAT_SECONDS + ) + except asyncio.TimeoutError: + try: + stream_mtime = stream_path.stat().st_mtime_ns + except OSError: + stream_mtime = None + if stream_mtime is not None: + record["stream_log_mtime_ns"] = stream_mtime + if stream_mtime != last_stream_mtime: + last_stream_mtime = stream_mtime + last_stream_progress_at = loop.time() + record.pop("pi_silence_inspection", None) + native_path = ( + str(pi_resume_session) + if pi_resume_session is not None + else native_session_path( + spec.cli, + workspace, + record.get("session_id"), + attempt_dir, + ) + ) + if native_path: + record["native_session_path"] = native_path + native_mtime = native_session_mtime_ns( + record.get("native_session_path") + ) + if native_mtime is not None: + record["native_session_mtime_ns"] = native_mtime + if native_mtime != last_native_mtime: + last_native_mtime = native_mtime + last_native_progress_at = loop.time() + # Native events and the separately flushed stream log + # are peer progress signals. A trailing toolResult only + # selects the timeout budget; it never overrides later + # reasoning/text output. + record["pi_activity_state"] = "working" + pi_session_state = pi_native_session_state( + record.get("native_session_path") + ) + pi_phase = pi_session_state.phase + is_pi_tool_execution = pi_phase == "tool-running" + # Outside a toolCall->toolResult interval, model stdout/stderr + # is the liveness signal. A completed tool result changes phase + # but must not reset the model-response silence clock. + pi_inactive_seconds = loop.time() - ( + max(last_native_progress_at, last_stream_progress_at) + if is_pi_tool_execution + else last_stream_progress_at + ) + if spec.local_pi: + record["pi_session_phase"] = pi_phase + record["pi_session_phase_reason"] = ( + pi_session_state.reason + ) + record["pi_expected_tool_call_ids"] = list( + pi_session_state.expected_tool_call_ids + ) + record["pi_completed_tool_call_ids"] = list( + pi_session_state.completed_tool_call_ids + ) + record["pi_pending_tool_call_ids"] = list( + pi_session_state.pending_tool_call_ids + ) + record["pi_stall_timeout_seconds"] = None + record.setdefault("pi_activity_state", "starting") + if ( + spec.local_pi + and not is_pi_tool_execution + and pi_inactive_seconds >= PI_MODEL_RESPONSE_STALL_SECONDS + and "pi_silence_inspection" not in record + ): + inspection = { + "at": now_iso(), + "silence_seconds": round(pi_inactive_seconds, 3), + "stream_tail": log_tail_excerpt(stream_path), + } + record["pi_silence_inspection"] = inspection + diagnostic = ( + f"Pi {pi_phase} stream produced no update for " + f"{pi_inactive_seconds:.1f}s; recorded stream tail for inspection " + "without terminating the model process" + ) + heartbeat_log.write(f"[silence-inspection] {diagnostic}\n") + heartbeat_log.flush() + persist_locator_record() + attempt_event(prefix, f"๋ชจ๋ธ์‘๋‹ต์ ๊ฒ€: {diagnostic}") + non_pi_inactive_seconds = loop.time() - max( + last_native_progress_at, last_stream_progress_at + ) + if ( + not spec.local_pi + and non_pi_inactive_seconds + >= PI_MODEL_RESPONSE_STALL_SECONDS + and "stream_silence_inspection" not in record + ): + inspection = { + "at": now_iso(), + "silence_seconds": round(non_pi_inactive_seconds, 3), + "stream_tail": log_tail_excerpt(stream_path), + } + record["stream_silence_inspection"] = inspection + diagnostic = ( + f"{spec.cli} emitted no stream output or native-session event for " + f"{non_pi_inactive_seconds:.1f}s; recorded stream tail for inspection " + "without terminating the model process" + ) + heartbeat_log.write(f"[silence-inspection] {diagnostic}\n") + heartbeat_log.flush() + persist_locator_record() + attempt_event(prefix, f"๋ชจ๋ธ์‘๋‹ต์ ๊ฒ€: {diagnostic}") + heartbeat = ( + f"์ž‘์—…์ค‘... locator={locator_path} " + f"native_session={record.get('native_session_path') or 'none'} " + f"native_mtime_ns={record.get('native_session_mtime_ns', 'none')}" + ) + if spec.local_pi: + heartbeat += ( + f" pi_activity={record.get('pi_activity_state')}" + f" pi_phase={pi_phase}" + ) + heartbeat_log.write(f"[heartbeat] {heartbeat}\n") + heartbeat_log.flush() + persist_locator_record() + # Heartbeat is recovery state, not a user-visible lifecycle + # event. Keep it out of the caller-facing event stream. + continue + if raw is None: + finished_streams += 1 + continue + record.pop("pi_silence_inspection", None) + record.pop("stream_silence_inspection", None) + if spec.local_pi and channel == "stdout": + record["pi_activity_state"] = "streaming" + line = raw.decode("utf-8", errors="replace").rstrip("\n") + stream_log.write(f"[{channel}] {line}\n") + stream_log.flush() + diagnostic = terminal_diagnostic(spec.cli, channel, line) + if diagnostic: + diagnostics.append(diagnostic) + diagnostic_origins.append(f"{spec.cli}:{channel}") + if spec.cli == "codex" and role == "review" and channel == "stdout": + collaboration_tool = codex_collaboration_tool(line) + if collaboration_tool and control_violation is None: + control_violation = collaboration_tool + diagnostics.append( + f"official review invoked forbidden collaboration tool: " + f"{collaboration_tool}" + ) + diagnostic_origins.append("dispatcher:review-control") + attempt_event( + prefix, + f"๋ฆฌ๋ทฐ ์ œ์–ด ๊ณ„์•ฝ ์œ„๋ฐ˜: collaboration-tool=" + f"{collaboration_tool}", + ) + await terminate_process_group(process) + rendered, discovered = ( + render_json_line(spec.cli, line) if channel == "stdout" else ([line], None) + ) + if discovered and record.get("session_id") != discovered: + record["session_id"] = discovered + if pi_resume_session is None: + record["native_session_path"] = native_session_path( + spec.cli, workspace, discovered, attempt_dir + ) + persist_locator_record() + for display_line in rendered: + if display_line: + normalized_output_log.write(display_line + "\n") + normalized_output_log.flush() + # Child output is retained for recovery and review but is + # not itself a dispatcher lifecycle event. + await asyncio.gather(*readers) + return_code = await process.wait() + except asyncio.CancelledError: + for reader in readers: + reader.cancel() + if readers: + await asyncio.gather(*readers, return_exceptions=True) + await terminate_process_group(process) + runtime_error: OSError | None = None + try: + append_milestone_event( + task, + event="FINISH", + execution_id=identity, + role=role, + attempt=attempt, + model=spec.display, + result="failed:cancelled", + locator=locator_path, + ) + except OSError as exc: + runtime_error = exc + record.update( + status="failed", + finished_at=now_iso(), + exit_code="cancelled", + failure_class="cancelled", + failure_source="caller-cancel", + provider_transport_failure_confirmed=False, + ) + if runtime_error is not None: + record["work_log_runtime_error"] = str(runtime_error) + persist_locator_record() + raise + + if spec.cli == "agy": + after_agy = agy_conversations() + changed = [ + path for path, mtime in after_agy.items() + if path not in before_agy or before_agy[path] != mtime + ] + if changed: + selected = max(changed, key=lambda path: after_agy[path]) + record["session_id"] = selected.stem + record["native_session_path"] = str(selected) + agy_diagnostics = agy_log_diagnostics(attempt_dir / "agy-cli.log") + diagnostics.extend(agy_diagnostics) + diagnostic_origins.extend("agy:cli-log" for _ in agy_diagnostics) + native_path = ( + str(pi_resume_session) + if pi_resume_session is not None + else native_session_path( + spec.cli, workspace, record.get("session_id"), attempt_dir + ) + ) + if native_path: + record["native_session_path"] = native_path + native_mtime = native_session_mtime_ns(record.get("native_session_path")) + if native_mtime is not None: + record["native_session_mtime_ns"] = native_mtime + failure_source: str | None = None + failure_evidence: str | None = None + failure_evidence_source: str | None = None + provider_transport_failure_confirmed = False + termination = termination_signal(return_code) + if termination is not None: + record["termination_signal"] = termination[0] + record["termination_signal_inferred"] = termination[1] + if control_violation: + failure_class = "review-control-violation" + failure_source = "dispatcher-control" + failure_evidence_source = "dispatcher:review-control" + for index in range(len(diagnostics) - 1, -1, -1): + if diagnostic_origins[index] == failure_evidence_source: + failure_evidence = diagnostics[index] + break + elif return_code != 0 and termination is not None: + failure_class = "process-terminated" + failure_source = "process-termination" + record["termination_initiator"] = "unknown" + else: + classified_failure, classified_evidence = classify_failure_with_evidence( + "\n".join(diagnostics[-50:]) + ) + if return_code != 0 or classified_evidence is not None: + failure_class = classified_failure + failure_evidence = classified_evidence + else: + failure_class = None + if failure_class is not None and failure_evidence is not None: + for index in range(len(diagnostics) - 1, -1, -1): + if diagnostics[index] == failure_evidence: + failure_evidence_source = diagnostic_origins[index] + break + if failure_class in PROVIDER_TRANSPORT_FAILURES: + failure_source = "provider-terminal-diagnostic" + provider_transport_failure_confirmed = failure_evidence is not None + elif failure_evidence is not None: + failure_source = "cli-terminal-diagnostic" + elif return_code != 0: + failure_source = "cli-exit" + try: + append_milestone_event( + task, + event="FINISH", + execution_id=identity, + role=role, + attempt=attempt, + model=spec.display, + result=( + f"succeeded:0" + if return_code == 0 and failure_class is None + else f"failed:{failure_class or 'generic-error'}:{return_code}" + ), + locator=locator_path, + ) + except OSError as exc: + if failure_class is not None: + record["prior_failure_class"] = failure_class + record["work_log_runtime_error"] = str(exc) + failure_class = "work-log-runtime-write" + failure_source = "work-log" + failure_evidence = None + failure_evidence_source = None + provider_transport_failure_confirmed = False + if failure_evidence is not None: + record["failure_evidence_excerpt"] = failure_evidence[:FAILURE_EVIDENCE_LIMIT] + record["failure_evidence_truncated"] = ( + len(failure_evidence) > FAILURE_EVIDENCE_LIMIT + ) + if failure_evidence_source is not None: + record["failure_evidence_source"] = failure_evidence_source + record.update( + status="succeeded" if return_code == 0 and failure_class is None else "failed", + finished_at=now_iso(), + exit_code=return_code, + failure_class=failure_class, + failure_source=failure_source, + provider_transport_failure_confirmed=provider_transport_failure_confirmed, + ) + persist_locator_record() + return return_code, failure_class, locator_path + + +def dispatcher_child_prompt(body: str) -> str: + return f"{DISPATCHER_CHILD_BOUNDARY_PROMPT} {body}" + + +def selfcheck_prompt(task: Task, *, unchecked_items: bool = False) -> str: + if task.plan is None: + raise RuntimeError("selfcheck PLAN์ด ์—†๋‹ค") + if task.review is None: + raise RuntimeError("selfcheck CODE_REVIEW ํŒŒ์ผ์ด ์—†๋‹ค") + if unchecked_items: + body = ( + f"Read {task.plan.resolve()}; complete every unchecked implementation " + f"item and update {task.review.resolve()}. Keep files in English." + ) + else: + body = ( + f"Read {task.plan.resolve()}; review all work once, fix omissions, " + f"and update {task.review.resolve()}. Keep files in English." + ) + return f"{SELF_CHECK_PROMPT_PREFIX} {body}" + + +def base_prompt( + task: Task, + role: str, + spec: AgentSpec, + *, + unchecked_items: bool = False, +) -> str: + if role == "review": + target = task.review or task.directory + if task.review: + return dispatcher_child_prompt( + f"Read {target.resolve()} and start the review. Keep artifact " + "content in English. Final in Korean." + ) + return dispatcher_child_prompt( + f"Continue the review for {target.resolve()}. Keep artifact content " + "in English. Final in Korean." + ) + if task.plan is None: + raise RuntimeError("worker PLAN์ด ์—†๋‹ค") + target = task.plan.resolve() + if role == "selfcheck": + return selfcheck_prompt(task, unchecked_items=unchecked_items) + if spec.local_pi: + return dispatcher_child_prompt( + f"Think in English. Keep artifact content in English. Final in " + f"Korean. Read {target} and complete the task." + ) + return dispatcher_child_prompt( + f"Read {target} and complete the task. Keep artifact content in English. " + "Final in Korean." + ) + + + +def build_context_package( + workspace: Path, task: Task, locator: Path, *, previous_spec: AgentSpec, next_spec: AgentSpec +) -> dict[str, Any]: + """Build the fail-closed continuation context for a target transition.""" + + if not locator.is_file(): + raise ExecutionDecisionError("logical context locator๊ฐ€ ์—†๋‹ค") + try: + record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise ExecutionDecisionError("logical context locator๋ฅผ ์ฝ์„ ์ˆ˜ ์—†๋‹ค") from exc + if not isinstance(record, dict): + raise ExecutionDecisionError("logical context locator ํ˜•์‹์ด ์ž˜๋ชป๋๋‹ค") + workspace_value = record.get("workspace") + if not isinstance(workspace_value, str) or not workspace_value: + raise ExecutionDecisionError("logical context workspace๊ฐ€ ์—†๋‹ค") + recorded_workspace = Path(workspace_value) + if not recorded_workspace.is_absolute() or recorded_workspace.resolve() != workspace.resolve(): + raise ExecutionDecisionError("logical context workspace๊ฐ€ ์ผ์น˜ํ•˜์ง€ ์•Š๋Š”๋‹ค") + if record.get("task") != task.name: + raise ExecutionDecisionError("logical context task๊ฐ€ ์ผ์น˜ํ•˜์ง€ ์•Š๋Š”๋‹ค") + if task.plan is None or not task.plan.is_file(): + raise ExecutionDecisionError("logical context PLAN์ด ์—†๋‹ค") + plan_value = record.get("plan_path") + if not isinstance(plan_value, str) or not plan_value: + raise ExecutionDecisionError("logical context PLAN ๊ฒฝ๋กœ๊ฐ€ ์—†๋‹ค") + plan_path = Path(plan_value) + if not plan_path.is_absolute() or plan_path.resolve() != task.plan.resolve(): + raise ExecutionDecisionError("logical context PLAN ๊ฒฝ๋กœ๊ฐ€ ์ผ์น˜ํ•˜์ง€ ์•Š๋Š”๋‹ค") + required = { + "normalized_output": ("normalized_output_log", "normalized-output.log"), + "raw_log": ("stream_log", "stream.log"), + } + paths: dict[str, str] = {} + attempt_dir = locator.resolve().parent + for field, (record_field, filename) in required.items(): + value = record.get(record_field) + if not isinstance(value, str) or not value: + raise ExecutionDecisionError(f"logical context {field} artifact๊ฐ€ ์—†๋‹ค") + path = Path(value) + expected = attempt_dir / filename + if not path.is_absolute() or path.resolve() != expected or not expected.is_file(): + raise ExecutionDecisionError(f"logical context {field} artifact๊ฐ€ locator attempt์™€ ์ผ์น˜ํ•˜์ง€ ์•Š๋Š”๋‹ค") + paths[field] = str(expected) + same_pi = previous_spec.local_pi and next_spec.local_pi + package = { + "plan": str(task.plan.resolve()), "locator": str(locator.resolve()), + "workspace": str(workspace.resolve()), **paths, + "resume_mode": "native" if same_pi else "logical", + } + if same_pi: + native = Path(str(record.get("native_session_path", ""))) + if not native.is_file(): + raise ExecutionDecisionError("same-Pi logical context native session์ด ์—†๋‹ค") + package["native_session_path"] = str(native.resolve()) + return package + + +def canonical_selector_failover_route(decision: dict[str, Any] | None) -> bool: + if not isinstance(decision, dict): + return False + candidates = decision.get("candidates") + return isinstance(candidates, list) and len(candidates) > 1 + + +def logical_context_prompt(context: dict[str, Any]) -> str: + plan = context["plan"] + locator = context["locator"] + workspace = context["workspace"] + raw_log = context["raw_log"] + normalized_output = context["normalized_output"] + return dispatcher_child_prompt( + f"Think in English. Keep artifact content in English. Final in Korean. " + f"Read plan={plan}, locator={locator}, workspace={workspace}, " + f"raw_log={raw_log}, normalized_output={normalized_output} and complete the task." + ) + + +def continuation_prompt_from_package( + context_package: dict[str, Any], + *, + target: dict[str, Any] | None = None, + native_resume: bool = False, +) -> str: + if native_resume or context_package.get("resume_mode") == "native": + return dispatcher_child_prompt( + "Think in English. Keep artifact content in English. Final in Korean. Continue this session and complete " + "the current task." + ) + plan = context_package["plan"] + locator = context_package["locator"] + workspace = context_package["workspace"] + raw_log = context_package["raw_log"] + normalized_output = context_package["normalized_output"] + return dispatcher_child_prompt( + f"Think in English. Keep artifact content in English. Final in Korean. " + f"Read plan={plan}, locator={locator}, workspace={workspace}, " + f"raw_log={raw_log}, normalized_output={normalized_output} and complete the task." + ) + + +def continuation_prompt( + task: Task, + role: str, + locator: Path | None = None, + *, + local_pi: bool = False, + resume_same_pi_session: bool = False, + context: dict[str, Any] | None = None, + unchecked_items: bool = False, +) -> str: + if local_pi and role == "selfcheck": + if resume_same_pi_session: + if unchecked_items: + return selfcheck_prompt(task, unchecked_items=True) + return ( + f"{SELF_CHECK_PROMPT_PREFIX} Continue. Keep files in English." + ) + return selfcheck_prompt(task, unchecked_items=unchecked_items) + if context is not None: + return continuation_prompt_from_package( + context, + native_resume=resume_same_pi_session or context.get("resume_mode") == "native", + ) + if local_pi: + if resume_same_pi_session: + return dispatcher_child_prompt( + "Think in English. Keep artifact content in English. Final in Korean. Continue this session and complete " + "the current task." + ) + target = task.plan or task.directory + return dispatcher_child_prompt( + f"Think in English. Keep artifact content in English. Final in " + f"Korean. Read {target.resolve()} and complete the task." + ) + if role == "review": + return dispatcher_child_prompt( + f"Continue the review for {task.directory.resolve()}. Keep artifact " + "content in English. Final in Korean." + ) + return dispatcher_child_prompt( + f"Continue from {locator.resolve() if locator else task.directory.resolve()}. Check the saved context and current " + "workspace. Keep artifact content in English. Final in Korean." + ) + + +async def run_escalating( + workspace: Path, + store: StateStore, + task: Task, + role: str, + initial: AgentSpec, + initial_resume_locator: Path | None = None, + *, + unchecked_items: bool = False, +) -> tuple[bool, Path | None]: + spec = initial + previous_locator = initial_resume_locator + codex_recovery_count = 0 + codex_session_stall_retries = 0 + review_control_retries = 0 + pi_recovery_retries = 0 + generic_retries = 0 + terminal_recovery_retries = 0 + pi_resume_locator = initial_resume_locator + recovery_failures = 0 + stage_budget: StageFailureBudget | None = None + if isinstance(store, StateStore): + state = store.task_state(task) + persisted = state.get("recovery_failures", {}) + if isinstance(persisted, dict): + recovery_failures = int(persisted.get(role, 0)) + decisions = state.get("execution_decisions", {}) + decision = decisions.get(role) if isinstance(decisions, dict) and role in {"worker", "review"} else None + if isinstance(decision, dict): + stage_budget = StageFailureBudget.from_decision(store, task, decision) + recovery_failures = stage_budget.count() + legacy_recovery: LegacyPromotionRecovery | None = None + if initial_resume_locator is not None and isinstance(store, StateStore): + state = store.task_state(task) + legacy_recovery = legacy_promotion_recovery( + store.runs, + task, + state, + ) + if legacy_recovery is not None: + recovery_failures = 1 + persisted_failures = dict(state.get("recovery_failures", {})) + persisted_failures[legacy_recovery.role] = recovery_failures + store.update_task( + task, + blocked=None, + recovery_failures=persisted_failures, + legacy_terminal_reclassification={ + "role": legacy_recovery.role, + "failure_class": legacy_recovery.failure_class, + "evidence_source": legacy_recovery.evidence_source, + "prior_dispatcher_sha256": + legacy_recovery.prior_dispatcher_sha256, + "current_dispatcher_sha256": + DISPATCHER_SOURCE_SHA256, + "locator": str(legacy_recovery.locator), + "failed_cli": legacy_recovery.failed_cli, + "failed_model": legacy_recovery.failed_model, + "failed_reasoning_effort": + legacy_recovery.failed_reasoning_effort, + }, + ) + else: + legacy_recovery = persisted_legacy_promotion_recovery( + task, + state, + initial_resume_locator, + role, + ) + if legacy_recovery is not None and legacy_recovery.role == role: + failed_spec = failed_spec_from_recovery(legacy_recovery) + next_spec = promoted_spec(failed_spec, codex_recovery_count) + if next_spec is not None: + banner( + "๋ชจ๋ธ์Šน๊ฒฉ", + task.name, + [ + f"from={failed_spec.display}", + f"to={next_spec.display}", + f"failure_class={legacy_recovery.failure_class}", + "failure_source=legacy-terminal-reclassification", + f"failure_evidence_source={legacy_recovery.evidence_source}", + "dispatcher_source_sha256=" + f"{legacy_recovery.prior_dispatcher_sha256}", + f"dispatcher_source_current_sha256={DISPATCHER_SOURCE_SHA256}", + f"locator={legacy_recovery.locator}", + ], + ) + spec = next_spec + if recovery_failures >= RECOVERY_FAILURE_LIMIT: + locator = initial_resume_locator + reason = ( + f"{role} recovery failure limit already exhausted: " + f"{recovery_failures}/{RECOVERY_FAILURE_LIMIT}" + ) + if isinstance(store, StateStore): + decision = store.task_state(task).get("execution_decisions", {}).get(role, {}) + store.update_task( + task, + blocked=f"{reason} locator={locator}", + blocker_evidence={ + "role": role, + "failure_class": None, + "locator": str(locator) if locator else None, + "selected": decision.get("selected") if isinstance(decision, dict) else None, + "work_unit_id": decision.get("work_unit_id") if isinstance(decision, dict) else None, + }, + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=recovery-failure-limit", + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + f"locator={locator}", + ], + ) + return False, locator + context: dict[str, Any] | None = None + while True: + prompt = ( + base_prompt(task, role, spec, unchecked_items=unchecked_items) + if previous_locator is None + else continuation_prompt( + task, + role, + previous_locator, + local_pi=spec.local_pi, + resume_same_pi_session=pi_resume_locator is not None, + context=context, + unchecked_items=unchecked_items, + ) + ) + context = None + rc, failure, locator = await invoke( + workspace, + store, + task, + role, + spec, + prompt, + resume_locator=pi_resume_locator, + ) + pi_resume_locator = None + if rc == 0 and failure is None: + if isinstance(store, StateStore): + state = store.task_state(task) + persisted = dict(state.get("recovery_failures", {})) + persisted.pop(role, None) + store.update_task(task, recovery_failures=persisted) + if stage_budget is not None: + stage_budget.reset_on_success() + return True, locator + failure = failure or "generic-error" + if failure in { + "work-log-blocked", + "work-log-incomplete", + "work-log-setup", + "work-log-runtime-write", + }: + banner("์ž‘์—…์ฐจ๋‹จ", task.name, failure_report_lines(failure, locator)) + return False, locator + recovery_failures += 1 + if stage_budget is not None: + selected = stage_budget.store.task_state(task)["execution_decisions"][role]["selected"] + transition = stage_budget.store.task_state(task)["execution_decisions"][role]["transition"]["trigger"] + recovery_failures = stage_budget.record_failure(target=selected, transition=transition) + if isinstance(store, StateStore): + state = store.task_state(task) + persisted = dict(state.get("recovery_failures", {})) + persisted[role] = recovery_failures + store.update_task(task, recovery_failures=persisted) + if recovery_failures >= RECOVERY_FAILURE_LIMIT: + reason = ( + f"{role} recovery failure limit exhausted: " + f"{recovery_failures}/{RECOVERY_FAILURE_LIMIT}" + ) + if isinstance(store, StateStore): + decision = store.task_state(task).get("execution_decisions", {}).get(role, {}) + store.update_task( + task, + blocked=f"{reason} locator={locator}", + blocker_evidence={ + "role": role, + "failure_class": failure, + "locator": str(locator) if locator else None, + "selected": decision.get("selected") if isinstance(decision, dict) else None, + "work_unit_id": decision.get("work_unit_id") if isinstance(decision, dict) else None, + }, + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=recovery-failure-limit", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + return False, locator + if role == "review" and failure == "review-control-violation": + review_control_retries += 1 + banner( + "๋ฆฌ๋ทฐ์žฌ์‹œ๋„", + task.name, + [ + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = None + await asyncio.sleep(min(30, 2 ** min(review_control_retries, 5))) + continue + current_decision = None + quota_snapshot = None + if isinstance(store, StateStore): + task_state = store.task_state(task) + decisions = task_state.get("execution_decisions", {}) + if isinstance(decisions, dict): + current_decision = decisions.get(role) + quota_snapshot = task_state.get("quota_snapshot") + + if canonical_selector_failover_route(current_decision) and failure in QUALIFIED_FAILOVER_FAILURES: + try: + if failure == "provider-quota": + derived = derive_work_unit_quota_evidence( + current_decision, + status="exhausted", + reason="confirmed_runtime_provider_quota", + ) + quota_snapshot = derived + next_decision = select_execution_decision( + task, + stage=role, + prior_decision=current_decision, + quota_snapshot=quota_snapshot, + transition="failover", + failure_class=failure, + ) + next_spec = agent_spec_from_decision(next_decision) + if next_spec != spec: + if locator is None: + raise ExecutionDecisionError("logical context locator๊ฐ€ ์—†๋‹ค") + context = build_context_package( + workspace, task, locator, previous_spec=spec, next_spec=next_spec + ) + commit_execution_decision(store, task, role, next_decision) + banner( + "๋ชจ๋ธ์Šน๊ฒฉ" if role == "worker" else "๋ฆฌ๋ทฐ์Šน๊ฒฉ", + task.name, + [ + f"from={spec.display}", + f"to={next_spec.display}", + *failure_report_lines(failure, locator), + ], + ) + spec = next_spec + previous_locator = locator + continue + else: + commit_execution_decision(store, task, role, next_decision) + except (ExecutionDecisionError, OSError, ValueError) as exc: + code = getattr(exc, "code", "") + if not code: + if "no_failover_candidate" in str(exc): + code = "no_failover_candidate" + else: + code = exc.__class__.__name__ + store.update_task( + task, blocked=f"{role} selector decision ์‹คํŒจ [{code}]: {exc}" + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [f"reason={code}", *failure_report_lines(failure, locator)], + ) + return False, locator + if spec.local_pi: + if ( + spec.model.startswith("laguna-s") + and failure in {"context-limit", "session-stall"} + ): + pi_recovery_retries += 1 + banner( + "Pi์„ธ์…˜์—ฐ์†์žฌ์‹œ์ž‘", + task.name, + [ + f"model={spec.display}", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + pi_resume_locator = locator + await asyncio.sleep(min(30, 2 ** min(pi_recovery_retries, 5))) + continue + pi_recovery_retries += 1 + if failure == "session-stall": + event = "์„ธ์…˜์‘๋‹ต๋ณต๊ตฌ์žฌ์‹œ๋„" + elif failure in { + "provider-connection", + "provider-stream-disconnect", + }: + event = "์„ธ์…˜์—ฐ๊ฒฐ์žฌ์‹œ๋„" + else: + event = "Pi๋ณต๊ตฌ์žฌ์‹œ๋„" + banner( + event, + task.name, + [ + f"model={spec.display}", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + await asyncio.sleep(min(30, 2 ** min(pi_recovery_retries, 5))) + continue + if spec.cli == "codex" and failure == "session-stall": + codex_session_stall_retries += 1 + banner( + "์„ธ์…˜์‘๋‹ต๋ณต๊ตฌ์žฌ์‹œ๋„", + task.name, + [ + f"model={spec.display}", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + await asyncio.sleep(min(30, 2 ** min(codex_session_stall_retries, 5))) + continue + if failure == "generic-error": + generic_retries += 1 + banner( + "์ž‘์—…๋ณต๊ตฌ์žฌ์‹œ๋„", + task.name, + [ + f"model={spec.display}", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + await asyncio.sleep(min(30, 2 ** min(generic_retries, 5))) + continue + if failure not in CLOUD_PROMOTION_FAILURES: + terminal_recovery_retries += 1 + banner( + "๋ชจ๋ธ๋ณต๊ตฌ์žฌ์‹œ๋„", + task.name, + [ + f"model={spec.display}", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + await asyncio.sleep( + min(30, 2 ** min(terminal_recovery_retries, 5)) + ) + continue + if role == "review": + terminal_recovery_retries += 1 + banner( + "๋ฆฌ๋ทฐ์žฌ์‹œ๋„", + task.name, + [ + f"model={spec.display}", + "reason=official-review-fixed-target", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + await asyncio.sleep( + min(30, 2 ** min(terminal_recovery_retries, 5)) + ) + continue + if ( + isinstance(store, StateStore) + and role == "worker" + and isinstance(current_decision, dict) + ): + try: + next_decision = select_execution_decision( + task, + stage=role, + prior_decision=current_decision, + quota_snapshot=quota_snapshot, + transition="promotion", + failure_class=failure, + ) + except ExecutionDecisionError as exc: + if "no_promotion_target" in str(exc): + # canonical promotion ๋Œ€์ƒ ์—†์Œ: selector-backed worker๋Š” + # ํ˜„์žฌ target recovery/budget exhaustion ๋˜๋Š” block์œผ๋กœ๋งŒ ์ข…๊ฒฐ. + # legacy promoted_spec()์œผ๋กœ์˜ fallthrough ๊ธˆ์ง€. + banner( + "๋ชจ๋ธ์žฌ์‹œ๋„", + task.name, + [ + f"model={spec.display}", + "reason=no-promotion-target", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + terminal_recovery_retries += 1 + await asyncio.sleep( + min(30, 2 ** min(terminal_recovery_retries, 5)) + ) + continue + store.update_task( + task, + blocked=f"{role} selector promotion ์‹คํŒจ: {exc}", + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=selector-promotion", + *failure_report_lines(failure, locator), + ], + ) + return False, locator + else: + next_spec = agent_spec_from_decision(next_decision) + if next_spec == spec: + store.update_task( + task, + blocked="selector promotion์ด ํ˜„์žฌ target์„ ๋‹ค์‹œ ์„ ํƒํ–ˆ๋‹ค", + ) + return False, locator + if locator is None: + store.update_task( + task, blocked="logical context locator๊ฐ€ ์—†๋‹ค" + ) + return False, locator + try: + context = build_context_package( + workspace, + task, + locator, + previous_spec=spec, + next_spec=next_spec, + ) + except ExecutionDecisionError as exc: + store.update_task(task, blocked=str(exc)) + return False, locator + commit_execution_decision(store, task, role, next_decision) + banner( + "๋ชจ๋ธ์Šน๊ฒฉ", + task.name, + [ + f"from={spec.display}", + f"to={next_spec.display}", + *failure_report_lines(failure, locator), + ], + ) + spec = next_spec + previous_locator = locator + continue + next_spec = promoted_spec(spec, codex_recovery_count) + if next_spec is None: + terminal_recovery_retries += 1 + banner( + "๋ชจ๋ธ๋ณต๊ตฌ์žฌ์‹œ๋„", + task.name, + [ + f"model={spec.display}", + *failure_report_lines(failure, locator), + f"retry={recovery_failures}/{RECOVERY_FAILURE_LIMIT}", + ], + ) + previous_locator = locator + await asyncio.sleep(min(30, 2 ** min(terminal_recovery_retries, 5))) + continue + if spec.cli == "codex": + codex_recovery_count += 1 + banner( + "๋ชจ๋ธ์Šน๊ฒฉ", + task.name, + [ + f"from={spec.display}", + f"to={next_spec.display}", + *failure_report_lines(failure, locator), + ], + ) + spec = next_spec + previous_locator = locator + + +def task_signature(workspace: Path, task: Task) -> str: + digest = hashlib.sha256() + if not task.directory.exists(): + return "moved" + for path in sorted(p for p in task.directory.iterdir() if p.is_file()): + if ( + PLAN_RE.match(path.name) + or REVIEW_RE.match(path.name) + or path.name.endswith(".log") + ): + digest.update(path.name.encode()) + digest.update(sha256_file(path).encode()) + for raw_path in sorted(task.write_set): + path = Path(raw_path) + path = path if path.is_absolute() else workspace / path + digest.update(raw_path.encode()) + if path.is_file(): + digest.update(str(path.stat().st_mode).encode()) + digest.update(sha256_file(path).encode()) + elif path.exists(): + digest.update(b"non-file") + else: + digest.update(b"missing") + return digest.hexdigest() + + +def read_verdict(path: Path) -> str | None: + if not path.exists(): + return None + return verdict_from_text(path.read_text(encoding="utf-8", errors="replace")) + + +def verdict_from_text(text: str) -> str | None: + selected: tuple[re.Match[str], re.Pattern[str], re.Pattern[str]] | None = None + for heading_re, line_re, block_re in VERDICT_SCHEMA_MATCHERS: + headings = list(heading_re.finditer(text)) + if not headings: + continue + # A duplicated heading, or headings from both schemas, is ambiguous. + if len(headings) != 1 or selected is not None: + return None + selected = (headings[0], line_re, block_re) + if selected is None: + return None + heading, line_re, block_re = selected + next_heading = re.search(r"^##\s+", text[heading.end():], re.MULTILINE) + end = heading.end() + next_heading.start() if next_heading else len(text) + section = text[heading.end():end] + inline_matches = list(line_re.finditer(section)) + block_matches = list(block_re.finditer(section)) + matches = inline_matches + block_matches + return matches[0].group(1) if len(matches) == 1 else None + + +def matching_archive_directories_by_name( + workspace: Path, + task_name: str, + *, + require_complete: bool = True, +) -> list[Path]: + archive = workspace / "agent-task" / "archive" + parts = task_name.split("/") + if not archive.is_dir() or len(parts) not in {1, 2}: + return [] + group = parts[0] + final_name = parts[-1] + suffix_re = re.compile(rf"^{re.escape(final_name)}(?:_\d+)?$") + matches: list[Path] = [] + try: + years = list(archive.iterdir()) + except FileNotFoundError: + return [] + for year in years: + if not year.is_dir(): + continue + try: + months = list(year.iterdir()) + except FileNotFoundError: + continue + for month in months: + if not month.is_dir(): + continue + parent = month if len(parts) == 1 else month / group + if not parent.is_dir(): + continue + try: + candidates = list(parent.iterdir()) + except FileNotFoundError: + continue + for candidate in candidates: + if ( + candidate.is_dir() + and suffix_re.match(candidate.name) + and ( + not require_complete + or (candidate / "complete.log").is_file() + ) + ): + matches.append(candidate) + return sorted(matches) + + +def matching_archive_directories(workspace: Path, task: Task) -> list[Path]: + return matching_archive_directories_by_name(workspace, task.name) + + +def task_group_name(task_name: str) -> str: + return task_name.split("/", 1)[0] + + +def work_log_event_cells(line: str) -> list[str] | None: + stripped = line.strip() + if not stripped.startswith("|") or not stripped.endswith("|"): + return None + cells = [ + cell.strip().replace(r"\|", "|") + for cell in re.split(r"(? str: + """Merge duplicate dispatcher timelines without losing conflicting rows.""" + allowed_metadata = { + "# Milestone Work Log", + "## Dispatcher Timeline", + "> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.", + "> Dispatcher-owned. Workers and reviewers do not edit this section.", + WORK_LOG_HEADER, + WORK_LOG_SEPARATOR, + LEGACY_WORK_LOG_HEADER, + LEGACY_WORK_LOG_SEPARATOR, + } + unique_rows: dict[tuple[str, ...], tuple[str, ...]] = {} + ordered_rows: list[tuple[str, int, int, list[str]]] = [] + + for source_index, source in enumerate(sorted(sources)): + try: + lines = source.read_text( + encoding="utf-8", + errors="replace", + ).splitlines() + except OSError as exc: + raise ValueError( + f"WORK_LOG source๋ฅผ ์ฝ์„ ์ˆ˜ ์—†๋‹ค: source={source} error={exc}" + ) from exc + for line_number, line in enumerate(lines, start=1): + cells = work_log_event_cells(line) + if cells is None: + if not line.strip() or line.strip() in allowed_metadata: + continue + raise ValueError( + "WORK_LOG ๋ณ‘ํ•ฉ ๋Œ€์ƒ์— ์•ˆ์ „ํ•˜๊ฒŒ ๋ณด์กดํ•  ์ˆ˜ ์—†๋Š” ๋‚ด์šฉ์ด ์žˆ๋‹ค: " + f"source={source} line={line_number}" + ) + if cells[0] == "seq" or cells[0].startswith("---"): + continue + if cells[2] not in {"START", "FINISH"}: + raise ValueError( + "WORK_LOG ๋ณ‘ํ•ฉ ๋Œ€์ƒ์— ์ง€์›ํ•˜์ง€ ์•Š๋Š” event๊ฐ€ ์žˆ๋‹ค: " + f"source={source} line={line_number} event={cells[2]}" + ) + try: + sequence = int(cells[0]) + int(cells[4]) + int(cells[6]) + except ValueError as exc: + raise ValueError( + "WORK_LOG ๋ณ‘ํ•ฉ ๋Œ€์ƒ์˜ sequence, loop ๋˜๋Š” attempt๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: " + f"source={source} line={line_number}" + ) from exc + key = (cells[2], cells[3], cells[4], cells[5], cells[6], cells[9]) + fingerprint = tuple(cells[1:]) + previous = unique_rows.get(key) + if previous is not None: + if previous != fingerprint: + raise ValueError( + "WORK_LOG ๋ณ‘ํ•ฉ ์ถฉ๋Œ: " + f"event={cells[2]} task={cells[3]} loop={cells[4]} " + f"role={cells[5]} attempt={cells[6]} locator={cells[9]}" + ) + continue + unique_rows[key] = fingerprint + ordered_rows.append((cells[1], source_index, sequence, cells)) + + if not ordered_rows: + raise ValueError("WORK_LOG ๋ณ‘ํ•ฉ ๋Œ€์ƒ์— timeline row๊ฐ€ ์—†๋‹ค") + + header = ( + "# Milestone Work Log\n\n" + "> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n" + f"{WORK_LOG_HEADER}\n" + f"{WORK_LOG_SEPARATOR}\n" + ) + rendered_rows: list[str] = [] + for sequence, (_, _, _, cells) in enumerate(sorted(ordered_rows), start=1): + escaped = [str(value).replace("|", r"\|").replace("\n", " ") for value in cells] + escaped[0] = str(sequence) + rendered_rows.append("| " + " | ".join(escaped) + " |\n") + return header + "".join(rendered_rows) + + +def unfinished_work_log_attempts(path: Path) -> list[dict[str, Any]]: + """Return START rows that have no matching FINISH row.""" + try: + lines = path.read_text( + encoding="utf-8", + errors="replace", + ).splitlines() + except OSError: + raise + open_attempts: dict[str, dict[str, Any]] = {} + for line in lines: + cells = work_log_event_cells(line) + if cells is None or cells[2] not in {"START", "FINISH"}: + continue + try: + sequence = int(cells[0]) + loop = int(cells[4]) + attempt = int(cells[6]) + except ValueError: + continue + locator = cells[9] + key = locator or "\0".join( + (cells[3], cells[4], cells[5], cells[6], cells[7]) + ) + if cells[2] == "START": + open_attempts[key] = { + "sequence": sequence, + "task_name": cells[3], + "loop": loop, + "role": cells[5], + "attempt": attempt, + "model": cells[7], + "locator": locator, + } + else: + open_attempts.pop(key, None) + return sorted( + open_attempts.values(), + key=lambda record: int(record["sequence"]), + ) + + +def close_unfinished_work_log_attempts(path: Path) -> int: + """Close orphaned START rows after verified group completion.""" + unfinished = unfinished_work_log_attempts(path) + for record in unfinished: + locator = Path(str(record["locator"])) + append_work_log_event( + path, + task_name=str(record["task_name"]), + loop=int(record["loop"]), + event="FINISH", + execution_id=f"reconciled-{record['sequence']}", + role=str(record["role"]), + attempt=int(record["attempt"]), + model=str(record["model"]), + result="reconciled:verified-complete-archive", + locator=locator, + ) + return len(unfinished) + + +def archived_task_group_directories( + workspace: Path, + task_group: str, +) -> list[Path]: + """Return month-local archive directories for one logical task group.""" + archive_root = workspace / "agent-task" / "archive" + if not archive_root.is_dir(): + return [] + suffix_re = re.compile(rf"^{re.escape(task_group)}(?:_\d+)?$") + matches: list[Path] = [] + try: + years = list(archive_root.iterdir()) + except FileNotFoundError: + return [] + for year in years: + if not year.is_dir(): + continue + try: + months = list(year.iterdir()) + except FileNotFoundError: + continue + for month in months: + if not month.is_dir(): + continue + try: + candidates = list(month.iterdir()) + except FileNotFoundError: + continue + matches.extend( + candidate + for candidate in candidates + if candidate.is_dir() and suffix_re.fullmatch(candidate.name) + ) + return sorted(matches) + + +def next_work_log_archive_number( + workspace: Path, + task_group: str, +) -> int: + numbers = [ + int(match.group(1)) + for directory in archived_task_group_directories(workspace, task_group) + for path in directory.glob("work_log_*.log") + if (match := WORK_LOG_ARCHIVE_RE.fullmatch(path.name)) + ] + return max(numbers, default=-1) + 1 + + +def completed_group_archive_directory( + task_group: str, + task_names: set[str], + completed_tasks: dict[str, str], +) -> Path | None: + candidates: list[tuple[int, str, Path]] = [] + for task_name in task_names: + archive_raw = completed_tasks.get(task_name) + if not archive_raw: + continue + archive = Path(archive_raw) + if not archive.is_dir(): + continue + complete_log = archive / "complete.log" + target = archive if task_name == task_group else archive.parent + try: + completed_at = ( + complete_log.stat().st_mtime_ns + if complete_log.is_file() + else archive.stat().st_mtime_ns + ) + except OSError: + continue + candidates.append((completed_at, str(target), target)) + return ( + max(candidates, key=lambda item: (item[0], item[1]))[2] + if candidates + else None + ) + + +def archive_completed_group_work_logs( + workspace: Path, + observed_tasks: set[str], + completed_tasks: dict[str, str], + active_or_running: set[str], +) -> tuple[dict[str, str], dict[str, str]]: + """Archive each completed task-group timeline after its last writer exits.""" + observed_by_group: dict[str, set[str]] = {} + for task_name in observed_tasks: + observed_by_group.setdefault(task_group_name(task_name), set()).add( + task_name + ) + active_groups = { + task_group_name(task_name) + for task_name in active_or_running + } + archived: dict[str, str] = {} + errors: dict[str, str] = {} + for task_group, task_names in sorted(observed_by_group.items()): + if task_group in active_groups or not task_names <= set(completed_tasks): + continue + active_source = ( + workspace / "agent-task" / task_group / WORK_LOG_NAME + ) + legacy_sources = { + Path(completed_tasks[task_name]) / WORK_LOG_NAME + for task_name in task_names + if (Path(completed_tasks[task_name]) / WORK_LOG_NAME).is_file() + } + sources = { + *legacy_sources, + *([active_source] if active_source.is_file() else []), + } + if not sources: + continue + target_directory = completed_group_archive_directory( + task_group, + task_names, + completed_tasks, + ) + if target_directory is None: + errors[task_group] = ( + "๊ฒ€์ฆ๋œ task archive์—์„œ WORK_LOG ๋Œ€์ƒ ๋””๋ ‰ํ„ฐ๋ฆฌ๋ฅผ ์ •ํ•  ์ˆ˜ ์—†๋‹ค" + ) + continue + archive_number = next_work_log_archive_number( + workspace, + task_group, + ) + destination = target_directory / f"work_log_{archive_number}.log" + source = next(iter(sources)) + if destination.exists(): + errors[task_group] = ( + "WORK_LOG archive destination์ด ์ด๋ฏธ ์กด์žฌํ•œ๋‹ค: " + "sources=" + + ",".join(str(path) for path in sorted(sources)) + + f" destination={destination}" + ) + continue + source = next(iter(sources)) + temporary = destination.with_name(destination.name + ".tmp") + try: + if len(sources) == 1: + close_unfinished_work_log_attempts(source) + source.replace(destination) + else: + if temporary.exists(): + raise OSError( + "WORK_LOG archive temporary destination์ด ์ด๋ฏธ ์กด์žฌํ•œ๋‹ค: " + f"temporary={temporary}" + ) + temporary.write_text( + merge_work_log_sources(sources), + encoding="utf-8", + ) + close_unfinished_work_log_attempts(temporary) + temporary.replace(destination) + for merged_source in sources: + merged_source.unlink() + except (OSError, ValueError) as exc: + if temporary.exists(): + try: + temporary.unlink() + except OSError: + pass + errors[task_group] = ( + "WORK_LOG archive ์‹คํŒจ: sources=" + + ",".join(str(path) for path in sorted(sources)) + + " " + f"destination={destination} error={exc}" + ) + continue + if active_source in sources: + for task_name in sorted(task_names, reverse=True): + if "/" not in task_name: + continue + try: + (workspace / "agent-task" / task_name).rmdir() + except OSError: + pass + try: + active_source.parent.rmdir() + except OSError: + pass + archived[task_group] = str(destination.resolve()) + return archived, errors + + +def task_attempt_log_directories(runs: Path, task_name: str) -> list[Path]: + """Return dispatcher-owned attempt directories whose locator names the task.""" + matches: list[Path] = [] + if not runs.is_dir(): + return matches + for attempt_dir in runs.iterdir(): + if not attempt_dir.is_dir(): + continue + locator = attempt_dir / "locator.json" + try: + record = json.loads(locator.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + if record.get("task") != task_name: + continue + matches.append(attempt_dir) + return matches + + +def cleanup_completed_task_attempt_logs(runs: Path, task_name: str) -> int: + """Remove only dispatcher-owned logs for a task after its complete archive exists.""" + removed = 0 + for attempt_dir in task_attempt_log_directories(runs, task_name): + try: + shutil.rmtree(attempt_dir) + except OSError as exc: + attempt_event( + "[attempt-log-cleanup-warning]", + f"task={task_name} path={attempt_dir} error={exc}", + ) + continue + removed += 1 + return removed + + +def review_fingerprints(workspace: Path, task: Task) -> set[tuple[str, str]]: + directories = [task.directory] if task.directory.is_dir() else [] + directories.extend(matching_archive_directories(workspace, task)) + fingerprints: set[tuple[str, str]] = set() + for directory in directories: + for path in directory.iterdir(): + if path.is_file() and (path.name.startswith("code_review_") or REVIEW_RE.match(path.name)): + fingerprints.add((str(path.resolve()), sha256_file(path))) + return fingerprints + + +def review_outcome( + workspace: Path, task: Task, prior_fingerprints: set[tuple[str, str]] +) -> dict[str, str]: + if task.directory.is_dir(): + directories = [task.directory] + archives: list[Path] = [] + else: + archives = matching_archive_directories(workspace, task) + directories = list(archives) + newest_directory: Path | None = None + newest_log: Path | None = None + newest_mtime = -1 + for directory in directories: + logs = list(directory.glob("code_review_*.log")) + logs.extend(path for path in directory.iterdir() if path.is_file() and REVIEW_RE.match(path.name)) + for log in logs: + mtime = log.stat().st_mtime_ns + fingerprint = (str(log.resolve()), sha256_file(log)) + if fingerprint not in prior_fingerprints and mtime > newest_mtime and read_verdict(log): + newest_directory = directory + newest_log = log + newest_mtime = mtime + verdict = read_verdict(newest_log) if newest_log else "UNKNOWN" + if newest_directory in archives: + state = "archived" + elif newest_directory and (newest_directory / "USER_REVIEW.md").exists(): + state = "user-review" + elif newest_directory and (newest_directory / "complete.log").exists(): + state = "complete-finalization" + elif newest_log and REVIEW_RE.match(newest_log.name): + state = "finalization-pending" + elif newest_directory and any(REVIEW_RE.match(path.name) for path in newest_directory.iterdir() if path.is_file()): + state = "follow-up" + else: + state = "changed" + return { + "verdict": verdict or "UNKNOWN", + "state": state, + "path": str(newest_directory or task.directory), + "review_log": str(newest_log) if newest_log else "unknown", + } + + +async def run_worker( + workspace: Path, + store: StateStore, + task: Task, + resume_locator: Path | None = None, + quota_snapshot: dict[str, Any] | None = None, +) -> None: + retry_context = store.task_state(task).get("retry_quota_refresh_context") + if resume_locator is None and isinstance(retry_context, dict): + locator_value = retry_context.get("locator") + if isinstance(locator_value, str) and locator_value: + resume_locator = Path(locator_value) + + # If an active_locator already exists from a prior attempt that wrote its + # locator but crashed before consuming the pending handoff, consume it now + # to prevent a duplicate invocation. The locator write is the durable + # commitment; the pending handoff is the logical intent. Consume the intent + # when the commitment is already present. + if isinstance(store, StateStore): + prior_state = store.task_state(task) + prior_active = prior_state.get("active_locator") + prior_pending = prior_state.get("retry_quota_refresh_pending") + if prior_active and prior_pending: + consumed = False + # Prefer handoff_id matching: read the stable identity from the + # active locator file so we can match across crash boundaries + # where the locator path changes. + try: + prior_locator_data = json.loads(Path(prior_active).read_text(encoding="utf-8")) + if isinstance(prior_locator_data, dict): + handoff_id = prior_locator_data.get("retry_handoff_id") + if handoff_id: + consumed = store.commit_retry_handoff_locator( + task, handoff_id, prior_active, + ) + except (OSError, json.JSONDecodeError): + pass + if not consumed: + # Fallback: match by locator path when handoff_id is + # unavailable (e.g. crash between state update and locator + # write, or pre-existing state from a prior dispatcher + # version). + consumed = store.consume_matching_retry_handoff(task, prior_active) + + try: + decision, spec = persisted_execution_decision( + store, task, stage="worker", quota_snapshot=quota_snapshot + ) + except ExecutionDecisionError as exc: + store.update_task(task, blocked=str(exc)) + banner("์ž‘์—…์ฐจ๋‹จ", task.name, [f"reason={exc}"]) + return + work_log = milestone_work_log_path(task) + banner( + "์ž‘์—…์‹œ์ž‘", + task.name, + [ + f"model={spec.display}", + f"plan={task.plan.resolve()}", + f"work_log={work_log.resolve()}", + *task_observation_lines(task), + ], + ) + success, locator = await run_escalating( + workspace, + store, + task, + "worker", + spec, + initial_resume_locator=resume_locator, + ) + if not success: + current = store.task_state(task).get("blocked") + store.update_task( + task, blocked=current or f"worker failure locator={locator}" + ) + return + completed_spec = agent_spec_from_locator(locator) or spec + try: + _mark_worker_done( + store, + task, + initial_decision=decision, + worker_cli=completed_spec.cli, + worker_model=completed_spec.model, + ) + except ExecutionDecisionError as exc: + store.update_task(task, worker_done=False, blocked=f"worker completion validation failed: {exc}") + banner("์ž‘์—…์ฐจ๋‹จ", task.name, [f"reason={exc}"]) + return + + +def _require_same_runtime_identity( + expected_spec: AgentSpec, + worker_cli: str, + worker_model: str, +) -> None: + """Verify the worker CLI/model matches the validated completing decision spec. + + Ensures the actual worker that ran is the same runtime identity that the + completing decision authorizes. Prevents a cloud-completed worker from + being recorded as a Pi selfcheck target or vice versa. + """ + if expected_spec.cli != worker_cli: + raise ExecutionDecisionError( + f"worker runtime CLI ๋ถˆ์ผ์น˜: expected={expected_spec.cli} actual={worker_cli}" + ) + if expected_spec.model != worker_model: + raise ExecutionDecisionError( + f"worker runtime model ๋ถˆ์ผ์น˜: expected={expected_spec.model} actual={worker_model}" + ) + + +def _mark_worker_done( + store: StateStore, + task: Task, + *, + initial_decision: dict[str, Any], + worker_cli: str, + worker_model: str, +) -> None: + """Persist worker completion with the authoritative completing decision. + + Uses the persisted execution_decisions worker entry as the sole authoritative + source. Does not fall back to initial_decision even when the persisted + decision is malformedโ€”malformed persisted state blocks completion rather + than silently reverting to a speculative initial decision. + + Validates the completing decision through the strict contract validator + and verifies the worker CLI/model identity matches the normalized spec. + On any validation failure, raises ExecutionDecisionError to prevent + worker_done from being recorded. + """ + decisions = store.task_state(task).get("execution_decisions", {}) + if not isinstance(decisions, dict) or "worker" not in decisions: + raise ExecutionDecisionError( + "persisted worker decision์ด execution_decisions์— ์—†๋‹ค" + ) + decision = decisions["worker"] + if not isinstance(decision, dict): + raise ExecutionDecisionError( + "persisted worker decision์ด dict๊ฐ€ ์•„๋‹ˆ๋‹ค" + ) + validated_decision, expected_spec = _validated_completing_decision( + task, decision + ) + _require_same_runtime_identity(expected_spec, worker_cli, worker_model) + execution_class = validated_decision["selected"]["execution_class"] + store.update_task( + task, + worker_done=True, + worker_cli=worker_cli, + worker_model=worker_model, + completing_decision=validated_decision, + execution_class=execution_class, + selfcheck_done=(execution_class == "cloud_model"), + blocked=None, + ) + + +async def run_selfcheck( + workspace: Path, + store: StateStore, + task: Task, + resume_locator: Path | None = None, +) -> None: + completing = store.task_state(task).get("completing_decision") + if not isinstance(completing, dict): + store.update_task( + task, blocked="completing decision์ด ์—†์–ด selfcheck๋ฅผ ์‹คํ–‰ํ•  ์ˆ˜ ์—†๋‹ค" + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", task.name, + ["reason=missing-completing-decision"], + ) + return + try: + _completed_decision, spec = _validated_completing_decision(task, completing) + except ExecutionDecisionError as exc: + store.update_task(task, blocked=str(exc)) + banner("์ž‘์—…์ฐจ๋‹จ", task.name, [f"reason={exc}"]) + return + if not spec.local_pi: + raise RuntimeError("Pi๊ฐ€ ์•„๋‹Œ route์— selfcheck stage๊ฐ€ ๋ฐฐ์ •๋๋‹ค") + work_log = milestone_work_log_path(task) + banner( + "์ž๊ฐ€๊ฒ€์ฆ์‹œ์ž‘", + task.name, + [ + f"model={spec.display}", + f"plan={task.plan.resolve()}", + f"work_log={work_log.resolve()}", + *task_observation_lines(task), + ], + ) + # 0 means the full pass is pending. After it fails the checklist gate, + # each additional count represents one completed unchecked-item retry. + incomplete_results = 0 + if isinstance(store, StateStore): + incomplete_results = int( + store.task_state(task).get("selfcheck_incomplete", 0) + ) + incomplete_retries = max(0, incomplete_results - 1) + if incomplete_retries >= SELF_CHECK_UNCHECKED_RETRY_LIMIT: + locator = resume_locator + reason = ( + "selfcheck unchecked-item retry limit already exhausted: " + f"{incomplete_retries}/{SELF_CHECK_UNCHECKED_RETRY_LIMIT}" + ) + store.update_task(task, blocked=f"{reason} locator={locator}") + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=selfcheck-incomplete-limit", + "mode=unchecked-items", + f"retry={incomplete_retries}/{SELF_CHECK_UNCHECKED_RETRY_LIMIT}", + f"locator={locator}", + ], + ) + return + if incomplete_results > 0 and resume_locator is None: + resume_locator, context_error = selfcheck_context_resume_locator( + store.task_state(task), + task, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if resume_locator is None: + reason = f"selfcheck context resume ์‹คํŒจ: {context_error}" + store.update_task(task, blocked=reason) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + ["reason=selfcheck-context-unavailable", context_error], + ) + return + while True: + unchecked_items = incomplete_results > 0 + success, locator = await run_escalating( + workspace, + store, + task, + "selfcheck", + spec, + initial_resume_locator=resume_locator, + unchecked_items=unchecked_items, + ) + if not success: + current = store.task_state(task).get("blocked") + store.update_task( + task, blocked=current or f"selfcheck failure locator={locator}" + ) + return + errors = implementation_review_errors(task) + if not errors: + break + if locator is None: + reason = "selfcheck ์„ฑ๊ณต locator๊ฐ€ ์—†์–ด context๋ฅผ ์ด์–ด๊ฐˆ ์ˆ˜ ์—†๋‹ค" + store.update_task(task, blocked=reason) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + ["reason=selfcheck-context-unavailable", reason], + ) + return + incomplete_results += 1 + incomplete_retries = max(0, incomplete_results - 1) + if incomplete_retries >= SELF_CHECK_UNCHECKED_RETRY_LIMIT: + reason = ( + "selfcheck checklist remains incomplete after unchecked-item retry: " + f"{incomplete_retries}/{SELF_CHECK_UNCHECKED_RETRY_LIMIT}" + ) + store.update_task( + task, + blocked=f"{reason} locator={locator}", + selfcheck_incomplete=incomplete_results, + selfcheck_context_locator=str(locator), + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=selfcheck-incomplete-limit", + f"detail={'; '.join(errors)}", + f"retry={incomplete_retries}/{SELF_CHECK_UNCHECKED_RETRY_LIMIT}", + f"locator={locator}", + ], + ) + return + store.update_task( + task, + selfcheck_incomplete=incomplete_results, + selfcheck_context_locator=str(locator), + ) + resume_locator = locator + banner( + "์ž๊ฐ€๊ฒ€์ฆ์žฌ์‹œ๋„", + task.name, + [ + f"reason={'; '.join(errors)}", + "mode=unchecked-items", + f"retry={incomplete_retries + 1}/{SELF_CHECK_UNCHECKED_RETRY_LIMIT}", + f"locator={locator}", + ], + ) + store.update_task( + task, + selfcheck_done=True, + selfcheck_incomplete=0, + selfcheck_context_locator=None, + blocked=None, + ) + + +async def run_review( + workspace: Path, + store: StateStore, + task: Task, + resume_locator: Path | None = None, + quota_snapshot: dict[str, Any] | None = None, +) -> str | None: + try: + _, spec = persisted_execution_decision( + store, task, stage="review", quota_snapshot=quota_snapshot + ) + except ExecutionDecisionError as exc: + store.update_task(task, blocked=str(exc)) + banner("์ž‘์—…์ฐจ๋‹จ", task.name, [f"reason={exc}"]) + return None + state = store.task_state(task) + prior_no_progress = int(state.get("review_no_progress", 0)) + if prior_no_progress >= REVIEW_NO_PROGRESS_LIMIT: + locator = state.get("active_locator") + reason = ( + "review no-progress limit already exhausted: " + f"{prior_no_progress}/{REVIEW_NO_PROGRESS_LIMIT}" + ) + store.update_task(task, blocked=f"{reason} locator={locator}") + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=review-no-progress-limit", + f"unchanged_review_attempts={prior_no_progress}/{REVIEW_NO_PROGRESS_LIMIT}", + f"locator={locator}", + ], + ) + return None + before = task_signature(workspace, task) + prior_review_fingerprints = review_fingerprints(workspace, task) + target = task.review.resolve() if task.review else task.directory.resolve() + banner( + "๋ฆฌ๋ทฐ์‹œ์ž‘", + task.name, + [ + f"model={spec.display}", + f"review={target}", + *task_observation_lines(task), + ], + ) + success, locator = await run_escalating( + workspace, + store, + task, + "review", + spec, + initial_resume_locator=resume_locator, + ) + if not success: + current = store.task_state(task).get("blocked") + store.update_task( + task, blocked=current or f"review failure locator={locator}" + ) + return None + after = task_signature(workspace, task) + if before == after: + state = store.task_state(task) + count = int(state.get("review_no_progress", 0)) + 1 + if count >= REVIEW_NO_PROGRESS_LIMIT: + reason = ( + "review made no progress: " + f"{count}/{REVIEW_NO_PROGRESS_LIMIT} locator={locator}" + ) + store.update_task( + task, + review_no_progress=count, + blocked=reason, + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + [ + "reason=review-no-progress-limit", + f"unchanged_review_attempts={count}/{REVIEW_NO_PROGRESS_LIMIT}", + f"locator={locator}", + ], + ) + return None + store.update_task(task, review_no_progress=count) + banner( + "๋ฃจํ”„์ •์ฒด๊ฒฝ๊ณ ", + task.name, + [ + f"unchanged_review_attempts={count}/{REVIEW_NO_PROGRESS_LIMIT}", + f"locator={locator}", + ], + ) + await asyncio.sleep(min(30, count * 5)) + else: + store.update_task(task, review_no_progress=0, blocked=None) + outcome = review_outcome(workspace, task, prior_review_fingerprints) + banner( + "๋ฆฌ๋ทฐ๊ฒฐ๊ณผ", + task.name, + [ + f"verdict={outcome['verdict']}", + f"state={outcome['state']}", + f"path={outcome['path']}", + f"review_log={outcome['review_log']}", + f"locator={locator}", + ], + ) + if outcome["verdict"] == "PASS" and outcome["state"] == "archived": + banner("์ž‘์—…์™„๋ฃŒ", task.name, [f"archive={outcome['path']}", f"locator={locator}"]) + return outcome["path"] + if outcome["verdict"] == "UNKNOWN" or outcome["state"] == "changed": + raise RuntimeError( + "official review๊ฐ€ ํŒ์ •๊ณผ ๋‹ค์Œ ํŒŒ์ผ ์ƒํƒœ๋ฅผ materializeํ•˜์ง€ ์•Š์•˜๋‹ค; " + f"locator={locator}" + ) + if outcome["state"] == "archived": + raise RuntimeError( + f"PASS๊ฐ€ ์•„๋‹Œ review๊ฐ€ ์™„๋ฃŒ archive๋กœ ์ด๋™ํ–ˆ๋‹ค: " + f"verdict={outcome['verdict']} locator={locator}" + ) + return None + + +def status_lines( + task: Task, + stage: str, + dependency: str, + decision: dict[str, Any] | None = None, +) -> list[str]: + route = f"{task.lane}-G{task.grade:02d}" if task.lane and task.grade else "recovery" + base = [f"stage={stage}", f"route={route}", f"dependency={dependency}"] + if decision is not None: + return base + selector_evidence_lines(decision) + return base + + +def select_dispatch_candidates( + store: StateStore, + ready: list[tuple[Task, str]], + *, + persist: bool, + available_slots: int | None = None, +) -> tuple[ + list[tuple[Task, str]], + list[tuple[Task, str, str]], + str, +]: + ready_reviews = [(task, stage) for task, stage in ready if stage == "review"] + ready_workers = [(task, stage) for task, stage in ready if stage in {"worker", "selfcheck"}] + ordered = ready_reviews + ready_workers + claims = store.write_claim_snapshot() + selected: list[tuple[Task, str]] = [] + deferred: list[tuple[Task, str, str]] = [] + timestamp = now_iso() + for task, stage in ordered: + if not task.write_set_known or not task.write_set: + deferred.append( + ( + task, + stage, + "valid non-empty Modified Files Summary write claim์ด ํ•„์š”ํ•˜๋‹ค", + ) + ) + continue + requested = sorted(task.write_set) + invalid_path: str | None = None + for raw_path in requested: + path = Path(raw_path) + resolved = path.resolve() + try: + resolved.relative_to(store.workspace) + except ValueError: + invalid_path = raw_path + break + if ( + not path.is_absolute() + or str(resolved) != raw_path + or resolved == store.workspace + ): + invalid_path = raw_path + break + if invalid_path is not None: + deferred.append( + ( + task, + stage, + f"write claim ๊ฒฝ๋กœ๊ฐ€ canonical workspace file์ด ์•„๋‹ˆ๋‹ค: {invalid_path}", + ) + ) + continue + + conflict: tuple[str, str] | None = None + requested_set = set(requested) + for owner in sorted(claims): + if owner == task.name: + continue + other = claims[owner] + if other.get("exclusive"): + conflict = (owner, "") + break + intersection = sorted(requested_set & set(other.get("paths", []))) + if intersection: + conflict = (owner, intersection[0]) + break + if conflict is not None: + owner, path = conflict + deferred.append( + ( + task, + stage, + f"write claim ์ถฉ๋Œ ๋Œ€๊ธฐ: owner={owner}; path={path}", + ) + ) + continue + + # Capacity-only admission: admit and acquire/replace a claim only + # while a slot remains. A newly capacity-deferred task gets a stable + # wait reason and no new claim; a task that already owns its lifecycle + # claim keeps it unchanged while waiting. + if available_slots is not None and len(selected) >= available_slots: + if task.name in claims: + deferred.append( + ( + task, + stage, + f"capacity waiting: limit reached (selected={len(selected)}/{available_slots})", + ) + ) + else: + deferred.append( + ( + task, + stage, + f"capacity waiting: limit reached (selected={len(selected)}/{available_slots})", + ) + ) + continue + + previous = claims.get(task.name, {}) + claims[task.name] = { + "task": task.name, + "plan_hash": task.plan_hash, + "paths": requested, + "exclusive": False, + "workspace_id": store.workspace_id, + "acquired_at": previous.get("acquired_at") or timestamp, + "updated_at": timestamp, + "source": "plan", + } + selected.append((task, stage)) + + if persist: + store.replace_write_claims(claims, persist=True) + return selected, deferred, "" + + +def ensure_review_shared_state(workspace: Path) -> None: + helper = workspace / "agent-ops" / "bin" / "ai-ignore.sh" + if not helper.is_file(): + raise RuntimeError(f"review shared-state helper๊ฐ€ ์—†๋‹ค: {helper}") + command = [ + "bash", + "-c", + 'source "$1" && agent_ops_ensure_gitignore_task_artifact_block "$2"', + "agent-task-review-preflight", + str(helper), + str(workspace / ".gitignore"), + ] + completed = subprocess.run( + command, + cwd=workspace, + capture_output=True, + text=True, + check=False, + ) + if completed.returncode != 0: + diagnostic = (completed.stderr or completed.stdout).strip() + raise RuntimeError( + f"review shared-state preflight ์‹คํŒจ: {diagnostic or completed.returncode}" + ) + + +async def dispatch(args: argparse.Namespace) -> int: + workspace = Path(args.workspace).resolve() + store = StateStore(workspace) + try: + try: + return await dispatch_with_store(args, workspace, store) + except Exception as exc: + # A scheduler/control-plane exception must not make asyncio.run() + # cancel already-running agent attempts. Keep this loop alive until + # every owned background task finishes naturally; the next + # dispatcher run reconciles their file/state results. + current = asyncio.current_task() + active = [ + task + for task in asyncio.all_tasks() + if task is not current and not task.done() + ] + if active: + banner( + "๋””์ŠคํŒจ์ฒ˜๋ณต๊ตฌ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + f"running_async_tasks={len(active)}", + "scheduler ์˜ˆ์™ธ์™€ ๋ฌด๊ด€ํ•˜๊ฒŒ ์‹คํ–‰ ์ค‘ agent๋ฅผ ์ž์—ฐ ์ข…๋ฃŒ๊นŒ์ง€ ์ถ”์ ", + ], + ) + await asyncio.gather(*active, return_exceptions=True) + raise DispatcherInterruptedWithActiveWork( + f"running agent๊ฐ€ ์žˆ๋˜ scheduler ์˜ˆ์™ธ: {exc}" + ) from exc + raise + finally: + store.close() + + +async def dispatch_with_store( + args: argparse.Namespace, + workspace: Path, + store: StateStore, +) -> int: + orchestration_scope = args.task_group or "__all__" + if args.retry_blocked and not args.dry_run: + store.mark_retry_quota_refresh(args.task_group) + running: dict[str, asyncio.Task[str | None]] = {} + last_wait: dict[str, str] = {} + completed_tasks: dict[str, str] = {} + fatal_errors: dict[str, str] = {} + control_plane_errors: dict[str, str] = {} + work_log_archive_errors: dict[str, str] = {} + review_shared_state_ready = False + candidate_scope: set[str] | None = None + task_cache: dict[str, Task] | None = None + resume_locators: dict[str, Path] = {} + legacy_recoveries: dict[str, LegacyPromotionRecovery] = {} + live_external_processes: dict[str, str] = {} + capacity_waiting: set[str] = set() + max_parallel = validated_max_parallel( + getattr(args, "max_parallel", DEFAULT_MAX_PARALLEL) + ) + + while True: + if task_cache is None: + tasks = scan_tasks(workspace, args.task_group) + task_cache = {task.name: task for task in tasks} + else: + tasks = sorted(task_cache.values(), key=lambda task: (task.index, task.name)) + if args.dry_run: + persistent_errors: dict[str, str] = {} + observed_tasks: set[str] = set() + live_external_processes = {} + else: + store.prepare_orchestration(orchestration_scope, tasks, workspace) + live_external_processes = orchestration_live_agent_processes( + store, + orchestration_scope, + ) + active_or_running = ( + {task.name for task in tasks} + | set(running) + | set(live_external_processes) + ) + reconciled_completed, persistent_errors = store.reconcile_orchestration( + orchestration_scope, + workspace, + active_or_running, + ) + completed_tasks.update(reconciled_completed) + for task_name in persistent_errors: + completed_tasks.pop(task_name, None) + observed_tasks = store.orchestration_tasks(orchestration_scope) + work_log_archives, work_log_archive_errors = ( + archive_completed_group_work_logs( + workspace, + observed_tasks, + completed_tasks, + active_or_running, + ) + ) + for task_group, archive in sorted(work_log_archives.items()): + banner( + "์ž‘์—…๋กœ๊ทธ์•„์นด์ด๋ธŒ", + task_group, + [f"archive={archive}"], + ) + if not tasks and not running: + if live_external_processes: + for task_name, detail in sorted( + live_external_processes.items() + ): + banner( + "์ž‘์—…์ˆ˜ํ–‰์ค‘", + task_name, + [ + "์ด์ „ dispatcher์˜ model process๋ฅผ ์ข…๋ฃŒ์‹œํ‚ค์ง€ ์•Š๊ณ  ์ถ”์ ", + detail, + ], + ) + await asyncio.sleep(STREAM_HEARTBEAT_SECONDS) + continue + if control_plane_errors: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + "์˜ˆ์ƒํ•˜์ง€ ๋ชปํ•œ dispatcher ์ค‘๋‹จ ๊ฒฐ๊ณผ๋ฅผ ์žฌ์กฐ์ •ํ•ด์•ผ ํ•จ", + *( + f"interrupted[{name}]={reason}" + for name, reason in sorted(control_plane_errors.items()) + ), + ], + ) + return 3 + if work_log_archive_errors: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + "์™„๋ฃŒ task group์˜ WORK_LOG archive๋ฅผ ์žฌ์‹œ๋„ํ•ด์•ผ ํ•จ", + *( + f"work-log-archive[{group}]={reason}" + for group, reason in sorted( + work_log_archive_errors.items() + ) + ), + ], + ) + return 3 + if args.task_group and not observed_tasks and not completed_tasks: + reason = ( + "๋ช…์‹œํ•œ task group์—์„œ ๊ด€์ฐฐ๋œ active task๋‚˜ " + "๊ฒ€์ฆ๋œ complete.log ์ด๋ ฅ์ด ์—†๋‹ค" + ) + if not args.dry_run: + store.mark_orchestration_blocked(orchestration_scope, {}) + banner( + "๋””์ŠคํŒจ์น˜์ฐจ๋‹จ", + args.task_group, + [f"reason=unobserved-task-group", reason], + ) + return 2 + pending_attempt_logs = { + name: paths + for name in completed_tasks + if ( + paths := task_attempt_log_directories( + store.runs, + name, + ) + ) + } + if pending_attempt_logs: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + "์™„๋ฃŒ task์˜ attempt ๋กœ๊ทธ ์ •๋ฆฌ๊ฐ€ ์•„์ง ๋๋‚˜์ง€ ์•Š์Œ", + *( + f"attempt-log-cleanup-pending[{name}]=" + + ",".join(str(path) for path in paths) + for name, paths in sorted(pending_attempt_logs.items()) + ), + ], + ) + return 3 + incomplete = sorted(observed_tasks - completed_tasks.keys()) + if incomplete or fatal_errors or persistent_errors: + details = [ + *(f"incomplete={name}" for name in incomplete), + *( + f"persistent[{name}]={reason}" + for name, reason in sorted(persistent_errors.items()) + ), + *(f"error[{name}]={reason}" for name, reason in sorted(fatal_errors.items())), + ] + if not args.dry_run: + store.mark_orchestration_blocked( + orchestration_scope, + { + name: ( + "blocked", + persistent_errors.get(name) + or fatal_errors.get(name) + or "๊ด€์ฐฐ๋œ task๊ฐ€ ์™„๋ฃŒ๋˜์ง€ ์•Š์•˜๋‹ค", + ) + for name in ( + set(incomplete) + | set(fatal_errors) + | set(persistent_errors) + ) + }, + ) + banner("๋””์ŠคํŒจ์น˜์ฐจ๋‹จ", args.task_group or "agent-task", details) + return 2 + if not args.dry_run: + store.mark_orchestration_complete(orchestration_scope) + banner( + "์ž‘์—…์™„๋ฃŒ", + args.task_group or "agent-task", + [ + "active task ์—†์Œ", + f"verified_complete_tasks={len(completed_tasks)}", + *(f"complete[{name}]={path}" for name, path in sorted(completed_tasks.items())), + ], + ) + return 0 + + task_by_name = {task.name: task for task in tasks} + finished_names: set[str] = set() + complete_log_created = False + for name, future in list(running.items()): + if not future.done(): + continue + finished_names.add(name) + completed_archive: str | None = None + try: + completed_archive = future.result() + if completed_archive: + complete_log_created = True + store.mark_orchestration_task_complete( + orchestration_scope, name, completed_archive + ) + completed_tasks[name] = completed_archive + except Exception as exc: # keep other independent tasks alive + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + name, + [f"agent coroutine exception={exc}"], + ) + control_plane_errors[name] = str(exc) + task = task_by_name.get(name) + if task: + store.clear_active(task) + if not completed_archive: + refreshed = read_task_directory(workspace, task.directory) + if refreshed is None: + task_cache.pop(name, None) + else: + task_cache[name] = refreshed + if completed_archive: + task_cache.pop(name, None) + del running[name] + if finished_names: + if complete_log_created: + # Running reviewers may be archiving their own active directory + # while this completion-triggered scan runs. Preserve their + # already-loaded Task snapshots and do not reread those mutable + # directories until their futures finish. + running_snapshots = { + name: task_cache[name] + for name in running + if name in task_cache + } + tasks = scan_tasks( + workspace, + args.task_group, + exclude_names=set(running), + ) + task_cache = { + **{task.name: task for task in tasks}, + **running_snapshots, + } + tasks = sorted( + task_cache.values(), + key=lambda task: (task.index, task.name), + ) + store.prepare_orchestration(orchestration_scope, tasks, workspace) + live_external_processes = orchestration_live_agent_processes( + store, + orchestration_scope, + ) + active_or_running = ( + {task.name for task in tasks} + | set(running) + | set(live_external_processes) + ) + reconciled_completed, persistent_errors = store.reconcile_orchestration( + orchestration_scope, + workspace, + active_or_running, + ) + completed_tasks.update(reconciled_completed) + for task_name in persistent_errors: + completed_tasks.pop(task_name, None) + observed_tasks = store.orchestration_tasks(orchestration_scope) + work_log_archives, work_log_archive_errors = ( + archive_completed_group_work_logs( + workspace, + observed_tasks, + completed_tasks, + active_or_running, + ) + ) + for task_group, archive in sorted( + work_log_archives.items() + ): + banner( + "์ž‘์—…๋กœ๊ทธ์•„์นด์ด๋ธŒ", + task_group, + [f"archive={archive}"], + ) + candidate_scope = None + else: + tasks = sorted(task_cache.values(), key=lambda task: (task.index, task.name)) + candidate_scope = finished_names | capacity_waiting + capacity_waiting = set() + + if not tasks and not running: + # A completion-triggered full scan may have removed the last active task. + continue + + # Derive workspace-global capacity. Count unique task names across + # current running futures and same-workspace live/conservative evidence, + # regardless of --task-group. Do not count pump/heartbeat/selector/ + # quota-probe coroutines as extra slots. + workspace_live = workspace_live_agent_processes(store) + workspace_live = { + name: detail + for name, detail in workspace_live.items() + if name not in finished_names + } + occupied_names = set(running) | set(workspace_live) + available_slots: int | None = ( + None if max_parallel == 0 else max(0, max_parallel - len(occupied_names)) + ) + + ready: list[tuple[Task, str]] = [] + waiting_tasks: list[str] = [] + externally_active: list[tuple[Task, str]] = [] + blocked_details: dict[str, tuple[str, str, str]] = {} + for task in tasks: + # A running future owns this task directory. Re-reading its review + # or dependency files can race with review finalization/archive and + # must never interrupt unrelated tasks. + if task.name in running: + continue + if task.name in control_plane_errors: + reason = ( + "์˜ˆ์ƒํ•˜์ง€ ๋ชปํ•œ agent coroutine ์ค‘๋‹จ ๊ฒฐ๊ณผ๋ฅผ " + "๋‹ค์Œ dispatcher๊ฐ€ ์žฌ์กฐ์ •ํ•ด์•ผ ํ•จ: " + f"{control_plane_errors[task.name]}" + ) + blocked_details[task.name] = ( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + "interrupted", + reason, + ) + waiting_tasks.append(task.name) + continue + state = store.peek_task_state(task) if args.dry_run else store.task_state(task) + legacy_recovery: LegacyPromotionRecovery | None = None + legacy_blocker_reclassified = False + if state.get("blocked"): + legacy_recovery = legacy_promotion_recovery( + store.runs, + task, + state, + ) + legacy_blocker_reclassified = legacy_recovery is not None + else: + legacy_recovery = ( + pending_persisted_legacy_promotion_recovery(task, state) + ) + if legacy_recovery is not None: + legacy_recoveries[task.name] = legacy_recovery + resume_locators[task.name] = legacy_recovery.locator + if legacy_blocker_reclassified: + state = dict(state) + state["blocked"] = None + if not args.dry_run: + recovery_failures = dict( + state.get("recovery_failures", {}) + ) + # Ten identical generic retries represent one terminal + # quota/context/model failure after reclassification. + recovery_failures[legacy_recovery.role] = 1 + store.update_task( + task, + blocked=None, + recovery_failures=recovery_failures, + legacy_terminal_reclassification={ + "role": legacy_recovery.role, + "failure_class": legacy_recovery.failure_class, + "evidence_source": legacy_recovery.evidence_source, + "prior_dispatcher_sha256": + legacy_recovery.prior_dispatcher_sha256, + "current_dispatcher_sha256": + DISPATCHER_SOURCE_SHA256, + "locator": str(legacy_recovery.locator), + "failed_cli": legacy_recovery.failed_cli, + "failed_model": legacy_recovery.failed_model, + "failed_reasoning_effort": + legacy_recovery.failed_reasoning_effort, + }, + ) + state = store.task_state(task) + active_predecessors = live_predecessors( + task, + set(running) | set(live_external_processes), + ) + if active_predecessors: + dependency_ready = False + dependency = ( + "predecessor FINISH ๋Œ€๊ธฐ: " + + ",".join(active_predecessors) + ) + else: + dependency_ready, dependency = dependency_state( + workspace, + task, + ) + stage = task_stage(task, state) + if state.get("active_stage"): + active_stage = str(state["active_stage"]) + active_live, active_detail = external_active_is_live( + state, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if active_live: + reason = f"์™ธ๋ถ€ ์‹คํ–‰์ค‘: stage={active_stage}; {active_detail}" + active_key = ( + f"active|{active_stage}|{state.get('active_locator') or 'unknown'}" + ) + externally_active.append((task, active_stage)) + blocked_details[task.name] = ("์ž‘์—…์ค‘", stage, reason) + if not args.dry_run: + store.adopt_active_write_claim(task) + if not args.dry_run and last_wait.get(task.name) != active_key: + banner("์ž‘์—…์ค‘", task.name, status_lines(task, stage, reason)) + last_wait[task.name] = active_key + continue + if not args.dry_run: + resume_locator = laguna_resume_locator( + state, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + if resume_locator is not None: + resume_locators[task.name] = resume_locator + banner( + "์ž‘์—…๋ณต๊ตฌ", + task.name, + status_lines(task, stage, f"stale active ์ œ์™ธ: {active_detail}"), + ) + store.clear_active(task) + state = store.task_state(task) + reason = "" + if task.errors: + reason = "; ".join(task.errors) + elif task.user_review: + blocking, detail = user_review_blocker_state(task.user_review) + if task.plan is not None or task.review is not None: + blocking = False + detail = "active PLAN/CODE_REVIEW์™€ ๊ณต์กดํ•œ๋‹ค" + reason = ( + f"USER_REVIEW ๋Œ€๊ธฐ: {task.user_review}; {detail}" + if blocking + else ( + "USER_REVIEW stop ๊ณ„์•ฝ ๋ถˆ์ถฉ์กฑ: " + f"{task.user_review}; {detail}" + ) + ) + elif state.get("blocked"): + reason = str(state["blocked"]) + elif not dependency_ready: + reason = dependency + if reason: + wait_key = f"{stage}|{reason}" + event = ( + "์ž‘์—…์ฐจ๋‹จ" + if ( + task.errors + or stage in {"blocked", "user-review"} + or state.get("blocked") + ) + else "์ž‘์—…๋Œ€๊ธฐ" + ) + blocked_details[task.name] = (event, stage, reason) + if not args.dry_run and last_wait.get(task.name) != wait_key: + banner(event, task.name, status_lines(task, stage, reason)) + last_wait[task.name] = wait_key + waiting_tasks.append(task.name) + continue + if task.name not in running and ( + candidate_scope is None or task.name in candidate_scope + ): + ready.append((task, stage)) + + admission_time = datetime.now(KST) + if args.dry_run: + candidates, deferred, _ = select_dispatch_candidates( + store, + ready, + persist=False, + available_slots=available_slots, + ) + for task, stage, reason in deferred: + event = ( + "์ž‘์—…์ฐจ๋‹จ" + if reason.startswith( + ( + "valid non-empty", + "write claim ๊ฒฝ๋กœ๊ฐ€", + ) + ) + else "์ž‘์—…๋Œ€๊ธฐ" + ) + blocked_details[task.name] = (event, stage, reason) + waiting_tasks.append(task.name) + ready_by_name = {task.name: stage for task, stage in candidates} + batch_snapshot = build_admission_batch_snapshot( + store, + candidates, + admission_time, + ) + for task in tasks: + if task.name in ready_by_name: + stage = ready_by_name[task.name] + selector_stage = "review" if stage == "review" else "worker" + preview_state = store.peek_task_state(task) + decisions = preview_state.get("execution_decisions", {}) + prior_decision = ( + decisions.get(selector_stage) + if isinstance(decisions, dict) + else None + ) + quota_snapshot = ( + batch_snapshot + if batch_snapshot is not None + else ( + preview_state.get("quota_snapshot") + if isinstance(preview_state.get("quota_snapshot"), dict) + else None + ) + ) + decision = read_or_preview_stage_decision( + task, + preview_state, + stage=selector_stage, + dry_run=args.dry_run, + quota_snapshot=quota_snapshot, + ) + spec = agent_spec_from_decision(decision) + legacy_recovery = legacy_recoveries.get(task.name) + if legacy_recovery is not None: + failed_spec = failed_spec_from_recovery( + legacy_recovery + ) + spec = promoted_spec(failed_spec, 0) or failed_spec + lines = status_lines(task, stage, "ready", decision=decision) + if legacy_recovery is not None: + lines.extend( + [ + "recovery=legacy-terminal-reclassification", + f"failure_class={legacy_recovery.failure_class}", + f"locator={legacy_recovery.locator}", + ] + ) + banner( + "์ž‘์—…๋Œ€๊ธฐ", + task.name, + lines + [f"model={spec.display}"], + ) + else: + event, stage, reason = blocked_details[task.name] + banner(event, task.name, status_lines(task, stage, reason)) + return 2 if waiting_tasks and not candidates else 0 + + candidates, deferred, _ = select_dispatch_candidates( + store, + ready, + persist=True, + available_slots=available_slots, + ) + batch_snapshot = build_admission_batch_snapshot(store, candidates, admission_time) + for task, stage, reason in deferred: + event = ( + "์ž‘์—…์ฐจ๋‹จ" + if reason.startswith( + ( + "valid non-empty", + "write claim ๊ฒฝ๋กœ๊ฐ€", + ) + ) + else "์ž‘์—…๋Œ€๊ธฐ" + ) + blocked_details[task.name] = (event, stage, reason) + waiting_tasks.append(task.name) + wait_key = f"{stage}|{reason}" + if last_wait.get(task.name) != wait_key: + banner(event, task.name, status_lines(task, stage, reason)) + last_wait[task.name] = wait_key + + # Rebuild capacity_waiting from current capacity-only deferrals. + # Dependency, blocker, invalid-write-set, and claim-collision deferrals + # are not capacity waiters and rely on their existing wake-up event. + if available_slots is not None: + capacity_waiting = { + task.name + for task, stage, reason in deferred + if reason.startswith("capacity waiting:") + } + + if ( + not review_shared_state_ready + and any(stage == "review" for _, stage in candidates) + ): + try: + ensure_review_shared_state(workspace) + except (OSError, RuntimeError) as exc: + # Shared review setup is a blocker only for reviews. It must + # not prevent dependency-independent workers/selfchecks from + # starting and draining in the same scheduler pass. + review_removed: list[tuple[Task, str]] = [] + remaining_candidates: list[tuple[Task, str]] = [] + for task, stage in candidates: + if stage != "review": + remaining_candidates.append((task, stage)) + continue + reason = f"review shared-state preflight failed: {exc}" + store.update_task(task, blocked=reason) + fatal_errors[task.name] = reason + if task.name not in waiting_tasks: + waiting_tasks.append(task.name) + blocked_details[task.name] = ( + "์ž‘์—…์ฐจ๋‹จ", + stage, + reason, + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + status_lines(task, stage, reason), + ) + review_removed.append((task, stage)) + candidates = remaining_candidates + + # Block every ready review that was deferred (e.g. by capacity). + for task, stage, _ in deferred: + if stage == "review": + reason = f"review shared-state preflight failed: {exc}" + store.update_task(task, blocked=reason) + fatal_errors[task.name] = reason + if task.name not in waiting_tasks: + waiting_tasks.append(task.name) + blocked_details[task.name] = ( + "์ž‘์—…์ฐจ๋‹จ", + stage, + reason, + ) + banner( + "์ž‘์—…์ฐจ๋‹จ", + task.name, + status_lines(task, stage, reason), + ) + + # Refill freed runtime slots from disjoint non-review capacity + # waiters in stable deferred order using persistent claim admission. + # Reviews that had received slots retain their existing claims; + # reviews that never received slots do not synthesize claims. + if available_slots is not None and review_removed: + freed_slots = len(review_removed) + refill_inputs = [ + (task, stage) + for task, stage, reason in deferred + if stage in {"worker", "selfcheck"} + and reason.startswith("capacity waiting:") + ] + if refill_inputs: + refilled, refill_deferred, _ = select_dispatch_candidates( + store, + refill_inputs, + persist=True, + available_slots=freed_slots, + ) + candidates.extend(refilled) + for task, stage, reason in refill_deferred: + event = ( + "์ž‘์—…์ฐจ๋‹จ" + if reason.startswith( + ( + "valid non-empty", + "write claim ๊ฒฝ๋กœ๊ฐ€", + ) + ) + else "์ž‘์—…๋Œ€๊ธฐ" + ) + blocked_details[task.name] = (event, stage, reason) + wait_key = f"{stage}|{reason}" + if last_wait.get(task.name) != wait_key: + banner(event, task.name, status_lines(task, stage, reason)) + last_wait[task.name] = wait_key + capacity_waiting = { + task.name + for task, stage, reason in refill_deferred + if reason.startswith("capacity waiting:") + } + else: + capacity_waiting = set() + + batch_snapshot = build_admission_batch_snapshot( + store, candidates, admission_time, + ) + else: + review_shared_state_ready = True + + scheduled = False + for task, stage in candidates: + store.mark_active(task, stage) + resume_locator = resume_locators.pop(task.name, None) + state = store.task_state(task) + task_snapshot = batch_snapshot + if stage == "worker" and has_persisted_worker_decision(state, task) and not retry_quota_refresh_pending(state): + task_snapshot = None + + if stage == "review": + future = asyncio.create_task( + run_review( + workspace, + store, + task, + **( + {"quota_snapshot": task_snapshot} + if task_snapshot is not None + else {} + ), + **( + {"resume_locator": resume_locator} + if resume_locator is not None + else {} + ), + ) + ) + elif stage == "selfcheck": + future = asyncio.create_task( + run_selfcheck( + workspace, + store, + task, + **( + {"resume_locator": resume_locator} + if resume_locator is not None + else {} + ), + ) + ) + else: + future = asyncio.create_task( + run_worker( + workspace, + store, + task, + **( + {"quota_snapshot": task_snapshot} + if task_snapshot is not None + else {} + ), + **( + {"resume_locator": resume_locator} + if resume_locator is not None + else {} + ), + ) + ) + running[task.name] = future + last_wait.pop(task.name, None) + scheduled = True + + if running: + await asyncio.wait(running.values(), return_when=asyncio.FIRST_COMPLETED) + continue + if work_log_archive_errors: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + "์™„๋ฃŒ task group์˜ WORK_LOG archive๋ฅผ ์žฌ์‹œ๋„ํ•ด์•ผ ํ•จ", + *( + f"work-log-archive[{group}]={reason}" + for group, reason in sorted( + work_log_archive_errors.items() + ) + ), + ], + ) + return 3 + if externally_active: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + ["์ƒˆ ์‹คํ–‰ ํ›„๋ณด ์—†์Œ", "active task๋Š” caller๊ฐ€ ๊ณ„์† ์ถ”์ "], + ) + return 3 + if control_plane_errors: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + "์‹คํ–‰ ์ค‘์ด๋˜ ๋…๋ฆฝ ์ž‘์—…์„ ๋ชจ๋‘ ์†Œ์ง„ํ–ˆ๊ณ  ์žฌ์กฐ์ •์ด ํ•„์š”ํ•จ", + *( + f"interrupted[{name}]={reason}" + for name, reason in sorted(control_plane_errors.items()) + ), + ], + ) + return 3 + # Capacity-only wait: external live attempts fill the cap, but we must + # not mark the orchestration blocked. Report as non-terminal tracking + # state and return 3 so a restart with a larger limit or after occupancy + # drops can resume naturally. + if capacity_waiting and not scheduled: + external_fillers = set(occupied_names) - set(running) + if external_fillers: + banner( + "๋””์ŠคํŒจ์น˜์ถ”์ ๋Œ€๊ธฐ", + args.task_group or "agent-task", + [ + f"capacity_waiting={','.join(sorted(capacity_waiting))}", + f"occupied_by_external={','.join(sorted(external_fillers))}", + f"max_parallel={max_parallel}", + "์šฉ๋Ÿ‰ ๋Œ€๊ธฐ: ์™ธ๋ถ€ ์‹คํ–‰์ด ์šฉ๋Ÿ‰์„ ์ฑ„์›Œ ๋‹ค์Œ dispatcher๊ฐ€ ์žฌ์กฐ์ •", + ], + ) + return 3 + if not scheduled: + store.mark_orchestration_blocked( + orchestration_scope, + { + name: ( + "blocked" if event == "์ž‘์—…์ฐจ๋‹จ" else "waiting", + detail, + ) + for name, (event, _, detail) in blocked_details.items() + }, + ) + reason = f"waiting={','.join(sorted(waiting_tasks))}" + banner( + "๋””์ŠคํŒจ์น˜์ฐจ๋‹จ", + args.task_group or "agent-task", + [ + "์‹คํ–‰ ๊ฐ€๋Šฅํ•œ ๋…๋ฆฝ ์ž‘์—…์„ ๋ชจ๋‘ ์†Œ์ง„ํ•จ", + reason, + f"verified_complete_tasks={len(completed_tasks)}", + *( + f"complete[{name}]={path}" + for name, path in sorted(completed_tasks.items()) + ), + *( + f"{name}: stage={stage}; reason={detail}" + for name, (_, stage, detail) in sorted( + blocked_details.items() + ) + ), + ], + ) + return 2 + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--workspace", default=".", help="repository root (default: current directory)") + parser.add_argument("--task-group", help="run only agent-task/") + parser.add_argument("--dry-run", action="store_true", help="classify and print without launching CLIs") + parser.add_argument("--retry-blocked", action="store_true", help="clear dispatcher-local blocked state") + parser.add_argument( + "--max-parallel", + type=int, + default=DEFAULT_MAX_PARALLEL, + metavar="MAX_PARALLEL", + help=( + "physical-workspace global cap on unique active task-stage " + f"attempts; default is {DEFAULT_MAX_PARALLEL}; 0 is unlimited" + ), + ) + parser.add_argument( + "--validate-plan", + metavar="PATH", + help="validate one PLAN Modified Files Summary without starting the dispatcher", + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + try: + validated_max_parallel( + getattr(args, "max_parallel", DEFAULT_MAX_PARALLEL) + ) + except ValueError as exc: + print(f"dispatcher error: {exc}", file=sys.stderr) + return 2 + validate_plan = getattr(args, "validate_plan", None) + if validate_plan: + workspace = Path(args.workspace).resolve() + candidate = Path(validate_plan) + if not candidate.is_absolute(): + candidate = (Path.cwd() / candidate).resolve() + write_set, diagnostics = inspect_write_set(candidate, workspace) + if diagnostics: + for diagnostic in diagnostics: + print(f"plan write-set error: {diagnostic}", file=sys.stderr) + return 2 + for path in sorted(write_set): + validation_claim(path) + return 0 + if os.environ.get(AGENT_PROCESS_MARKER_ENV): + print( + "nested dispatcher invocation rejected: this process is already a " + "dispatcher child; continue the assigned role directly and do not " + "wait for the parent dispatcher", + file=sys.stderr, + ) + return 4 + try: + return asyncio.run(dispatch(args)) + except KeyboardInterrupt: + print("\n์ค‘๋‹จ๋จ", file=sys.stderr) + return 130 + except DispatcherAlreadyRunning as exc: + print(f"dispatcher active: {exc}", file=sys.stderr) + return 3 + except DispatcherTerminalStateError as exc: + print(f"dispatcher error: {exc}", file=sys.stderr) + return 2 + except Exception as exc: + # An unexpected dispatcher failure is not proof that the task group is + # drained or terminal. The caller must inspect active PIDs/locators and + # recover instead of treating it like exit 2. + print(f"dispatcher interrupted: {exc}", file=sys.stderr) + return 3 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatcher_observation.py b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatcher_observation.py new file mode 100644 index 0000000..3d2aa6f --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatcher_observation.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +"""Observation output emitter and formatting utilities for agent-task dispatcher.""" + +from __future__ import annotations + +SEP = "-" * 42 + + +def banner(event: str, task: str, lines: list[str] | None = None) -> None: + display_task = task.rsplit("/", 1)[-1] + print(SEP, flush=True) + print(f"{event}: {display_task}", flush=True) + print(SEP, flush=True) + if display_task != task: + print(f"task={task}", flush=True) + for line in lines or []: + print(line, flush=True) + + +def attempt_event(prefix: str, message: str) -> None: + print(f"{prefix} {message}", flush=True) + + +def validation_claim(path: str) -> None: + """Emit one canonical write claim for standalone PLAN validation.""" + print(path, flush=True) diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/execution_target_policy.py b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/execution_target_policy.py new file mode 100644 index 0000000..6028d62 --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/execution_target_policy.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 +"""Pure execution-target policy for Agent Task worker and review stages.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime + +from zoneinfo import ZoneInfo + + +KST = ZoneInfo("Asia/Seoul") + +VALID_STAGES = {"worker", "review"} +VALID_LANES = {"local", "cloud"} + + +@dataclass(frozen=True) +class RouteTarget: + adapter: str + target: str + execution_class: str + selfcheck_required: bool + + +@dataclass(frozen=True) +class PolicyDecision: + rule_id: str + policy_priority: int + reason_codes: tuple[str, ...] + time_window: str + candidates: tuple[RouteTarget, ...] + + +PI_ORNITH = RouteTarget("pi", "iop/ornith:35b", "local_model", True) +AGY_GEMINI_LOW = RouteTarget( + "agy", "Gemini 3.6 Flash (Low)", "cloud_model", False +) +AGY_GEMINI_MEDIUM = RouteTarget( + "agy", "Gemini 3.6 Flash (Medium)", "cloud_model", False +) +AGY_GEMINI_HIGH = RouteTarget( + "agy", "Gemini 3.6 Flash (High)", "cloud_model", False +) +PI_LAGUNA = RouteTarget("pi", "iop/laguna-s:2.1", "local_model", True) +CLAUDE_OPUS = RouteTarget("claude", "claude-opus-4-8", "cloud_model", False) +CLAUDE_HAIKU_XHIGH = RouteTarget( + "claude", "claude-haiku-4-5", "cloud_model", False +) +CODEX_SPARK_XHIGH = RouteTarget( + "codex", "gpt-5.3-codex-spark", "cloud_model", False +) +CODEX_SOL_XHIGH = RouteTarget("codex", "gpt-5.6-sol", "cloud_model", False) +CODEX_TERRA_HIGH = RouteTarget("codex", "gpt-5.6-terra", "cloud_model", False) + + +CANONICAL_TARGETS = ( + PI_ORNITH, + AGY_GEMINI_LOW, + AGY_GEMINI_MEDIUM, + AGY_GEMINI_HIGH, + PI_LAGUNA, + CLAUDE_OPUS, + CLAUDE_HAIKU_XHIGH, + CODEX_SPARK_XHIGH, + CODEX_SOL_XHIGH, + CODEX_TERRA_HIGH, +) + + +def canonical_target(adapter: str, target: str) -> RouteTarget | None: + """Resolve one policy-owned adapter + target identity.""" + return next( + ( + candidate + for candidate in CANONICAL_TARGETS + if candidate.adapter == adapter and candidate.target == target + ), + None, + ) + + +def promotion_target(current: RouteTarget) -> RouteTarget | None: + """Return the next target in the policy-owned cloud promotion chain.""" + if current.adapter == "agy" and current in { + AGY_GEMINI_LOW, + AGY_GEMINI_MEDIUM, + AGY_GEMINI_HIGH, + }: + return CLAUDE_OPUS + if current == CLAUDE_OPUS: + return CODEX_TERRA_HIGH + return None + + +@dataclass(frozen=True) +class QuotaProbeSpec: + command: str + target: str + required_caps: tuple[str, ...] + + +def quota_probe_spec(target: RouteTarget) -> QuotaProbeSpec | None: + """Return the policy-owned quota probe spec for a route target.""" + if target.execution_class == "local_model": + return None + if target.adapter == "agy": + return QuotaProbeSpec( + command="agy", + target=target.target, + required_caps=("overall", f"model:{target.target}"), + ) + if target.adapter in {"claude", "codex"}: + return QuotaProbeSpec( + command=target.adapter, + target=target.target, + required_caps=("overall",), + ) + return None + + +def _validate(stage: str, lane: str, grade: int, evaluated_at: datetime) -> None: + if stage not in VALID_STAGES: + raise ValueError(f"unsupported stage: {stage}") + if lane not in VALID_LANES: + raise ValueError(f"unsupported lane: {lane}") + if not 1 <= grade <= 10: + raise ValueError(f"grade must be in G01..G10: {grade}") + if evaluated_at.tzinfo is None or evaluated_at.utcoffset() is None: + raise ValueError("evaluated_at must be timezone-aware") + + +def _kst_time_window(evaluated_at: datetime) -> str: + kst_time = evaluated_at.astimezone(KST).time() + if 7 <= kst_time.hour < 23: + return "kst-day-[07:00,23:00)" + return "kst-night-[23:00,07:00)" + + +def select_policy( + *, stage: str, lane: str, grade: int, evaluated_at: datetime +) -> PolicyDecision: + """Return the ordered target policy for one initial route evaluation.""" + + _validate(stage, lane, grade, evaluated_at) + + if stage == "review": + return PolicyDecision( + rule_id="official-review-codex", + policy_priority=10, + reason_codes=("official_review_fixed",), + time_window="not_applicable", + candidates=(CODEX_SOL_XHIGH,), + ) + + if lane == "local": + if grade <= 6: + return PolicyDecision( + rule_id="worker-local-g01-g06", + policy_priority=30, + reason_codes=("local_low_grade",), + time_window="not_applicable", + candidates=(PI_ORNITH,), + ) + if grade <= 8: + time_window = _kst_time_window(evaluated_at) + if time_window == "kst-day-[07:00,23:00)": + rule_id = "worker-local-g07-g08-kst-day" + reason_code = "kst_day_gemini_medium" + candidates = (AGY_GEMINI_MEDIUM, PI_LAGUNA) + else: + rule_id = "worker-local-g07-g08-kst-night" + reason_code = "kst_night_laguna" + candidates = (PI_LAGUNA, AGY_GEMINI_MEDIUM) + return PolicyDecision( + rule_id=rule_id, + policy_priority=20, + reason_codes=(reason_code,), + time_window=time_window, + candidates=candidates, + ) + return PolicyDecision( + rule_id="worker-local-g09-g10", + policy_priority=30, + reason_codes=("local_high_grade_cloud_target",), + time_window="not_applicable", + candidates=(CLAUDE_OPUS,), + ) + + if grade <= 2: + candidates = ( + CODEX_SPARK_XHIGH, + AGY_GEMINI_LOW, + CLAUDE_HAIKU_XHIGH, + ) + rule_id = "worker-cloud-g01-g02" + reason_code = "cloud_spark_priority_grade" + elif grade <= 4: + candidates = (AGY_GEMINI_MEDIUM,) + rule_id = "worker-cloud-g03-g04" + reason_code = "cloud_gemini_medium_grade" + elif grade <= 6: + candidates = (AGY_GEMINI_HIGH,) + rule_id = "worker-cloud-g05-g06" + reason_code = "cloud_gemini_high_grade" + elif grade <= 8: + candidates = (CLAUDE_OPUS,) + rule_id = "worker-cloud-g07-g08" + reason_code = "cloud_opus_grade" + else: + candidates = (CODEX_SOL_XHIGH,) + rule_id = "worker-cloud-g09-g10" + reason_code = "cloud_codex_grade" + return PolicyDecision( + rule_id=rule_id, + policy_priority=30, + reason_codes=(reason_code,), + time_window="not_applicable", + candidates=candidates, + ) diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/select_execution_target.py b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/select_execution_target.py new file mode 100644 index 0000000..1ec22b3 --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/scripts/select_execution_target.py @@ -0,0 +1,1430 @@ +#!/usr/bin/env python3 +"""Deterministic execution-target selector CLI over the pure route policy. + +The selector consumes a static routing task file (``PLAN-*`` or +``CODE_REVIEW-*``), its ``task/plan/tag`` generation header and optional prior +decision / quota snapshot, and returns a stable JSON contract that the +dispatcher can persist. This module exposes the schema/invalid-input, +worker/review grade matrix, resume, failover, and policy-owned promotion +transitions at the selector surface. It also owns shell-less quota probe +normalization for live dispatcher routing. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import re +import shlex +import subprocess +import sys +from datetime import datetime +from pathlib import Path + + +SCHEMA_VERSION = "1.0" +TIMEZONE_NAME = "Asia/Seoul" +DEFAULT_QUOTA_PROBE_COMMAND = "iop-node quota-probe" + +_FILENAME_RE = re.compile(r"^(PLAN|CODE_REVIEW)-(local|cloud)-G(\d{2})\.md$") +_HEADER_RE = re.compile(r"") +_STAGE_BY_KIND = {"PLAN": "worker", "CODE_REVIEW": "review"} +_VALID_TRANSITIONS = {"initial", "resume", "failover", "promotion"} +_VALID_EXECUTION_CLASSES = {"local_model", "cloud_model"} +_VALID_QUOTA_MODES = {"unbounded", "bounded"} +_QUALIFIED_FAILOVER_FAILURES = { + "provider-quota", + "context-limit", + "model-unavailable", + "provider-stream-disconnect", +} +_QUALIFIED_PROMOTION_FAILURES = _QUALIFIED_FAILOVER_FAILURES | { + "provider-connection" +} + +_VALID_QUOTA_STATUSES = {"not_applicable", "available", "exhausted", "unknown"} +_VALID_ELIGIBILITY = {"eligible", "ineligible"} +_VALID_REJECTION_REASONS = {"quota_exhausted"} +# Local G07~G08 initial decisions record their KST window; resume preserves it. +_VALID_TIME_WINDOWS = { + "kst-day-[07:00,23:00)", + "kst-night-[23:00,07:00)", + "not_applicable", +} + + +def _load_policy(): + path = Path(__file__).resolve().parent / "execution_target_policy.py" + spec = importlib.util.spec_from_file_location("execution_target_policy", path) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +policy = _load_policy() + + +class SelectorInputError(Exception): + """Input contract violation returned as stderr JSON with a non-zero exit.""" + + def __init__(self, code: str, message: str) -> None: + super().__init__(message) + self.code = code + + +def _parse_filename(task_file: Path) -> tuple[str, str, int]: + name = Path(task_file).name + match = _FILENAME_RE.match(name) + if match is None: + raise SelectorInputError( + "invalid_task_filename", + f"task file must match (PLAN|CODE_REVIEW)-(local|cloud)-GNN.md: {name!r}", + ) + kind, lane, grade_str = match.group(1), match.group(2), match.group(3) + grade = int(grade_str) + if not 1 <= grade <= 10: + raise SelectorInputError( + "invalid_grade", f"grade must be G01..G10: G{grade_str}" + ) + return kind, lane, grade + + +def _parse_header(task_file: Path) -> tuple[str, int, str]: + try: + with Path(task_file).open("rb") as handle: + head = handle.read(1024) + except OSError as exc: + raise SelectorInputError("task_file_unreadable", str(exc)) from exc + text = head.decode("utf-8", errors="replace") + match = _HEADER_RE.search(text) + if match is None: + raise SelectorInputError( + "malformed_header", + "first 1KiB must contain ", + ) + return match.group(1), int(match.group(2)), match.group(3) + + +def _work_unit_id(header: tuple[str, int, str]) -> str: + task, plan, tag = header + return f"{task}::plan-{plan}::tag-{tag}" + + +def _validate_evaluated_at(evaluated_at: datetime) -> None: + if evaluated_at.tzinfo is None or evaluated_at.utcoffset() is None: + raise SelectorInputError( + "naive_evaluated_at", "evaluated_at must be timezone-aware" + ) + + +def _validate_prior_selected(selected: object) -> None: + code = "malformed_prior_decision" + if not isinstance(selected, dict): + raise SelectorInputError(code, "prior_decision.selected must be an object") + for field in ("adapter", "target", "execution_class"): + candidate = selected.get(field) + if not isinstance(candidate, str) or not candidate: + raise SelectorInputError( + code, + f"prior_decision.selected.{field} must be a non-empty string", + ) + if selected["execution_class"] not in _VALID_EXECUTION_CLASSES: + raise SelectorInputError( + code, + "prior_decision.selected.execution_class must be one of " + f"{sorted(_VALID_EXECUTION_CLASSES)}", + ) + if not isinstance(selected.get("selfcheck_required"), bool): + raise SelectorInputError( + code, + "prior_decision.selected.selfcheck_required must be a boolean", + ) + + +def _require_non_empty_string( + container: dict, field: str, prefix: str, code: str +) -> None: + value = container.get(field) + if not isinstance(value, str) or not value: + raise SelectorInputError( + code, f"{prefix}.{field} must be a non-empty string" + ) + + +def _require_string_enum( + container: dict, field: str, allowed: set, prefix: str, code: str +) -> None: + value = container.get(field) + if not isinstance(value, str) or value not in allowed: + raise SelectorInputError( + code, f"{prefix}.{field} must be one of {sorted(allowed)}" + ) + + +def _require_nullable_string( + container: dict, field: str, prefix: str, code: str +) -> None: + if field not in container: + raise SelectorInputError(code, f"{prefix}.{field} is required") + value = container[field] + if value is not None and not isinstance(value, str): + raise SelectorInputError( + code, f"{prefix}.{field} must be null or a string" + ) + + +def _validate_prior_candidates(candidates: object) -> None: + """Validate every reused candidate against the initial output schema. + + Each entry must carry the full ``_initial`` candidate field set with the + correct types and enum values, and ``candidate_rank`` must be 1-based and + consecutive so a resume cannot re-emit a partial ranking. + """ + + code = "malformed_prior_decision" + if not isinstance(candidates, list) or not candidates: + raise SelectorInputError( + code, "prior_decision.candidates must be a non-empty list" + ) + for index, entry in enumerate(candidates): + prefix = f"prior_decision.candidates[{index}]" + if not isinstance(entry, dict): + raise SelectorInputError(code, f"{prefix} must be an object") + rank = entry.get("candidate_rank") + if ( + isinstance(rank, bool) + or not isinstance(rank, int) + or rank != index + 1 + ): + raise SelectorInputError( + code, + f"{prefix}.candidate_rank must be {index + 1} " + "(1-based and consecutive)", + ) + _require_non_empty_string(entry, "adapter", prefix, code) + _require_non_empty_string(entry, "target", prefix, code) + _require_string_enum( + entry, "execution_class", _VALID_EXECUTION_CLASSES, prefix, code + ) + if not isinstance(entry.get("selfcheck_required"), bool): + raise SelectorInputError( + code, f"{prefix}.selfcheck_required must be a boolean" + ) + _require_string_enum(entry, "quota_mode", _VALID_QUOTA_MODES, prefix, code) + _require_string_enum( + entry, "quota_status", _VALID_QUOTA_STATUSES, prefix, code + ) + _require_string_enum(entry, "eligibility", _VALID_ELIGIBILITY, prefix, code) + if "rejection_reason" not in entry: + raise SelectorInputError( + code, f"{prefix}.rejection_reason is required" + ) + rejection = entry["rejection_reason"] + if rejection is not None and ( + not isinstance(rejection, str) + or rejection not in _VALID_REJECTION_REASONS + ): + raise SelectorInputError( + code, + f"{prefix}.rejection_reason must be null or one of " + f"{sorted(_VALID_REJECTION_REASONS)}", + ) + + +def _validate_prior_decision_evidence(decision: object) -> None: + """Validate the reused decision evidence block against initial output.""" + + code = "malformed_prior_decision" + prefix = "prior_decision.decision" + if not isinstance(decision, dict): + raise SelectorInputError(code, f"{prefix} must be an object") + _require_non_empty_string(decision, "rule_id", prefix, code) + priority = decision.get("policy_priority") + if isinstance(priority, bool) or not isinstance(priority, int): + raise SelectorInputError( + code, f"{prefix}.policy_priority must be an integer" + ) + reason_codes = decision.get("reason_codes") + if not isinstance(reason_codes, list) or not all( + isinstance(item, str) and item for item in reason_codes + ): + raise SelectorInputError( + code, f"{prefix}.reason_codes must be a list of non-empty strings" + ) + _require_non_empty_string(decision, "evaluated_at", prefix, code) + if decision.get("timezone") != TIMEZONE_NAME: + raise SelectorInputError( + code, f"{prefix}.timezone must be {TIMEZONE_NAME!r}" + ) + _require_string_enum( + decision, "time_window", _VALID_TIME_WINDOWS, prefix, code + ) + if not isinstance(decision.get("pinned"), bool): + raise SelectorInputError(code, f"{prefix}.pinned must be a boolean") + + +def _validate_prior_quota(quota: object) -> None: + """Validate the reused quota block against the initial output schema.""" + + code = "malformed_prior_decision" + prefix = "prior_decision.quota" + if not isinstance(quota, dict): + raise SelectorInputError(code, f"{prefix} must be an object") + _require_nullable_string(quota, "snapshot_id", prefix, code) + _require_string_enum(quota, "mode", _VALID_QUOTA_MODES, prefix, code) + _require_string_enum(quota, "status", _VALID_QUOTA_STATUSES, prefix, code) + _require_non_empty_string(quota, "source", prefix, code) + _require_nullable_string(quota, "checked_at", prefix, code) + + +def _validate_prior_decision(value: object) -> dict: + """Validate the nested prior-decision schema before it is reused on resume. + + Only container/field/type/enum shape is enforced here; identity equality + against the current work unit stays in ``_resume``. Unknown extra keys are + tolerated so forward-compatible producers are not rejected. + """ + + code = "malformed_prior_decision" + if not isinstance(value, dict): + raise SelectorInputError(code, "prior_decision must be an object") + if value.get("schema_version") != SCHEMA_VERSION: + raise SelectorInputError( + code, f"prior_decision.schema_version must be {SCHEMA_VERSION!r}" + ) + required = ( + "work_unit_id", + "stage", + "lane", + "grade", + "selected", + "candidates", + "decision", + "quota", + ) + missing = [key for key in required if key not in value] + if missing: + raise SelectorInputError( + code, f"prior_decision missing keys: {missing}" + ) + if not isinstance(value["work_unit_id"], str): + raise SelectorInputError( + code, "prior_decision.work_unit_id must be a string" + ) + stage = value["stage"] + if not isinstance(stage, str) or stage not in policy.VALID_STAGES: + raise SelectorInputError( + code, + f"prior_decision.stage must be one of {sorted(policy.VALID_STAGES)}", + ) + lane = value["lane"] + if not isinstance(lane, str) or lane not in policy.VALID_LANES: + raise SelectorInputError( + code, + f"prior_decision.lane must be one of {sorted(policy.VALID_LANES)}", + ) + grade = value["grade"] + if isinstance(grade, bool) or not isinstance(grade, int) or not 1 <= grade <= 10: + raise SelectorInputError( + code, "prior_decision.grade must be an integer in G01..G10" + ) + _validate_prior_selected(value["selected"]) + _validate_prior_candidates(value["candidates"]) + _validate_prior_decision_evidence(value["decision"]) + _validate_prior_quota(value["quota"]) + return value + + +def _validate_quota_snapshot( + value: object | None, + *, + require_producer_shape: bool = False, +) -> dict | None: + """Validate the optional quota snapshot container before it is reflected. + + Required-cap tri-state normalization and admission stay with + ``02+01_quota_input``; here we only reject malformed containers and target + entries so ``_snapshot_status`` never dereferences a non-object. + """ + + if value is None: + return None + code = "malformed_quota_snapshot" + if not isinstance(value, dict): + raise SelectorInputError(code, "quota_snapshot must be an object") + if ( + require_producer_shape + and value.get("schema_version") != SCHEMA_VERSION + ): + raise SelectorInputError(code, "quota_snapshot.schema_version must be '1.0'") + for field in ("snapshot_id", "checked_at"): + if field in value and value[field] is not None and not isinstance( + value[field], str + ): + raise SelectorInputError( + code, f"quota_snapshot.{field} must be null or a string" + ) + if require_producer_shape and ( + field not in value or not value[field] + ): + raise SelectorInputError( + code, f"quota_snapshot.{field} must be a non-empty string" + ) + if "source" in value: + _require_non_empty_string(value, "source", "quota_snapshot", code) + elif require_producer_shape: + raise SelectorInputError( + code, "quota_snapshot.source must be a non-empty string" + ) + targets = value.get("targets", []) + if not isinstance(targets, list): + raise SelectorInputError(code, "quota_snapshot.targets must be a list") + if require_producer_shape and not targets: + raise SelectorInputError( + code, "quota_snapshot.targets must be a non-empty list" + ) + for index, entry in enumerate(targets): + if not isinstance(entry, dict): + raise SelectorInputError( + code, f"quota_snapshot.targets[{index}] must be an object" + ) + for field in ("adapter", "target", "status"): + candidate = entry.get(field) + if not isinstance(candidate, str) or not candidate: + raise SelectorInputError( + code, + f"quota_snapshot.targets[{index}].{field} must be a " + "non-empty string", + ) + if entry["status"] not in {"available", "exhausted", "unknown"}: + raise SelectorInputError( + code, + f"quota_snapshot.targets[{index}].status must be available, " + "exhausted, or unknown", + ) + if require_producer_shape: + caps = value.get("required_caps") + if not isinstance(caps, list) or not caps: + raise SelectorInputError( + code, "quota_snapshot.required_caps must be a non-empty list" + ) + for index, cap in enumerate(caps): + prefix = f"quota_snapshot.required_caps[{index}]" + if not isinstance(cap, dict): + raise SelectorInputError(code, f"{prefix} must be an object") + _require_non_empty_string(cap, "name", prefix, code) + _require_string_enum( + cap, + "status", + {"available", "exhausted", "unknown"}, + prefix, + code, + ) + remaining = cap.get("remaining_percent") + if remaining is not None and ( + isinstance(remaining, bool) + or not isinstance(remaining, (int, float)) + ): + raise SelectorInputError( + code, f"{prefix}.remaining_percent must be null or numeric" + ) + reasons = value.get("reason_codes") + if not isinstance(reasons, list) or not all( + isinstance(reason, str) and reason for reason in reasons + ): + raise SelectorInputError( + code, + "quota_snapshot.reason_codes must be a list of non-empty strings", + ) + return value + + +def _snapshot_status(target, quota_snapshot: dict | None) -> str: + """Reflect an already-normalized snapshot status for a cloud target. + + The required-cap tri-state derivation from the Go usage checker is owned by + ``02+01_quota_input``. Here we only surface an injected, pre-normalized + status; an absent or unmatched snapshot is reported as ``unknown``. + """ + + if quota_snapshot is None: + return "unknown" + for entry in quota_snapshot.get("targets", []): + if ( + entry.get("adapter") == target.adapter + and entry.get("target") == target.target + ): + status = entry.get("status", "unknown") + if status in {"available", "exhausted", "unknown"}: + return status + return "unknown" + return "unknown" + + +def probe_candidate_quota( + *, + target: str, + adapter: str, + required_caps: tuple[str, ...] | list[str], + checked_at: datetime, + quota_probe_command: str = DEFAULT_QUOTA_PROBE_COMMAND, + probe_command: str | None = None, +) -> dict: + """Execute shell-less quota probe command and return a snapshot dict.""" + checked_at_iso = checked_at.astimezone(policy.KST).isoformat() + try: + cmd = shlex.split(quota_probe_command) + cmd.extend(["--target", target, "--command", probe_command or adapter]) + for cap in required_caps: + cmd.extend(["--required-cap", cap]) + cmd.extend(["--checked-at", checked_at_iso]) + res = subprocess.run(cmd, capture_output=True, text=True, timeout=5) + if res.returncode == 0 and res.stdout: + snapshot = _validate_quota_snapshot( + json.loads(res.stdout), require_producer_shape=True + ) + assert snapshot is not None + if not any( + entry["adapter"] == adapter and entry["target"] == target + for entry in snapshot["targets"] + ): + raise SelectorInputError( + "malformed_quota_snapshot", + "quota snapshot does not contain the requested target identity", + ) + return snapshot + except Exception: + pass + + return { + "schema_version": SCHEMA_VERSION, + "snapshot_id": None, + "source": quota_probe_command, + "checked_at": checked_at_iso, + "targets": [ + {"adapter": adapter, "target": target, "status": "unknown"} + ], + "required_caps": [ + { + "name": cap, + "status": "unknown", + "remaining_percent": None, + } + for cap in required_caps + ], + "reason_codes": ["probe_error"], + } + + +class QuotaBatchProvider: + """Aggregate shell-less quota probes across unique probe keys into a single snapshot.""" + + def __init__(self, quota_probe_command: str = DEFAULT_QUOTA_PROBE_COMMAND): + self.quota_probe_command = quota_probe_command + + def aggregate( + self, + *, + snapshot_id: str, + checked_at: datetime, + keys: list | set, + ) -> dict | None: + if not keys: + return None + + checked_at_iso = checked_at.astimezone(policy.KST).isoformat() + targets = [] + caps = [] + reasons = [] + + seen_keys = set() + unique_keys = [] + for key in keys: + if len(key) == 3: + adapter, target_name, required_caps = key + probe_command = adapter + elif len(key) >= 4: + adapter, target_name, probe_command, required_caps = key[:4] + else: + continue + req_tuple = tuple(required_caps) + k = (adapter, target_name, probe_command, req_tuple) + if k not in seen_keys: + seen_keys.add(k) + unique_keys.append((adapter, target_name, probe_command, req_tuple)) + + for adapter, target_name, probe_command, required_caps in unique_keys: + try: + child = probe_candidate_quota( + target=target_name, + adapter=adapter, + probe_command=probe_command, + required_caps=required_caps, + checked_at=checked_at, + quota_probe_command=self.quota_probe_command, + ) + except Exception: + child = None + if not isinstance(child, dict): + child = { + "schema_version": SCHEMA_VERSION, + "snapshot_id": None, + "source": self.quota_probe_command, + "checked_at": checked_at_iso, + "targets": [ + {"adapter": adapter, "target": target_name, "status": "unknown"} + ], + "required_caps": [ + {"name": cap, "status": "unknown", "remaining_percent": None} + for cap in required_caps + ], + "reason_codes": ["probe_error"], + } + for t in child.get("targets", []): + t_entry = { + "adapter": t["adapter"], + "target": t["target"], + "status": t.get("status", "unknown"), + } + if child.get("snapshot_id"): + t_entry["child_snapshot_id"] = child["snapshot_id"] + if child.get("reason_codes"): + t_entry["child_reason_codes"] = child["reason_codes"] + if probe_command: + t_entry["command"] = probe_command + targets.append(t_entry) + + for c in child.get("required_caps", []): + if c not in caps: + caps.append(c) + for r in child.get("reason_codes", []): + if r not in reasons: + reasons.append(r) + + return { + "schema_version": SCHEMA_VERSION, + "snapshot_id": snapshot_id, + "source": self.quota_probe_command, + "checked_at": checked_at_iso, + "targets": targets, + "required_caps": caps or [ + {"name": "overall", "status": "available", "remaining_percent": None} + ], + "reason_codes": reasons or ["batch_probe"], + } + + +quota_provider = QuotaBatchProvider() + + +def derive_work_unit_quota_evidence( + prior_decision: dict | None, + *, + status: str = "exhausted", + reason: str = "confirmed_runtime_provider_quota", +) -> dict: + """Derive task-local quota evidence inheriting observation identity from a prior decision.""" + if not isinstance(prior_decision, dict): + return { + "schema_version": SCHEMA_VERSION, + "snapshot_id": None, + "source": DEFAULT_QUOTA_PROBE_COMMAND, + "checked_at": datetime.now(policy.KST).isoformat(), + "targets": [], + "required_caps": [], + "reason_codes": [reason], + } + + selected = prior_decision.get("selected") + adapter = selected.get("adapter") if isinstance(selected, dict) else None + target_name = selected.get("target") if isinstance(selected, dict) else None + + prior_quota = prior_decision.get("quota") + if not isinstance(prior_quota, dict): + prior_quota = {} + + snapshot_id = prior_quota.get("snapshot_id") + checked_at = prior_quota.get("checked_at") or datetime.now(policy.KST).isoformat() + source = prior_quota.get("source", DEFAULT_QUOTA_PROBE_COMMAND) + + targets = [] + found = False + for t_entry in prior_quota.get("targets", []): + if isinstance(t_entry, dict): + new_entry = dict(t_entry) + if ( + adapter + and target_name + and t_entry.get("adapter") == adapter + and t_entry.get("target") == target_name + ): + new_entry["status"] = status + found = True + targets.append(new_entry) + + if not found and adapter and target_name: + targets.append( + { + "adapter": adapter, + "target": target_name, + "status": status, + } + ) + + reason_codes = list(prior_quota.get("reason_codes", [])) + if reason not in reason_codes: + reason_codes.append(reason) + + return { + "schema_version": prior_quota.get("schema_version", SCHEMA_VERSION), + "snapshot_id": snapshot_id, + "source": source, + "checked_at": checked_at, + "targets": targets, + "required_caps": list(prior_quota.get("required_caps", [])), + "reason_codes": reason_codes, + } + + + +def _candidate_quota( + target, + quota_snapshot: dict | None, + evaluated_at: datetime, + quota_probe_command: str, +) -> tuple[str, str, dict | None]: + if target.execution_class == "local_model": + return "unbounded", "not_applicable", None + if quota_snapshot is not None: + return "bounded", _snapshot_status(target, quota_snapshot), quota_snapshot + probe_spec = policy.quota_probe_spec(target) + if probe_spec is not None: + snapshot = probe_candidate_quota( + target=target.target, + adapter=target.adapter, + required_caps=probe_spec.required_caps, + checked_at=evaluated_at, + quota_probe_command=quota_probe_command, + ) + return "bounded", _snapshot_status(target, snapshot), snapshot + return "bounded", "unknown", None + + +def _selected_quota( + selected, + quota_snapshot: dict | None, + quota_probe_command: str, + probed_snapshot: dict | None = None, +) -> dict: + if selected.execution_class == "local_model": + return { + "snapshot_id": None, + "mode": "unbounded", + "status": "not_applicable", + "source": "local_unbounded", + "checked_at": None, + "targets": [], + } + if quota_snapshot is not None: + return { + "snapshot_id": quota_snapshot.get("snapshot_id"), + "mode": "bounded", + "status": _snapshot_status(selected, quota_snapshot), + "source": quota_snapshot.get("source", quota_probe_command), + "checked_at": quota_snapshot.get("checked_at"), + "targets": [ + dict(entry) for entry in quota_snapshot.get("targets", []) + ], + } + if probed_snapshot is not None: + return { + "snapshot_id": probed_snapshot.get("snapshot_id"), + "mode": "bounded", + "status": _snapshot_status(selected, probed_snapshot), + "source": probed_snapshot.get("source", quota_probe_command), + "checked_at": probed_snapshot.get("checked_at"), + "targets": [ + dict(entry) for entry in probed_snapshot.get("targets", []) + ], + } + return { + "snapshot_id": None, + "mode": "bounded", + "status": "unknown", + "source": quota_probe_command, + "checked_at": None, + "targets": [], + } + + +def _initial( + *, + work_unit_id: str, + stage: str, + lane: str, + grade: int, + evaluated_at: datetime, + quota_snapshot: dict | None, + quota_probe_command: str, +) -> dict: + try: + decision = policy.select_policy( + stage=stage, lane=lane, grade=grade, evaluated_at=evaluated_at + ) + except ValueError as exc: + raise SelectorInputError("invalid_route", str(exc)) from exc + candidates = [] + selected = None + selected_probed_snapshot = None + for rank, target in enumerate(decision.candidates, start=1): + mode, status, probed_snapshot = _candidate_quota( + target, quota_snapshot, evaluated_at, quota_probe_command + ) + eligible = status != "exhausted" + candidates.append( + { + "candidate_rank": rank, + "adapter": target.adapter, + "target": target.target, + "execution_class": target.execution_class, + "selfcheck_required": target.selfcheck_required, + "quota_mode": mode, + "quota_status": status, + "eligibility": "eligible" if eligible else "ineligible", + "rejection_reason": None if eligible else "quota_exhausted", + } + ) + if eligible and selected is None: + selected = target + selected_probed_snapshot = probed_snapshot + if selected is None: + raise SelectorInputError( + "no_eligible_target", + "all policy candidates are exhausted according to the quota snapshot", + ) + return { + "schema_version": SCHEMA_VERSION, + "work_unit_id": work_unit_id, + "stage": stage, + "lane": lane, + "grade": grade, + "selected": { + "adapter": selected.adapter, + "target": selected.target, + "execution_class": selected.execution_class, + "selfcheck_required": selected.selfcheck_required, + }, + "candidates": candidates, + "decision": { + "rule_id": decision.rule_id, + "policy_priority": decision.policy_priority, + "reason_codes": list(decision.reason_codes), + "evaluated_at": evaluated_at.astimezone(policy.KST).isoformat(), + "timezone": TIMEZONE_NAME, + "time_window": decision.time_window, + "pinned": False, + }, + "quota": _selected_quota( + selected, quota_snapshot, quota_probe_command, selected_probed_snapshot + ), + "transition": { + "previous_target": None, + "next_target": None, + "trigger": "initial", + "context_transfer": "none", + }, + } + + + +def _validate_selected_and_used_history( + prior_decision: dict, + canonical_targets: list, +) -> None: + code = "malformed_prior_decision" + selected = prior_decision.get("selected") + if not isinstance(selected, dict): + raise SelectorInputError(code, "prior_decision.selected must be an object") + + sel_key = (selected.get("adapter"), selected.get("target")) + canon_keys_list = [(c.adapter, c.target) for c in canonical_targets] + canon_keys_set = set(canon_keys_list) + + matching_cand = policy.canonical_target(*sel_key) + if matching_cand is None: + raise SelectorInputError( + code, + f"prior_decision.selected {sel_key} is not a policy-owned target", + ) + if ( + selected.get("execution_class") != matching_cand.execution_class + or selected.get("selfcheck_required") != matching_cand.selfcheck_required + ): + raise SelectorInputError( + code, + f"prior_decision.selected attributes do not match canonical target for {sel_key}", + ) + + if sel_key not in canon_keys_set: + promotion_path = prior_decision.get("promotion_path") + if not isinstance(promotion_path, list) or len(promotion_path) < 2: + raise SelectorInputError( + code, + "promoted prior_decision requires promotion_path evidence", + ) + path_targets = [] + for index, entry in enumerate(promotion_path): + if not isinstance(entry, dict): + raise SelectorInputError( + code, f"promotion_path[{index}] must be an object" + ) + target = policy.canonical_target( + entry.get("adapter"), entry.get("target") + ) + if target is None: + raise SelectorInputError( + code, f"promotion_path[{index}] is not policy-owned" + ) + path_targets.append(target) + if (path_targets[0].adapter, path_targets[0].target) not in canon_keys_set: + raise SelectorInputError( + code, "promotion_path must begin at the initial policy target" + ) + for previous, current in zip(path_targets, path_targets[1:]): + if policy.promotion_target(previous) != current: + raise SelectorInputError( + code, "promotion_path contains a non-adjacent transition" + ) + if path_targets[-1] != matching_cand: + raise SelectorInputError( + code, "promotion_path tail does not match selected target" + ) + if "used_candidates" in prior_decision: + raise SelectorInputError( + code, "promotion decision must not carry failover used_candidates" + ) + return + + if "used_candidates" in prior_decision: + used = prior_decision["used_candidates"] + if not isinstance(used, list): + raise SelectorInputError(code, "prior_decision.used_candidates must be a list") + + used_keys = [] + for idx, entry in enumerate(used): + if not isinstance(entry, dict): + raise SelectorInputError( + code, f"prior_decision.used_candidates[{idx}] must be an object" + ) + u_key = (entry.get("adapter"), entry.get("target")) + if u_key not in canon_keys_set: + raise SelectorInputError( + code, + f"prior_decision.used_candidates[{idx}] {u_key} is not in canonical policy targets {canon_keys_set}", + ) + used_keys.append(u_key) + + if len(used_keys) != len(set(used_keys)): + raise SelectorInputError( + code, "prior_decision.used_candidates contains duplicate targets" + ) + + indices = [canon_keys_list.index(k) for k in used_keys] + if indices != sorted(indices): + raise SelectorInputError( + code, "prior_decision.used_candidates order does not match candidate rank order" + ) + + if used_keys and sel_key != used_keys[-1]: + raise SelectorInputError( + code, + f"prior_decision.selected {sel_key} does not match tail of used_candidates {used_keys[-1]}", + ) + else: + prior_cands = prior_decision.get("candidates", []) + eligible_cands = [ + (c.get("adapter"), c.get("target")) + for c in prior_cands + if isinstance(c, dict) and c.get("eligibility") == "eligible" + ] + if eligible_cands and sel_key != eligible_cands[0]: + raise SelectorInputError( + code, + f"prior_decision.selected {sel_key} does not match first eligible candidate {eligible_cands[0]} when used_candidates is absent", + ) + + +def _validate_prior_candidate_identity( + prior_decision: dict, + *, + stage: str, + lane: str, + grade: int, +) -> None: + code = "malformed_prior_decision" + decision_info = prior_decision.get("decision") + if not isinstance(decision_info, dict): + raise SelectorInputError(code, "prior_decision.decision must be an object") + + eval_str = decision_info.get("evaluated_at") + if not isinstance(eval_str, str): + raise SelectorInputError(code, "prior_decision.decision.evaluated_at must be a string") + + try: + prior_eval_at = datetime.fromisoformat(eval_str) + except (ValueError, TypeError) as exc: + raise SelectorInputError( + code, f"prior_decision.decision.evaluated_at is not a valid ISO datetime: {eval_str!r}" + ) from exc + + if prior_eval_at.tzinfo is None or prior_eval_at.utcoffset() is None: + raise SelectorInputError( + code, f"prior_decision.decision.evaluated_at must be timezone-aware: {eval_str!r}" + ) + + try: + canonical_decision = policy.select_policy( + stage=stage, lane=lane, grade=grade, evaluated_at=prior_eval_at + ) + except ValueError as exc: + raise SelectorInputError(code, str(exc)) from exc + + if decision_info.get("rule_id") != canonical_decision.rule_id: + raise SelectorInputError( + code, + f"prior_decision.decision.rule_id ({decision_info.get('rule_id')!r}) " + f"does not match canonical policy ({canonical_decision.rule_id!r})", + ) + if decision_info.get("policy_priority") != canonical_decision.policy_priority: + raise SelectorInputError( + code, + f"prior_decision.decision.policy_priority ({decision_info.get('policy_priority')!r}) " + f"does not match canonical policy ({canonical_decision.policy_priority!r})", + ) + if list(decision_info.get("reason_codes", [])) != list(canonical_decision.reason_codes): + raise SelectorInputError( + code, + f"prior_decision.decision.reason_codes ({decision_info.get('reason_codes')!r}) " + f"does not match canonical policy ({list(canonical_decision.reason_codes)!r})", + ) + if decision_info.get("time_window") != canonical_decision.time_window: + raise SelectorInputError( + code, + f"prior_decision.decision.time_window ({decision_info.get('time_window')!r}) " + f"does not match canonical policy ({canonical_decision.time_window!r})", + ) + + canonical_targets = canonical_decision.candidates + prior_candidates = prior_decision.get("candidates") + if not isinstance(prior_candidates, list) or len(prior_candidates) != len(canonical_targets): + raise SelectorInputError( + code, + f"prior_decision.candidates length ({len(prior_candidates) if isinstance(prior_candidates, list) else 0}) " + f"does not match canonical policy candidates length ({len(canonical_targets)})", + ) + + for idx, (p_cand, c_target) in enumerate(zip(prior_candidates, canonical_targets)): + if not isinstance(p_cand, dict): + raise SelectorInputError(code, f"prior_decision.candidates[{idx}] must be an object") + if ( + p_cand.get("adapter") != c_target.adapter + or p_cand.get("target") != c_target.target + or p_cand.get("execution_class") != c_target.execution_class + or p_cand.get("selfcheck_required") != c_target.selfcheck_required + ): + raise SelectorInputError( + code, + f"prior_decision.candidates[{idx}] identity ({p_cand.get('adapter')}, {p_cand.get('target')}) " + f"does not match canonical policy candidate ({c_target.adapter}, {c_target.target})", + ) + + _validate_selected_and_used_history(prior_decision, canonical_targets) + + +def _resume( + prior_decision: dict | None, + *, + work_unit_id: str, + stage: str, + lane: str, + grade: int, +) -> dict: + if prior_decision is None: + raise SelectorInputError( + "resume_requires_prior_decision", + "resume transition requires prior_decision", + ) + prior_decision = _validate_prior_decision(prior_decision) + for key, expected in ( + ("work_unit_id", work_unit_id), + ("stage", stage), + ("lane", lane), + ("grade", grade), + ): + if prior_decision[key] != expected: + raise SelectorInputError( + "resume_work_unit_mismatch", + f"prior_decision {key}={prior_decision[key]!r} != {expected!r}", + ) + _validate_prior_candidate_identity(prior_decision, stage=stage, lane=lane, grade=grade) + selected = prior_decision["selected"] + decision = dict(prior_decision["decision"]) + decision["pinned"] = True + target_ref = {"adapter": selected["adapter"], "target": selected["target"]} + return { + "schema_version": SCHEMA_VERSION, + "work_unit_id": work_unit_id, + "stage": stage, + "lane": lane, + "grade": grade, + "selected": selected, + "candidates": prior_decision["candidates"], + "decision": decision, + "quota": prior_decision["quota"], + **({"used_candidates": _validate_used_candidates(prior_decision.get("used_candidates"))} if "used_candidates" in prior_decision else {}), + **({"promotion_path": prior_decision["promotion_path"]} if "promotion_path" in prior_decision else {}), + "transition": { + "previous_target": target_ref, + "next_target": dict(target_ref), + "trigger": "resume", + "context_transfer": "none", + }, + } + + +def _target_ref(candidate: dict) -> dict: + return {"adapter": candidate["adapter"], "target": candidate["target"]} + + +def _validate_used_candidates(value: object) -> list[dict]: + if value is None: + return [] + if not isinstance(value, list): + raise SelectorInputError("malformed_prior_decision", "used_candidates must be a list") + refs = [] + for index, entry in enumerate(value): + if not isinstance(entry, dict): + raise SelectorInputError("malformed_prior_decision", f"used_candidates[{index}] must be an object") + adapter, target = entry.get("adapter"), entry.get("target") + if not isinstance(adapter, str) or not adapter or not isinstance(target, str) or not target: + raise SelectorInputError("malformed_prior_decision", f"used_candidates[{index}] needs adapter and target") + refs.append({"adapter": adapter, "target": target}) + return refs + + +def _failover( + prior_decision: dict | None, *, work_unit_id: str, stage: str, lane: str, + grade: int, evaluated_at: datetime, quota_snapshot: dict | None, + quota_probe_command: str, failure_class: str | None, +) -> dict: + if failure_class not in _QUALIFIED_FAILOVER_FAILURES: + raise SelectorInputError( + "unqualified_failover_trigger", + f"failover requires one of {sorted(_QUALIFIED_FAILOVER_FAILURES)}", + ) + if prior_decision is None: + raise SelectorInputError("failover_requires_prior_decision", "failover transition requires prior_decision") + prior = _validate_prior_decision(prior_decision) + for key, expected in (("work_unit_id", work_unit_id), ("stage", stage), ("lane", lane), ("grade", grade)): + if prior[key] != expected: + raise SelectorInputError("failover_work_unit_mismatch", f"prior_decision {key}={prior[key]!r} != {expected!r}") + _validate_prior_candidate_identity(prior, stage=stage, lane=lane, grade=grade) + previous = _target_ref(prior["selected"]) + previous_index = next(index for index, candidate in enumerate(prior["candidates"]) if _target_ref(candidate) == previous) + used = _validate_used_candidates(prior.get("used_candidates")) + if previous not in used: + used.append(previous) + used_set = {(entry["adapter"], entry["target"]) for entry in used} + selected_candidate = None + selected_probed_snapshot = None + candidates = [] + for index, candidate in enumerate(prior["candidates"]): + current = dict(candidate) + current_snapshot = None + if current["execution_class"] != "local_model": + if failure_class == "provider-quota" and _target_ref(current) == previous: + status = "exhausted" + elif quota_snapshot is not None: + current_snapshot = quota_snapshot + status = _snapshot_status(type("Target", (), current)(), quota_snapshot) + else: + cand_obj = type("Target", (), current)() + probe_spec = policy.quota_probe_spec(cand_obj) + if probe_spec is not None: + sn = probe_candidate_quota( + target=current["target"], + adapter=current["adapter"], + required_caps=probe_spec.required_caps, + checked_at=evaluated_at, + quota_probe_command=quota_probe_command, + ) + current_snapshot = sn + status = _snapshot_status(cand_obj, sn) + else: + status = "unknown" + current["quota_status"] = status + + current["eligibility"] = "ineligible" if status == "exhausted" else "eligible" + current["rejection_reason"] = "quota_exhausted" if status == "exhausted" else None + candidates.append(current) + key = (current["adapter"], current["target"]) + if index > previous_index and key not in used_set and current["eligibility"] == "eligible" and selected_candidate is None: + selected_candidate = current + selected_probed_snapshot = current_snapshot + if selected_candidate is None: + raise SelectorInputError("no_failover_candidate", "no unused eligible candidate remains for this work unit") + selected = {field: selected_candidate[field] for field in ("adapter", "target", "execution_class", "selfcheck_required")} + next_target = _target_ref(selected) + used.append(next_target) + decision = dict(prior["decision"]) + decision["pinned"] = True + selected_target = type("Target", (), selected)() + return { + "schema_version": SCHEMA_VERSION, "work_unit_id": work_unit_id, "stage": stage, + "lane": lane, "grade": grade, "selected": selected, "candidates": candidates, + "decision": decision, + "quota": _selected_quota( + selected_target, + quota_snapshot, + quota_probe_command, + selected_probed_snapshot, + ), + "used_candidates": used, + "transition": { + "previous_target": previous, "next_target": next_target, + "trigger": failure_class, "context_transfer": "logical", + "evaluated_at": evaluated_at.astimezone(policy.KST).isoformat(), + }, + } + + +def _promotion( + prior_decision: dict | None, + *, + work_unit_id: str, + stage: str, + lane: str, + grade: int, + evaluated_at: datetime, + quota_snapshot: dict | None, + quota_probe_command: str, + failure_class: str | None, +) -> dict: + if failure_class not in _QUALIFIED_PROMOTION_FAILURES: + raise SelectorInputError( + "unqualified_promotion_trigger", + "promotion requires one of " + f"{sorted(_QUALIFIED_PROMOTION_FAILURES)}", + ) + if prior_decision is None: + raise SelectorInputError( + "promotion_requires_prior_decision", + "promotion transition requires prior_decision", + ) + prior = _validate_prior_decision(prior_decision) + for key, expected in ( + ("work_unit_id", work_unit_id), + ("stage", stage), + ("lane", lane), + ("grade", grade), + ): + if prior[key] != expected: + raise SelectorInputError( + "promotion_work_unit_mismatch", + f"prior_decision {key}={prior[key]!r} != {expected!r}", + ) + _validate_prior_candidate_identity( + prior, stage=stage, lane=lane, grade=grade + ) + if len(prior["candidates"]) != 1: + raise SelectorInputError( + "no_promotion_target", + "multi-candidate policy routes use failover instead of promotion", + ) + current = policy.canonical_target( + prior["selected"]["adapter"], prior["selected"]["target"] + ) + promoted = policy.promotion_target(current) if current is not None else None + if promoted is None: + raise SelectorInputError( + "no_promotion_target", + "no unused canonical promotion target remains for this work unit", + ) + previous_target = {"adapter": current.adapter, "target": current.target} + next_target = {"adapter": promoted.adapter, "target": promoted.target} + promotion_path = list(prior.get("promotion_path", [previous_target])) + if not promotion_path or promotion_path[-1] != previous_target: + raise SelectorInputError( + "malformed_prior_decision", + "promotion_path tail does not match the selected target", + ) + promotion_path.append(next_target) + decision = dict(prior["decision"]) + decision["pinned"] = True + return { + "schema_version": SCHEMA_VERSION, + "work_unit_id": work_unit_id, + "stage": stage, + "lane": lane, + "grade": grade, + "selected": { + "adapter": promoted.adapter, + "target": promoted.target, + "execution_class": promoted.execution_class, + "selfcheck_required": promoted.selfcheck_required, + }, + "candidates": prior["candidates"], + "decision": decision, + "promotion_path": promotion_path, + "quota": _selected_quota( + promoted, quota_snapshot, quota_probe_command + ), + "transition": { + "kind": "promotion", + "previous_target": previous_target, + "next_target": next_target, + "trigger": failure_class, + "context_transfer": "logical", + "evaluated_at": evaluated_at.astimezone(policy.KST).isoformat(), + }, + } + + +def select_execution_target( + task_file: Path, + *, + stage: str | None = None, + evaluated_at: datetime, + transition: str = "initial", + prior_decision: dict | None = None, + quota_snapshot: dict | None = None, + quota_probe_command: str = DEFAULT_QUOTA_PROBE_COMMAND, + failure_class: str | None = None, +) -> dict: + """Return the stable JSON-serializable selector decision for one call.""" + + kind, lane, grade = _parse_filename(task_file) + prefix_stage = _STAGE_BY_KIND[kind] + if stage is not None and stage != prefix_stage: + raise SelectorInputError( + "stage_mismatch", + f"explicit stage {stage!r} conflicts with filename stage {prefix_stage!r}", + ) + resolved_stage = stage or prefix_stage + + header = _parse_header(task_file) + work_unit_id = _work_unit_id(header) + _validate_evaluated_at(evaluated_at) + quota_snapshot = _validate_quota_snapshot(quota_snapshot) + if not isinstance(quota_probe_command, str) or not quota_probe_command: + raise SelectorInputError( + "invalid_quota_probe_command", + "quota_probe_command must be a non-empty string", + ) + + if transition == "initial": + return _initial( + work_unit_id=work_unit_id, + stage=resolved_stage, + lane=lane, + grade=grade, + evaluated_at=evaluated_at, + quota_snapshot=quota_snapshot, + quota_probe_command=quota_probe_command, + ) + if transition == "resume": + return _resume( + prior_decision, + work_unit_id=work_unit_id, + stage=resolved_stage, + lane=lane, + grade=grade, + ) + if transition == "failover": + return _failover( + prior_decision, work_unit_id=work_unit_id, stage=resolved_stage, + lane=lane, grade=grade, evaluated_at=evaluated_at, + quota_snapshot=quota_snapshot, quota_probe_command=quota_probe_command, + failure_class=failure_class, + ) + if transition == "promotion": + return _promotion( + prior_decision, work_unit_id=work_unit_id, stage=resolved_stage, + lane=lane, grade=grade, evaluated_at=evaluated_at, + quota_snapshot=quota_snapshot, + quota_probe_command=quota_probe_command, + failure_class=failure_class, + ) + raise SelectorInputError( + "invalid_transition", f"unknown transition: {transition!r}" + ) + + +def to_json(payload: dict) -> str: + """Serialize a decision to byte-stable JSON (sorted keys, fixed indent).""" + + return json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + + +def _load_json_arg(value: str | None): + if value is None: + return None + candidate = Path(value) + if candidate.exists(): + text = candidate.read_text(encoding="utf-8") + else: + text = value + return json.loads(text) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Select the deterministic execution target for a task file.", + ) + parser.add_argument("task_file", type=Path) + parser.add_argument("--stage", choices=["worker", "review"]) + parser.add_argument("--evaluated-at") + parser.add_argument( + "--transition", default="initial", choices=sorted(_VALID_TRANSITIONS) + ) + parser.add_argument("--prior-decision") + parser.add_argument("--quota-snapshot") + parser.add_argument("--failure-class") + parser.add_argument( + "--quota-probe-command", default=DEFAULT_QUOTA_PROBE_COMMAND + ) + args = parser.parse_args(argv) + + try: + if args.evaluated_at is not None: + evaluated_at = datetime.fromisoformat(args.evaluated_at) + else: + evaluated_at = datetime.now(policy.KST) + payload = select_execution_target( + args.task_file, + stage=args.stage, + evaluated_at=evaluated_at, + transition=args.transition, + prior_decision=_load_json_arg(args.prior_decision), + quota_snapshot=_load_json_arg(args.quota_snapshot), + quota_probe_command=args.quota_probe_command, + failure_class=args.failure_class, + ) + except SelectorInputError as exc: + json.dump({"error": exc.code, "message": str(exc)}, sys.stderr) + sys.stderr.write("\n") + return 2 + except (ValueError, OSError, json.JSONDecodeError) as exc: + json.dump({"error": "input_error", "message": str(exc)}, sys.stderr) + sys.stderr.write("\n") + return 2 + + sys.stdout.write(to_json(payload) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_dispatch.py b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_dispatch.py new file mode 100644 index 0000000..35786b3 --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_dispatch.py @@ -0,0 +1,12590 @@ +import asyncio +import copy +from datetime import datetime, timezone, timedelta +import importlib.util +import inspect +import io +import json +import os +import re +import signal +import subprocess +import sys +import tempfile +import time +import unittest +import uuid +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + + +SCRIPT = Path(__file__).parents[1] / "scripts" / "dispatch.py" +SPEC = importlib.util.spec_from_file_location("agent_task_dispatch", SCRIPT) +assert SPEC and SPEC.loader +dispatch = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = dispatch +SPEC.loader.exec_module(dispatch) + + +def pi_session_jsonl(events, version=dispatch.PI_SESSION_SCHEMA_VERSION): + values = [ + { + "type": "session", + "version": version, + "id": "test-session", + "timestamp": "2026-07-25T00:00:00.000Z", + "cwd": "/tmp/test", + } + ] + parent_id = None + for index, event in enumerate(events): + value = dict(event) + value.setdefault("id", f"entry-{index}") + value.setdefault("parentId", parent_id) + values.append(value) + parent_id = value["id"] + return "".join(json.dumps(value) + "\n" for value in values) + + +def write_legacy_quota_attempts( + runs: Path, + task: dispatch.Task, + *, + cli: str = "claude", + model: str = "claude-opus-4-8", + reasoning_effort: str | None = "xhigh", + dispatcher_sha256: str = "older-dispatcher", +) -> list[Path]: + locators = [] + event = json.dumps( + { + "type": "rate_limit_event", + "rate_limit_info": {"status": "rejected"}, + } + ) + for attempt_number in range(dispatch.RECOVERY_FAILURE_LIMIT): + attempt = runs / f"legacy-attempt-{attempt_number}" + attempt.mkdir(parents=True) + locator = attempt / "locator.json" + locator.write_text( + json.dumps( + { + "task": task.name, + "plan_number": dispatch.plan_number(task), + "role": "worker", + "attempt": attempt_number, + "status": "failed", + "failure_class": "generic-error", + "cli": cli, + "model": model, + "reasoning_effort": reasoning_effort, + "dispatcher_source_sha256": dispatcher_sha256, + } + ), + encoding="utf-8", + ) + if cli == "agy": + (attempt / "stream.log").write_text( + "[stdout] AGY request failed\n", + encoding="utf-8", + ) + (attempt / "agy-cli.log").write_text( + ( + "rpc failed: code = ResourceExhausted " + "status=RESOURCE_EXHAUSTED HTTP 429 quota exceeded\n" + ), + encoding="utf-8", + ) + else: + (attempt / "stream.log").write_text( + f"[stdout] {event}\n", + encoding="utf-8", + ) + locators.append(locator) + return locators + + +class CommandConstructionTest(unittest.TestCase): + def test_agy_print_receives_prompt_before_timeout_option(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + prompt = "Implement the active plan." + command = dispatch.build_command( + dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)"), + prompt, workspace, "test-session", workspace / "attempt", + ) + + self.assertEqual(command[:5], ["agy", "--print", prompt, "--print-timeout", "8h"]) + self.assertEqual( + command[-2:], ["--log-file", str(workspace / "attempt" / "agy-cli.log")] + ) + + +class TaskStageTest(unittest.TestCase): + def make_task(self, root: Path, review_text: str = ""): + plan = root / "PLAN-local-G05.md" + review = root / "CODE_REVIEW-local-G05.md" + target = (root / "src" / "test.py").resolve() + plan.write_text( + "\n" + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + f"| `{target}` | TEST-1 |\n", + encoding="utf-8", + ) + review.write_text("\n" + review_text, encoding="utf-8") + return dispatch.Task( + name="test", + directory=root, + plan=plan, + review=review, + user_review=None, + recovery=False, + write_set={str(target)}, + write_set_known=True, + lane="local", + grade=5, + ) + + + @staticmethod + def blocking_user_review_text(): + return ( + "# User Review Required - test\n\n" + "## ์ƒํƒœ\n\nUSER_REVIEW\n\n" + "## ์‚ฌ์œ \n\n" + "- ์œ ํ˜•: milestone-lock\n" + "- ์—ฐ๊ฒฐ ๋Œ€์ƒ: agent-roadmap/phase/p/milestones/m.md\n\n" + "## ์ฐจ๋‹จ ๊ทผ๊ฑฐ\n\n" + "- ์ฐจ๋‹จ ํŒ๋‹จ ๊ทผ๊ฑฐ: API ownership decision blocks implementation.\n\n" + "## ์—ฐ๊ฒฐ ๊ฒฐ์ • ํ•„์š”\n\n" + "- [ ] API ownership ์„ ํƒ\n\n" + "## ์žฌ๊ฐœ ์กฐ๊ฑด\n\n" + "- Milestone ๊ฒฐ์ • ๋ฐ˜์˜ ํ›„ ์žฌ๊ฐœ\n" + ) + + @staticmethod + def blocking_external_user_review_text(): + return ( + "# User Review Required - test\n\n" + "## ์ƒํƒœ\n\nUSER_REVIEW\n\n" + "## ์‚ฌ์œ \n\n" + "- ์œ ํ˜•: external-execution\n" + "- ์—ฐ๊ฒฐ ๋Œ€์ƒ: ssh runner@example.test:/srv/agent-work/test-workspace\n\n" + "## ์ฐจ๋‹จ ๊ทผ๊ฑฐ\n\n" + "- ์ฐจ๋‹จ ํŒ๋‹จ ๊ทผ๊ฑฐ: Required dev smoke needs a user-controlled runner and no authorized SSH credential is available.\n\n" + "## ์‚ฌ์šฉ์ž ์กฐ์น˜ ๋˜๋Š” ๊ฒฐ์ •\n\n" + "- [ ] Grant runner access or provide the required sanitized smoke evidence.\n\n" + "## ์žฌ๊ฐœ ์กฐ๊ฑด\n\n" + "- Verify SSH access or the supplied evidence before resuming review.\n" + ) + + @staticmethod + def blocking_english_external_user_review_text(): + return ( + "# User Review Required - test\n\n" + "## Status\n\nUSER_REVIEW\n\n" + "## Reason\n\n" + "- Type: external-execution\n" + "- Target: ssh runner@example.test:/srv/agent-work/test-workspace\n\n" + "## Blocking Evidence\n\n" + "- Blocking rationale: Required dev smoke needs a user-controlled runner and renewed authorization.\n\n" + "## Required User Action\n\n" + "- [ ] Authorize one replacement live invocation.\n\n" + "## Resume Condition\n\n" + "- Verify idle provider capacity before resuming.\n" + ) + + def test_default_or_arbitrary_status_text_does_not_start_review(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task( + root, + "## ์‚ฌ์šฉ์ž ๋ฆฌ๋ทฐ ์š”์ฒญ\n- ์ƒํƒœ: ์—†์Œ\n" + "## unrelated\n- ์ƒํƒœ: ํ™•์ธ ํ•„์š”\n", + ) + self.assertEqual(dispatch.task_stage(task, {}), "worker") + + def test_verdict_text_outside_official_section_does_not_start_review(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task( + root, + "## ๊ฒ€์ฆ ๊ฒฐ๊ณผ\n" + "๋ช…๋ น ์ถœ๋ ฅ ์˜ˆ์‹œ: ์ข…ํ•ฉ ํŒ์ •: PASS\n", + ) + self.assertEqual(dispatch.task_stage(task, {}), "worker") + + def test_exact_official_verdict_section_starts_review_recovery(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task( + root, + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n" + "- **์ข…ํ•ฉ ํŒ์ •**: WARN\n", + ) + self.assertEqual(dispatch.task_stage(task, {}), "review") + + def test_only_explicit_user_review_file_stops_the_loop(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root, "- ์ƒํƒœ: ์—†์Œ\n") + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_user_review_text(), encoding="utf-8" + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "user-review") + + def test_external_execution_user_review_stops_the_loop(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_external_user_review_text(), encoding="utf-8" + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "user-review") + + def test_english_external_execution_user_review_stops_the_loop(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_english_external_user_review_text(), + encoding="utf-8", + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "user-review") + + def test_user_review_rejects_mixed_language_schema(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_english_external_user_review_text().replace( + "## Status\n\nUSER_REVIEW\n\n", + "## Status\n\nUSER_REVIEW\n\n## ์ƒํƒœ\n\nUSER_REVIEW\n\n", + ), + encoding="utf-8", + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "blocked") + + def test_external_execution_user_review_requires_concrete_target(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_external_user_review_text().replace( + "ssh runner@example.test:/srv/agent-work/test-workspace", + "์—†์Œ", + ), + encoding="utf-8", + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "blocked") + + def test_user_review_rejects_multiple_gate_types(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_external_user_review_text().replace( + "- ์œ ํ˜•: external-execution\n", + "- ์œ ํ˜•: external-execution\n- ์œ ํ˜•: milestone-lock\n", + ), + encoding="utf-8", + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "blocked") + + def test_user_review_without_blocking_contract_is_state_blocked(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + "## ์ƒํƒœ\n\nUSER_REVIEW\n", encoding="utf-8" + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "blocked") + + def test_user_review_with_active_pair_is_state_blocked(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + self.blocking_user_review_text(), encoding="utf-8" + ) + self.assertEqual(dispatch.task_stage(task, {}), "blocked") + + def test_user_review_only_directory_without_logs_is_readable(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + directory = workspace / "agent-task" / "group" / "01_gate" + directory.mkdir(parents=True) + user_review = directory / "USER_REVIEW.md" + user_review.write_text( + self.blocking_user_review_text(), encoding="utf-8" + ) + task = dispatch.read_task_directory(workspace, directory) + self.assertIsNotNone(task) + self.assertEqual(dispatch.task_stage(task, {}), "user-review") + + def test_user_review_none_values_do_not_form_a_valid_stop(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + task.user_review = root / "USER_REVIEW.md" + task.user_review.write_text( + "## ์ƒํƒœ\n\nUSER_REVIEW\n\n" + "## ์‚ฌ์œ \n\n" + "- ์œ ํ˜•: milestone-lock\n" + "- ์—ฐ๊ฒฐ ๋Œ€์ƒ: ์—†์Œ\n\n" + "## ์ฐจ๋‹จ ๊ทผ๊ฑฐ\n\n" + "- ์ฐจ๋‹จ ํŒ๋‹จ ๊ทผ๊ฑฐ: ์—†์Œ\n\n" + "## ์—ฐ๊ฒฐ ๊ฒฐ์ • ํ•„์š”\n\n" + "- [ ] ์—†์Œ\n\n" + "## ์žฌ๊ฐœ ์กฐ๊ฑด\n\n" + "- ์—†์Œ\n", + encoding="utf-8", + ) + task.plan = None + task.review = None + task.recovery = True + self.assertEqual(dispatch.task_stage(task, {}), "blocked") + + def test_completed_implementation_checklist_does_not_bypass_worker(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task( + root, + "- [x] CODE_REVIEW-*-G??.md์˜ ๊ตฌํ˜„ ์—์ด์ „ํŠธ ์†Œ์œ  ์„น์…˜์„ ์ฑ„์šด๋‹ค.\n", + ) + self.assertEqual(dispatch.task_stage(task, {}), "worker") + + def test_pi_worker_success_requires_selfcheck_before_review(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + local_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + cloud_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + self.assertEqual( + dispatch.task_stage( + task, + {"worker_done": True, "selfcheck_done": False, "completing_decision": local_decision}, + ), + "selfcheck", + ) + self.assertEqual( + dispatch.task_stage( + task, + {"worker_done": True, "selfcheck_done": True, "completing_decision": local_decision}, + ), + "review", + ) + self.assertEqual( + dispatch.task_stage( + task, + {"worker_done": True, "selfcheck_done": False, "completing_decision": cloud_decision}, + ), + "review", + ) + + def test_local_route_grade_boundaries(self): + with tempfile.TemporaryDirectory() as temporary: + task = self.make_task(Path(temporary)) + expected = { + 5: ("pi", "ornith:35b", True), + 6: ("pi", "ornith:35b", True), + 9: ("claude", "claude-opus-4-8", False), + 10: ("claude", "claude-opus-4-8", False), + } + for grade, (cli, model, local_pi) in expected.items(): + with self.subTest(grade=grade): + assert task.plan is not None + graded_plan = task.plan.with_name(f"PLAN-local-G{grade:02d}.md") + task.plan.rename(graded_plan) + task.plan = graded_plan + task.grade = grade + decision = dispatch.select_execution_decision(task, stage="worker") + spec = dispatch.agent_spec_from_decision(decision) + self.assertEqual(spec.cli, cli) + self.assertEqual(spec.model, model) + self.assertEqual(spec.local_pi, local_pi) + + def test_local_g07_g08_route_uses_explicit_kst_boundaries(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + nighttime = datetime(2026, 7, 26, 1, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + task = self.make_task(Path(temporary)) + for grade in (7, 8): + assert task.plan is not None + graded_plan = task.plan.with_name(f"PLAN-local-G{grade:02d}.md") + task.plan.rename(graded_plan) + task.plan = graded_plan + task.grade = grade + + day_dec = dispatch.select_execution_decision(task, stage="worker", evaluated_at=daytime) + self.assertEqual(day_dec["selected"]["adapter"], "agy") + self.assertEqual(day_dec["selected"]["target"], "Gemini 3.6 Flash (Medium)") + + night_dec = dispatch.select_execution_decision(task, stage="worker", evaluated_at=nighttime) + self.assertEqual(night_dec["selected"]["adapter"], "pi") + self.assertEqual(night_dec["selected"]["target"], "iop/laguna-s:2.1") + + + + def test_selfcheck_requires_nonempty_checklist_values(self): + with tempfile.TemporaryDirectory() as temporary: + task = self.make_task( + Path(temporary), + "## ๊ตฌํ˜„ ํ•ญ๋ชฉ๋ณ„ ์™„๋ฃŒ ์—ฌ๋ถ€\n\n" + "| ํ•ญ๋ชฉ | ์™„๋ฃŒ ์—ฌ๋ถ€ |\n|---|---|\n| TEST-1 | [ ] |\n\n" + "## ๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ\n\n- [ ] TEST-1\n", + ) + self.assertEqual( + dispatch.implementation_review_errors(task), + ["๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ ๋ฏธ์™„๋ฃŒ"], + ) + + def test_selfcheck_accepts_any_nonempty_checklist_values(self): + with tempfile.TemporaryDirectory() as temporary: + task = self.make_task( + Path(temporary), + "## ๊ตฌํ˜„ ํ•ญ๋ชฉ๋ณ„ ์™„๋ฃŒ ์—ฌ๋ถ€\n\n" + "| ํ•ญ๋ชฉ | ์™„๋ฃŒ ์—ฌ๋ถ€ |\n|---|---|\n| TEST-1 | [ ] |\n\n" + "## ๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ\n\n" + "- [x] TEST-1\n" + "- [v] TEST-2\n" + "- [โœ…] TEST-3\n", + ) + self.assertEqual(dispatch.implementation_review_errors(task), []) + + +class CompletingTargetSelfcheckTest(unittest.IsolatedAsyncioTestCase): + """Verify selfcheck is determined by the completing decision's execution_class. + + - Worker success persists the actual completing decision with execution_class. + - selfcheck schedules exactly once when execution_class=local_model. + - local selfcheck reuses the completing target without re-evaluating selector. + - Geminiโ†’Laguna, Lagunaโ†’Gemini, cloud completions follow the policy. + - Restart does not duplicate selfcheck execution. + - Provider-deny guard prevents actual provider calls during tests. + """ + + _WORK_UNIT_ID = "completing_target_test::plan-0::tag-TEST" + + _CLOUD_CASES = ( + ("agy", "Gemini 3.6 Flash (Medium)"), + ("claude", "claude-opus-4-8"), + ("codex", "gpt-5.6-sol"), + ) + + @classmethod + def make_cloud_decision( + cls, adapter: str, target: str, execution_class: object = "cloud_model", + ) -> dict: + return { + "work_unit_id": cls._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": adapter, + "target": target, + "execution_class": execution_class, + "selfcheck_required": False, + }, + } + + def setUp(self) -> None: + super().setUp() + self._provider_deny = mock.patch.object( + dispatch, "invoke", + new=mock.AsyncMock(side_effect=RuntimeError( + "provider invoke must not be called in selfcheck tests" + )), + ) + self._build_command_deny = mock.patch.object( + dispatch, "build_command", + side_effect=RuntimeError( + "build_command must not be called in selfcheck tests" + ), + ) + self._provider_deny.start() + self._build_command_deny.start() + + def tearDown(self) -> None: + self._provider_deny.stop() + self._build_command_deny.stop() + super().tearDown() + + def make_task(self, workspace: Path, lane: str = "local", grade: int = 8) -> dispatch.Task: + directory = workspace / "agent-task" / "completing_target_test" + directory.mkdir(parents=True, exist_ok=True) + header = f"\n" + (directory / f"PLAN-{lane}-G{grade:02d}.md").write_text( + header + + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + "| `src/completing-target.py` | TEST-1 |\n", + encoding="utf-8", + ) + (directory / f"CODE_REVIEW-{lane}-G{grade:02d}.md").write_text(header, encoding="utf-8") + tasks = dispatch.scan_tasks(workspace, None) + return tasks[0] + + def make_locator(self, workspace: Path, cli: str, model: str) -> Path: + attempt = workspace / f"attempt-{cli}" + attempt.mkdir(parents=True, exist_ok=True) + locator = attempt / "locator.json" + locator.write_text( + json.dumps({"cli": cli, "model": model}), + encoding="utf-8", + ) + return locator + + async def test_worker_persists_actual_completing_decision_and_execution_class(self): + """Worker success records the completing decision, not the initial one. + + Simulates a Geminiโ†’Laguna failover where the worker actually completed + on Laguna. The persisted completing decision must reflect the actual + target (Laguna), not the initial target (Gemini). + """ + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + # Initial decision: cloud (Gemini) + initial_decision = { + "schema_version": "1.0", + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + "decision": { + "rule_id": "test-rule", + "policy_priority": 0, + "reason_codes": [], + "evaluated_at": "2026-07-26T14:00:00+09:00", + "timezone": "Asia/Seoul", + "time_window": {}, + }, + "transition": {"trigger": "initial"}, + "stage": "worker", + "lane": "local", + "grade": 8, + "candidates": [], + "quota": {}, + } + # Simulate failover: worker completed on Laguna + laguna_decision = { + "schema_version": "1.0", + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + "decision": { + "rule_id": "test-rule", + "policy_priority": 0, + "reason_codes": [], + "evaluated_at": "2026-07-26T14:00:00+09:00", + "timezone": "Asia/Seoul", + "time_window": {}, + }, + "transition": {"trigger": "failover"}, + "stage": "worker", + "lane": "local", + "grade": 8, + "candidates": [], + "quota": {}, + } + loc_laguna = self.make_locator(workspace, "pi", "laguna-s:2.1") + + # Persist the completing decision to execution_decisions["worker"] + # so _mark_worker_done can find it as the authoritative source. + store.task_state(task) # initialize state + store.data["tasks"][task.name]["execution_decisions"]["worker"] = laguna_decision + store.save() + + def mock_persisted_execution_decision(store_obj, task_obj, *, stage, **kwargs): + if stage == "worker": + return laguna_decision, dispatch.AgentSpec( + "pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True + ) + return initial_decision, dispatch.AgentSpec( + "agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)" + ) + + with ( + mock.patch.object( + dispatch, "persisted_execution_decision", + side_effect=mock_persisted_execution_decision, + ), + mock.patch.object( + dispatch, "run_escalating", + new=mock.AsyncMock(return_value=(True, loc_laguna)), + ), + ): + await dispatch.run_worker( + workspace, store, task + ) + + state = store.task_state(task) + self.assertTrue(state["worker_done"]) + self.assertEqual(state["execution_class"], "local_model") + self.assertFalse(state["selfcheck_done"]) + completing = state["completing_decision"] + self.assertEqual(completing["selected"]["adapter"], "pi") + self.assertEqual(completing["selected"]["target"], "iop/laguna-s:2.1") + self.assertEqual(completing["selected"]["execution_class"], "local_model") + self.assertTrue(completing["selected"]["selfcheck_required"]) + # Verify stage is selfcheck, not review + self.assertEqual(dispatch.task_stage(task, state), "selfcheck") + finally: + store.close() + + async def test_local_completing_decision_triggers_selfcheck(self): + """execution_class=local_model schedules exactly one selfcheck.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + local_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + loc_laguna = self.make_locator(workspace, "pi", "laguna-s:2.1") + + # Manually set worker_done with local completing decision + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + completing_decision=local_decision, + execution_class="local_model", + selfcheck_done=False, + blocked=None, + ) + + state = store.task_state(task) + self.assertEqual(dispatch.task_stage(task, state), "selfcheck") + self.assertTrue( + dispatch.completing_decision_requires_selfcheck(state) + ) + + # Run selfcheck once + with ( + mock.patch.object( + dispatch, "run_escalating", + new=mock.AsyncMock(return_value=(True, loc_laguna)), + ), + mock.patch.object( + dispatch, "implementation_review_errors", + return_value=[], + ), + ): + await dispatch.run_selfcheck( + workspace, store, task + ) + + state2 = store.task_state(task) + self.assertTrue(state2["selfcheck_done"]) + self.assertEqual(dispatch.task_stage(task, state2), "review") + finally: + store.close() + + async def test_cloud_completing_decision_skips_selfcheck(self): + """execution_class=cloud_model skips selfcheck entirely.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + cloud_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + store.update_task( + task, + worker_done=True, + worker_cli="claude", + worker_model="claude-opus-4-8", + completing_decision=cloud_decision, + execution_class="cloud_model", + selfcheck_done=True, + blocked=None, + ) + + state = store.task_state(task) + self.assertFalse( + dispatch.completing_decision_requires_selfcheck(state) + ) + self.assertEqual(dispatch.task_stage(task, state), "review") + finally: + store.close() + + async def test_selfcheck_reuses_completing_target_no_selector_call(self): + """Selfcheck uses the completing decision's target, not re-evaluating selector.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + local_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + loc_laguna = self.make_locator(workspace, "pi", "laguna-s:2.1") + + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + completing_decision=local_decision, + execution_class="local_model", + selfcheck_done=False, + blocked=None, + ) + + selector_calls = [] + with ( + mock.patch.object( + dispatch, "persisted_execution_decision", + side_effect=lambda *a, **kw: selector_calls.append(1) or ( + {}, dispatch.AgentSpec("pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + ), + ), + mock.patch.object( + dispatch, "run_escalating", + new=mock.AsyncMock(return_value=(True, loc_laguna)), + ), + mock.patch.object( + dispatch, "implementation_review_errors", + return_value=[], + ), + ): + await dispatch.run_selfcheck( + workspace, store, task + ) + + # persisted_execution_decision must NOT be called during selfcheck + self.assertEqual(len(selector_calls), 0) + state = store.task_state(task) + self.assertTrue(state["selfcheck_done"]) + finally: + store.close() + + async def test_restart_does_not_duplicate_selfcheck(self): + """After restart, already-completed selfcheck is not re-executed.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + local_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + completing_decision=local_decision, + execution_class="local_model", + selfcheck_done=True, + blocked=None, + ) + + state = store.task_state(task) + self.assertEqual(dispatch.task_stage(task, state), "review") + # selfcheck is required for local_model but already completed + self.assertTrue( + dispatch.completing_decision_requires_selfcheck(state) + ) + self.assertTrue(state["selfcheck_done"]) + finally: + store.close() + + async def test_identity_mismatch_fails_closed(self): + """Missing or malformed completing decision blocks selfcheck.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + # No completing_decision set + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + selfcheck_done=False, + blocked=None, + ) + + with mock.patch.object( + dispatch, "run_escalating", new=mock.AsyncMock() + ) as run_escalating: + await dispatch.run_selfcheck( + workspace, store, task + ) + + self.assertEqual(run_escalating.await_count, 0) + state = store.task_state(task) + self.assertIn("completing decision์ด ์—†์–ด", state["blocked"]) + finally: + store.close() + + async def test_identity_mismatch_decision_blocked_at_scheduler(self): + """worker_done=True with missing completing decision blocks at scheduler entry.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + # worker_done=True but no completing_decision + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + selfcheck_done=False, + blocked=None, + ) + + state = store.task_state(task) + # Should be blocked, not review + self.assertEqual(dispatch.task_stage(task, state), "blocked") + self.assertFalse( + dispatch.completing_decision_requires_selfcheck(state) + ) + finally: + store.close() + + async def test_identity_mismatch_malformed_decision_blocked(self): + """worker_done=True with malformed completing decision blocks at scheduler.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + # completing_decision with invalid schema + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + completing_decision={"selected": None}, + selfcheck_done=False, + blocked=None, + ) + + state = store.task_state(task) + self.assertEqual(dispatch.task_stage(task, state), "blocked") + finally: + store.close() + + async def test_canonical_model_command_generation(self): + """Selfcheck spec.model is normalized (iop/ prefix stripped) for command generation.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + local_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + + spec = dispatch._spec_from_completing_decision(local_decision) + + # Model should be normalized (iop/ prefix stripped) + self.assertEqual(spec.model, "laguna-s:2.1") + # Display should preserve canonical identity + self.assertEqual(spec.display, "pi/iop/laguna-s:2.1") + # local_pi should be True + self.assertTrue(spec.local_pi) + # adapter should be pi + self.assertEqual(spec.cli, "pi") + + # Temporarily disable the build_command deny guard for this test + self._build_command_deny.stop() + try: + # Verify build_command generates correct command + command = dispatch.build_command( + spec, + "test prompt", + workspace, + "test-session", + workspace / "attempt", + ) + self.assertIn("--provider", command) + self.assertIn("iop", command) + self.assertIn("--model", command) + # Model should be laguna-s:2.1 (not iop/laguna-s:2.1) + model_idx = command.index("--model") + self.assertEqual(command[model_idx + 1], "laguna-s:2.1") + finally: + self._build_command_deny.start() + finally: + store.close() + + async def test_selector_probe_not_invoked_during_selfcheck(self): + """Selfcheck does not call selector or quota probe.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + local_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + loc_laguna = self.make_locator(workspace, "pi", "laguna-s:2.1") + + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + completing_decision=local_decision, + execution_class="local_model", + selfcheck_done=False, + blocked=None, + ) + + selector_select_calls = [] + quota_probe_calls = [] + + def mock_select(*args, **kwargs): + selector_select_calls.append(1) + raise RuntimeError("selector must not be called") + + def mock_quota_probe(*args, **kwargs): + quota_probe_calls.append(1) + raise RuntimeError("quota probe must not be called") + + # Patch the selector module directly + selector_module = dispatch._selector_module() + with ( + mock.patch.object( + dispatch, "run_escalating", + new=mock.AsyncMock(return_value=(True, loc_laguna)), + ), + mock.patch.object( + dispatch, "implementation_review_errors", + return_value=[], + ), + mock.patch.object( + selector_module.policy, "select_policy", + side_effect=mock_select, + ), + mock.patch.object( + selector_module, "probe_candidate_quota", + side_effect=mock_quota_probe, + ), + ): + await dispatch.run_selfcheck( + workspace, store, task + ) + + self.assertEqual(len(selector_select_calls), 0) + self.assertEqual(len(quota_probe_calls), 0) + state = store.task_state(task) + self.assertTrue(state["selfcheck_done"]) + finally: + store.close() + + def test_completing_decision_requires_selfcheck_matrix(self): + """Matrix: local_modelโ†’True, cloud_modelโ†’False, missingโ†’False.""" + local_state = { + "completing_decision": { + "selected": { + "execution_class": "local_model", + }, + }, + } + cloud_state = { + "completing_decision": { + "selected": { + "execution_class": "cloud_model", + }, + }, + } + missing_state = {"worker_done": True} + empty_selected_state = { + "completing_decision": {"selected": {}}, + } + self.assertTrue( + dispatch.completing_decision_requires_selfcheck(local_state) + ) + self.assertFalse( + dispatch.completing_decision_requires_selfcheck(cloud_state) + ) + self.assertFalse( + dispatch.completing_decision_requires_selfcheck(missing_state) + ) + self.assertFalse( + dispatch.completing_decision_requires_selfcheck(empty_selected_state) + ) + + def test_completing_decision_validation_matrix(self): + """_completing_decision_is_valid enforces stage, work_unit_id, and schema contract.""" + workspace = Path(tempfile.mkdtemp()) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + try: + valid_pi = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + # Canonical valid cloud decisions for every adapter via class factory + valid_cloud_cases = { + adapter: self.make_cloud_decision(adapter, target) + for adapter, target in self._CLOUD_CASES + } + wrong_stage = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "review", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + wrong_work_unit = { + "work_unit_id": "wrong-task::plan-1::tag-WRONG", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + pi_with_selfcheck_false = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": False, + }, + } + cloud_with_selfcheck_true = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": True, + }, + } + missing = {} + no_selected = {"selected": None} + invalid_execution_class = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "invalid", + "selfcheck_required": True, + }, + } + self.assertTrue( + dispatch._completing_decision_is_valid(task, {"completing_decision": valid_pi}) + ) + # Every canonical cloud adapter must be accepted as valid + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter, target=target): + self.assertTrue( + dispatch._completing_decision_is_valid( + task, {"completing_decision": valid_cloud_cases[adapter]} + ) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": wrong_stage}) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": wrong_work_unit}) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": pi_with_selfcheck_false}) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": cloud_with_selfcheck_true}) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, missing) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, no_selected) + ) + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": invalid_execution_class}) + ) + # cloud adapter + local_model + selfcheck_required=False must be rejected for every adapter + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter, target=target): + invalid = self.make_cloud_decision(adapter, target, "local_model") + self.assertFalse( + dispatch._completing_decision_is_valid( + task, {"completing_decision": invalid} + ) + ) + # non-string selected fields must be rejected (adapter, target, execution_class) + non_string_adapter = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": 123, + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": non_string_adapter}) + ) + non_string_target = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "claude", + "target": None, + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": non_string_target}) + ) + non_string_execution_class = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": None, + "selfcheck_required": False, + }, + } + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": non_string_execution_class}) + ) + # empty string selected fields must be rejected + empty_adapter = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + self.assertFalse( + dispatch._completing_decision_is_valid(task, {"completing_decision": empty_adapter}) + ) + # direct validator must receive full decision shape, not selected sub-dict + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter): + invalid_full = self.make_cloud_decision(adapter, target, "local_model") + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._spec_from_completing_decision(invalid_full) + finally: + import shutil + shutil.rmtree(workspace, ignore_errors=True) + + def test_spec_from_completing_decision_normalizes_pi_target(self): + """_spec_from_completing_decision strips iop/ prefix from model.""" + decision = { + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + spec = dispatch._spec_from_completing_decision(decision) + self.assertEqual(spec.model, "laguna-s:2.1") + self.assertEqual(spec.display, "pi/iop/laguna-s:2.1") + self.assertTrue(spec.local_pi) + + def test_spec_from_completing_decision_rejects_invalid_pi_target(self): + """_spec_from_completing_decision rejects Pi target without iop/ prefix.""" + decision = { + "selected": { + "adapter": "pi", + "target": "laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._spec_from_completing_decision(decision) + + def test_spec_from_completing_decision_rejects_non_pi_cloud(self): + """_spec_from_completing_decision rejects cloud target with selfcheck_required=True.""" + decision = { + "selected": { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": True, + }, + } + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._spec_from_completing_decision(decision) + + def test_mark_worker_done_validates_pi_identity(self): + """_mark_worker_done rejects Pi decision with mismatched worker model.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.task_state(task) # initialize state + store.data["tasks"][task.name]["execution_decisions"]["worker"] = decision + store.save() + # Worker model doesn't match target + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._mark_worker_done( + store, task, + initial_decision=decision, + worker_cli="pi", + worker_model="ornith:35b", + ) + + state = store.task_state(task) + self.assertFalse(state["worker_done"]) + finally: + store.close() + + def test_mark_worker_done_validates_cloud_identity(self): + """_mark_worker_done rejects cloud decision with mismatched worker CLI.""" + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + store.task_state(task) # initialize state + store.data["tasks"][task.name]["execution_decisions"]["worker"] = decision + store.save() + # Worker CLI doesn't match adapter + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._mark_worker_done( + store, task, + initial_decision=decision, + worker_cli="codex", + worker_model="gpt-5.6-sol", + ) + + state = store.task_state(task) + self.assertFalse(state["worker_done"]) + finally: + store.close() + + def test_mark_worker_done_does_not_fallback_from_malformed_persisted_worker_decision( + self, + ): + """Malformed persisted worker decision blocks without falling back to initial. + + When execution_decisions["worker"] exists but is malformed (e.g. missing + selected block), _mark_worker_done must raise rather than silently + reverting to the initial_decision parameter. + """ + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + malformed_decision = {"selected": None} + valid_initial = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.task_state(task) # initialize state + store.data["tasks"][task.name]["execution_decisions"]["worker"] = malformed_decision + store.save() + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._mark_worker_done( + store, task, + initial_decision=valid_initial, + worker_cli="pi", + worker_model="laguna-s:2.1", + ) + + state = store.task_state(task) + self.assertFalse(state["worker_done"]) + self.assertIsNone(state.get("completing_decision")) + finally: + store.close() + + def test_mark_worker_done_rejects_cloud_decision_with_local_execution_class( + self, + ): + """_mark_worker_done rejects cloud adapter + local_model + False for every adapter. + + Regression: cloud adapter with local_model execution_class and + selfcheck_required=False must not be accepted as a valid completing + decision. This prevents worker_done from being recorded with a wrong + execution_class that would route restart to selfcheck. + """ + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + bad_decision = self.make_cloud_decision(adapter, target, "local_model") + store.task_state(task) + store.data["tasks"][task.name]["execution_decisions"]["worker"] = bad_decision + store.save() + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch._mark_worker_done( + store, task, + initial_decision=bad_decision, + worker_cli=adapter, + worker_model=target, + ) + + state = store.task_state(task) + self.assertFalse(state["worker_done"]) + finally: + store.close() + + def test_restart_with_malformed_completed_state_blocks_not_selfcheck(self): + """Restart with malformed completing decision blocks, does not enter selfcheck. + + Regression: when persisted completing_decision has non-string selected + fields or invalid adapter/class/selfcheck combination, task_stage() + must return 'blocked' rather than 'selfcheck'. + """ + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + # worker_done=True with a completing decision that has non-string execution_class + bad_decision = self.make_cloud_decision(adapter, target, 42) + store.update_task( + task, + worker_done=True, + worker_cli=adapter, + worker_model=target, + completing_decision=bad_decision, + execution_class="local_model", + selfcheck_done=False, + blocked=None, + ) + state = store.task_state(task) + self.assertTrue(state["worker_done"]) + # task_stage must not return selfcheck for invalid completing decision + stage = dispatch.task_stage(task, state) + self.assertNotEqual(stage, "selfcheck") + # should be blocked because _completing_decision_is_valid returns False + self.assertEqual(stage, "blocked") + finally: + store.close() + + def test_restart_with_cloud_local_mismatch_blocks(self): + """Restart with cloud adapter + local_model + False blocks for every adapter, not selfcheck. + + Regression: cloud adapter with local_model execution_class must not + route restart to selfcheck. + """ + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + bad_decision = self.make_cloud_decision(adapter, target, "local_model") + store.update_task( + task, + worker_done=True, + worker_cli=adapter, + worker_model=target, + completing_decision=bad_decision, + execution_class="local_model", + selfcheck_done=False, + blocked=None, + ) + state = store.task_state(task) + stage = dispatch.task_stage(task, state) + self.assertNotEqual(stage, "selfcheck") + self.assertEqual(stage, "blocked") + finally: + store.close() + + def test_mark_worker_done_validates_cloud_decision_commit_matrix(self): + """Valid cloud decision commits worker_done=True, selfcheck_done=True, stage=review. + + Regression: every cloud adapter (agy/claude/codex) with a valid + completing decision must record worker completion and advance to + review without selfcheck. This covers the valid half of the + adapter ร— validity ร— consumption path matrix. + """ + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter, target=target): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + valid_decision = self.make_cloud_decision(adapter, target) + store.task_state(task) + store.data["tasks"][task.name]["execution_decisions"]["worker"] = valid_decision + store.save() + dispatch._mark_worker_done( + store, task, + initial_decision=valid_decision, + worker_cli=adapter, + worker_model=target, + ) + + state = store.task_state(task) + self.assertTrue( + state["worker_done"], + f"{adapter}: worker_done must be True after valid commit", + ) + self.assertTrue( + state["selfcheck_done"], + f"{adapter}: selfcheck_done must be True for cloud_model", + ) + self.assertEqual( + state["execution_class"], + "cloud_model", + f"{adapter}: execution_class must remain cloud_model", + ) + self.assertEqual( + dispatch.task_stage(task, state), + "review", + f"{adapter}: task_stage must advance to review after valid cloud completion", + ) + # completing_decision must be persisted with validated shape + persisted = state["completing_decision"] + self.assertEqual( + persisted["selected"]["adapter"], adapter + ) + self.assertEqual( + persisted["selected"]["execution_class"], "cloud_model" + ) + finally: + store.close() + + def test_restart_valid_cloud_advances_to_review(self): + """Restart with valid cloud completing decision goes to review, not selfcheck. + + Regression: a task that already has worker_done=True with a valid + cloud completing decision must resume to review on restart. + """ + for adapter, target in self._CLOUD_CASES: + with self.subTest(adapter=adapter, target=target): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + valid_decision = self.make_cloud_decision(adapter, target) + store.update_task( + task, + worker_done=True, + worker_cli=adapter, + worker_model=target, + completing_decision=valid_decision, + execution_class="cloud_model", + selfcheck_done=True, + blocked=None, + ) + state = store.task_state(task) + self.assertTrue(state["worker_done"]) + self.assertTrue(state["selfcheck_done"]) + stage = dispatch.task_stage(task, state) + self.assertNotEqual(stage, "selfcheck") + self.assertEqual( + stage, + "review", + f"{adapter}: restart with valid cloud must go to review", + ) + finally: + store.close() + + async def test_run_worker_completion_mismatch_blocks_task_without_raise( + self, + ): + """run_worker converts completion validation failure to task-local blocker. + + When the persisted worker decision's runtime identity does not match + the actual worker that completed, run_worker must catch the error, + keep worker_done=False, record a task-local blocked reason, and + return normallyโ€”never propagating ExecutionDecisionError to the + scheduler. + """ + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + # Persisted decision says Pi/Laguna, but worker actually completed + # as cloud/codex โ€” identity mismatch. + laguna_decision = { + "work_unit_id": self._WORK_UNIT_ID, + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + loc_codex = self.make_locator(workspace, "codex", "gpt-5.6-sol") + + store.task_state(task) # initialize state + store.data["tasks"][task.name]["execution_decisions"]["worker"] = laguna_decision + store.save() + + def mock_persisted(*a, **kw): + return laguna_decision, dispatch.AgentSpec( + "pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True + ) + + # run_worker returns normally; does not raise. + with mock.patch.object( + dispatch, "persisted_execution_decision", + side_effect=mock_persisted, + ), mock.patch.object( + dispatch, "run_escalating", + new=mock.AsyncMock(return_value=(True, loc_codex)), + ): + await dispatch.run_worker( + workspace, store, task + ) + + state = store.task_state(task) + self.assertFalse(state["worker_done"]) + self.assertIsNotNone(state.get("blocked")) + self.assertIn("worker completion validation failed", state["blocked"]) + # Provider-deny guard must not have been bypassed + self.assertIsNone(state.get("active_locator")) + finally: + store.close() + + +class LegacyWorkLogContractHelpers: + @staticmethod + def completed_replacements(): + return { + "### ๋ชฉํ‘œ์™€ ๋ฒ”์œ„\n\n- ๋ฏธ์ž‘์„ฑ": ( + "### ๋ชฉํ‘œ์™€ ๋ฒ”์œ„\n\n- PLAN ๋ฒ”์œ„ ๊ตฌํ˜„ ๋ฐ ๊ฒ€์ฆ" + ), + "### ์ฒดํฌํฌ์ธํŠธ\n\n- ๊ธฐ๋ก ์—†์Œ": ( + "### ์ฒดํฌํฌ์ธํŠธ\n\n" + "- 2026-07-24T00:01:00Z | ๊ตฌํ˜„ | ์™„๋ฃŒ | ํ•ต์‹ฌ ๊ฒฝ๋กœ ์ˆ˜์ • | " + "evidence=`src/test.go` | next=๊ฒ€์ฆ" + ), + "### ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ\n\n- ๊ธฐ๋ก ์—†์Œ": ( + "### ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ\n\n" + "- 2026-07-24T00:02:00Z | correctness | ๊ณ„ํš ๋ฐ– race ๊ฐ€๋Šฅ์„ฑ | " + "impact=๋™์‹œ์„ฑ ์˜ค๋ฅ˜ | action=์ˆ˜์ • ๋ฐ ํ…Œ์ŠคํŠธ | disposition=ํ•ด๊ฒฐ" + ), + "### ๊ฒ€์ฆ\n\n- ๊ธฐ๋ก ์—†์Œ": ( + "### ๊ฒ€์ฆ\n\n- `go test ./...` - PASS" + ), + "- ์ƒํƒœ: ๋ฏธ์ž‘์„ฑ": "- ์ƒํƒœ: ์™„๋ฃŒ", + "- ์š”์•ฝ: ๋ฏธ์ž‘์„ฑ": "- ์š”์•ฝ: ๊ตฌํ˜„ ๋ฐ ๊ฒ€์ฆ ์™„๋ฃŒ", + "- ์™„๋ฃŒ ํ•ญ๋ชฉ: ๋ฏธ์ž‘์„ฑ": "- ์™„๋ฃŒ ํ•ญ๋ชฉ: ๊ณ„ํš ์ฒดํฌ๋ฆฌ์ŠคํŠธ ์ „์ฒด", + "- ๋ณ€๊ฒฝ ํŒŒ์ผ: ๋ฏธ์ž‘์„ฑ": "- ๋ณ€๊ฒฝ ํŒŒ์ผ: `src/test.go`", + "- ๊ฒ€์ฆ: ๋ฏธ์ž‘์„ฑ": "- ๊ฒ€์ฆ: `go test ./...` PASS", + "- ๋ฏธํ•ด๊ฒฐ/ํ›„์†: ๋ฏธ์ž‘์„ฑ": "- ๋ฏธํ•ด๊ฒฐ/ํ›„์†: ์—†์Œ", + "- ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ ์š”์•ฝ: ๋ฏธ์ž‘์„ฑ": ( + "- ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ ์š”์•ฝ: race ๊ฐ€๋Šฅ์„ฑ ์ˆ˜์ • ์™„๋ฃŒ" + ), + "- CODE_REVIEW ๋™๊ธฐํ™”: ๋ฏธ์ž‘์„ฑ": "- CODE_REVIEW ๋™๊ธฐํ™”: ์™„๋ฃŒ", + } + + def make_completed_log(self, root: Path, task=None): + task = task or TaskStageTest().make_task(root) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + execution_id = "test__p0__worker__a00" + path = dispatch.append_work_log_attempt( + task, + execution_id, + "worker", + spec, + root / "locator.json", + "2026-07-24T00:00:00+00:00", + ) + text = path.read_text(encoding="utf-8") + for before, after in self.completed_replacements().items(): + self.assertIn(before, text) + text = text.replace(before, after, 1) + path.write_text(text, encoding="utf-8") + return task, path, execution_id + + def test_template_and_attempt_require_checkpoints_and_final_report(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = TaskStageTest().make_task(root) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + execution_id = "test__p0__worker__a00" + path = dispatch.append_work_log_attempt( + task, + execution_id, + "worker", + spec, + root / "locator.json", + "2026-07-24T00:00:00+00:00", + ) + rendered = path.read_text(encoding="utf-8") + self.assertNotIn(dispatch.WORK_LOG_TEMPLATE_START, rendered) + self.assertIn(f"## ์‹คํ–‰ `{execution_id}`", rendered) + status, errors = dispatch.work_log_attempt_result(path, execution_id) + self.assertIsNone(status) + self.assertIn("์ฒดํฌํฌ์ธํŠธ ๋ฏธ์ž‘์„ฑ", errors) + self.assertIn("์ตœ์ข… ๋ฆฌํฌํŠธ ์ƒํƒœ ๋ฏธ์ž‘์„ฑ", errors) + + def test_completed_attempt_preserves_unexpected_issue_and_runtime_result(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + _, path, execution_id = self.make_completed_log(root) + status, errors = dispatch.work_log_attempt_result(path, execution_id) + self.assertEqual(status, "์™„๋ฃŒ") + self.assertEqual(errors, []) + dispatch.append_work_log_runtime_result( + path, + execution_id, + exit_code=0, + failure_class=None, + locator=root / "locator.json", + ) + text = path.read_text(encoding="utf-8") + self.assertIn("๊ณ„ํš ๋ฐ– race ๊ฐ€๋Šฅ์„ฑ", text) + self.assertIn(f"### ๋Ÿฐํƒ€์ž„ ์ข…๋ฃŒ ๊ธฐ๋ก `{execution_id}`", text) + self.assertIn("- failure_class: `none`", text) + + def test_checkpoint_cannot_be_replaced_with_none(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + _, path, execution_id = self.make_completed_log(root) + text = path.read_text(encoding="utf-8") + checkpoint = self.completed_replacements()[ + "### ์ฒดํฌํฌ์ธํŠธ\n\n- ๊ธฐ๋ก ์—†์Œ" + ] + path.write_text( + text.replace(checkpoint, "### ์ฒดํฌํฌ์ธํŠธ\n\n- ์—†์Œ", 1), + encoding="utf-8", + ) + status, errors = dispatch.work_log_attempt_result(path, execution_id) + self.assertEqual(status, "์™„๋ฃŒ") + self.assertIn("์ฒดํฌํฌ์ธํŠธ ํ˜•์‹ ๋ถˆ์ผ์น˜", errors) + + def test_unexpected_issue_accepts_explicit_none(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + _, path, execution_id = self.make_completed_log(root) + text = path.read_text(encoding="utf-8") + unexpected = self.completed_replacements()[ + "### ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ\n\n- ๊ธฐ๋ก ์—†์Œ" + ] + path.write_text( + text.replace(unexpected, "### ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ\n\n- ์—†์Œ", 1), + encoding="utf-8", + ) + status, errors = dispatch.work_log_attempt_result(path, execution_id) + self.assertEqual(status, "์™„๋ฃŒ") + self.assertEqual(errors, []) + + def test_code_review_sync_must_be_exactly_complete(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + _, path, execution_id = self.make_completed_log(root) + text = path.read_text(encoding="utf-8") + path.write_text( + text.replace( + "- CODE_REVIEW ๋™๊ธฐํ™”: ์™„๋ฃŒ", + "- CODE_REVIEW ๋™๊ธฐํ™”: ์‹คํŒจ", + 1, + ), + encoding="utf-8", + ) + status, errors = dispatch.work_log_attempt_result(path, execution_id) + self.assertEqual(status, "์™„๋ฃŒ") + self.assertIn( + "์ตœ์ข… ๋ฆฌํฌํŠธ CODE_REVIEW ๋™๊ธฐํ™”๋Š” ์™„๋ฃŒ์—ฌ์•ผ ํ•œ๋‹ค", + errors, + ) + + def test_completed_review_checklist_does_not_depend_on_worker_log(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = TaskStageTest().make_task( + root, + "- [x] CODE_REVIEW-*-G??.md์˜ ๊ตฌํ˜„ ์—์ด์ „ํŠธ ์†Œ์œ  ์„น์…˜์„ " + "์‹ค์ œ ๊ตฌํ˜„ ๋‚ด์šฉ๊ณผ ๊ฒ€์ฆ ์ถœ๋ ฅ์œผ๋กœ ์ฑ„์šด๋‹ค.\n" + "- [x] `WORK_LOG.md` ํ˜„์žฌ ์‹คํ–‰ ๋ธ”๋ก์˜ ์ฒดํฌํฌ์ธํŠธ, ์˜ˆ์ƒ ๋ฐ– ์ด์Šˆ, " + "๊ฒ€์ฆ, ์ตœ์ข… ๋ฆฌํฌํŠธ๋ฅผ ๋ชจ๋‘ ์ฑ„์šด๋‹ค. ์ด ํ•ญ๋ชฉ์ด ์™„๋ฃŒ๋˜๊ธฐ ์ „์—๋Š” " + "๊ตฌํ˜„์ด ์™„๋ฃŒ๋œ ๊ฒƒ์ด ์•„๋‹ˆ๋‹ค.\n", + ) + task.plan.write_text( + task.plan.read_text(encoding="utf-8") + + "## ์ž‘์—… ๋กœ๊ทธ ๊ณ„์•ฝ\n", + encoding="utf-8", + ) + self.assertEqual(dispatch.task_stage(task, {}), "review") + + def test_prompt_requires_checkpoints_unexpected_issues_and_final_report(self): + path = Path("/tmp/task/WORK_LOG.md") + prompt = dispatch.work_log_prompt(path, "task__p0__worker__a00") + self.assertIn("checkpoint after each meaningful phase", prompt) + self.assertIn("unexpected issues", prompt) + self.assertIn("์ตœ์ข… ๋ฆฌํฌํŠธ", prompt) + self.assertIn("CODE_REVIEW", prompt) + + def test_state_loss_skips_execution_id_already_present_in_work_log(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task, _, _ = self.make_completed_log(root) + store = mock.Mock() + store.next_attempt.side_effect = [0, 1] + attempt, execution_id = dispatch.next_execution_identity( + store, + task, + "worker", + ) + self.assertEqual(attempt, 1) + self.assertEqual(execution_id, "test__p0__worker__a01") + self.assertEqual(store.next_attempt.call_count, 2) + + +class WorkLogInvokeIntegrationTest(unittest.IsolatedAsyncioTestCase): + def test_work_log_timestamp_uses_compact_kst_format(self): + fixed_kst = datetime(2026, 7, 26, 7, 40, 15, tzinfo=dispatch.KST) + with mock.patch.object(dispatch, "datetime") as datetime_mock: + datetime_mock.now.return_value = fixed_kst + self.assertEqual(dispatch.work_log_now_kst(), "26-07-26 07:40:15") + datetime_mock.now.assert_called_once_with(dispatch.KST) + + self.assertRegex(dispatch.now_iso(), r"\+00:00$") + + def test_milestone_timeline_uses_active_artifact_and_plan_loop(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + task_directory = ( + workspace + / "agent-task" + / "m-principal-provider-credential-slot-routing" + / "02+01_credential_catalog" + ) + task_directory.mkdir(parents=True) + plan = task_directory / "PLAN-local-G07.md" + review = task_directory / "CODE_REVIEW-cloud-G07.md" + plan.write_text( + "\n", + encoding="utf-8", + ) + review.write_text("review\n", encoding="utf-8") + task = dispatch.Task( + name=( + "m-principal-provider-credential-slot-routing/" + "02+01_credential_catalog" + ), + directory=task_directory, + plan=plan, + review=review, + user_review=None, + recovery=False, + ) + + for role in ("worker", "selfcheck", "review"): + dispatch.append_milestone_event( + task, + event="START", + execution_id=f"test__p1__{role}__a00", + role=role, + attempt=0, + model="test-model", + result="running", + locator=workspace / role / "locator.json", + ) + + plan.rename(task_directory / "plan_local_G07_1.log") + dispatch.append_milestone_event( + task, + event="FINISH", + execution_id="test__p1__review__a00", + role="review", + attempt=0, + model="test-model", + result="succeeded:0", + locator=workspace / "review" / "locator.json", + ) + + log = ( + task_directory.parent / dispatch.WORK_LOG_NAME + ).read_text(encoding="utf-8") + self.assertIn( + "| seq | time | event | task | loop | role | attempt |", + log, + ) + task_name = task.name + self.assertIn( + f"| START | {task_name}/PLAN-local-G07.md | 1 | worker | 0 |", + log, + ) + self.assertIn( + f"| START | {task_name}/CODE_REVIEW-cloud-G07.md | " + "1 | selfcheck | 0 |", + log, + ) + self.assertIn( + f"| START | {task_name}/CODE_REVIEW-cloud-G07.md | " + "1 | review | 0 |", + log, + ) + self.assertIn( + f"| FINISH | {task_name}/CODE_REVIEW-cloud-G07.md | " + "1 | review | 0 |", + log, + ) + + def test_legacy_timeline_infers_loop_from_locator_identity(self): + cells = dispatch.work_log_event_cells( + "| 21 | 26-08-01 14:18:01 | START | group/task | selfcheck | " + "0 | pi | running | /workspace__p9__/runs/" + "group__task__p1__selfcheck__a00/locator.json |" + ) + + self.assertIsNotNone(cells) + assert cells is not None + self.assertEqual(cells[3:7], ["group/task", "1", "selfcheck", "0"]) + + async def test_invoke_writes_dispatcher_owned_milestone_timeline(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + command = [ + sys.executable, + "-c", + ( + "import os;" + f"assert os.environ.get('{dispatch.AGENT_PROCESS_MARKER_ENV}');" + "print('work complete', flush=True)" + ), + ] + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=command, + ) as build_command: + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual( + record["work_log"], + str((workspace / dispatch.WORK_LOG_NAME).resolve()), + ) + self.assertTrue( + record["agent_process_marker"].startswith( + f"w{store.workspace_id}__test__p0__worker__a00__" + ) + ) + self.assertEqual(record["workspace"], str(workspace.resolve())) + self.assertEqual(record["workspace_id"], store.workspace_id) + self.assertEqual(record["status"], "succeeded") + prompt = build_command.call_args.args[1] + self.assertEqual(prompt, "Read the plan.") + self.assertNotIn("checkpoint after each meaningful phase", prompt) + log = (workspace / dispatch.WORK_LOG_NAME).read_text(encoding="utf-8") + self.assertIn("Dispatcher-owned execution timeline", log) + self.assertRegex(log, r"\| \d+ \| \d{2}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \| START \|") + self.assertRegex(log, r"\| \d+ \| \d{2}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \| FINISH \|") + self.assertIn( + "| START | test/PLAN-local-G05.md | 0 | worker | 0 | pi | running |", + log, + ) + self.assertIn( + "| FINISH | test/PLAN-local-G05.md | 0 | worker | 0 | pi | " + "succeeded:0 |", + log, + ) + + async def test_invoke_logs_task_directory_and_plan_declared_file_targets(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + command = [sys.executable, "-c", "print('done')"] + try: + with ( + mock.patch.object( + dispatch, "build_command", return_value=command + ), + mock.patch("sys.stdout", new_callable=io.StringIO) as stdout, + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + target = str((workspace / "src" / "test.py").resolve()) + output = stdout.getvalue() + self.assertIn(f"task_dir={workspace.resolve()}", output) + self.assertIn(f"target_file={target}", output) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["task_directory"], str(workspace.resolve())) + self.assertEqual(record["target_files"], [target]) + self.assertTrue(record["target_files_known"]) + + async def test_invoke_does_not_require_model_written_work_log(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + command = [sys.executable, "-c", "print('done without report')"] + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=command, + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + + async def test_locator_refresh_failure_does_not_abort_live_model(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + real_write_json = dispatch.write_json + failed_once = False + + def flaky_write_json(path, value): + nonlocal failed_once + if ( + path.name == "locator.json" + and value.get("agent_pid") is not None + and not failed_once + ): + failed_once = True + raise OSError("transient locator write failure") + return real_write_json(path, value) + + spec = dispatch.AgentSpec( + "pi", "ornith:35b", "pi", local_pi=True + ) + try: + with ( + mock.patch.object( + dispatch, + "build_command", + return_value=[ + sys.executable, + "-c", + "print('completed', flush=True)", + ], + ), + mock.patch.object( + dispatch, "write_json", side_effect=flaky_write_json + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "worker", spec, "Work." + ) + finally: + store.close() + + self.assertTrue(failed_once) + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "succeeded") + self.assertIn("transient locator write failure", record["locator_write_error"]) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "succeeded") + self.assertNotIn("work_log_contract_errors", record) + + async def test_existing_milestone_log_is_preserved_and_extended(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + (workspace / dispatch.WORK_LOG_NAME).write_text( + "# malformed\n", + encoding="utf-8", + ) + store = dispatch.StateStore(workspace) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + command = [sys.executable, "-c", "print('done')"] + try: + with mock.patch.object( + dispatch, "build_command", return_value=command + ) as build_command: + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + build_command.assert_called_once() + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "succeeded") + log = (workspace / dispatch.WORK_LOG_NAME).read_text(encoding="utf-8") + self.assertIn("# malformed", log) + self.assertIn("## Dispatcher Timeline", log) + + async def test_runtime_log_write_failure_finishes_locator_as_blocked_failure(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + command = [sys.executable, "-c", "print('done')"] + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + try: + with ( + mock.patch.object( + dispatch, + "build_command", + return_value=command, + ), + mock.patch.object( + dispatch, + "append_milestone_event", + side_effect=[ + workspace / dispatch.WORK_LOG_NAME, + OSError("disk full"), + ], + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertEqual(failure, "work-log-runtime-write") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "failed") + self.assertEqual(record["failure_class"], "work-log-runtime-write") + self.assertEqual(record["work_log_runtime_error"], "disk full") + + async def test_cancelled_invoke_finishes_locator_and_runtime_record(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + command = [ + sys.executable, + "-c", + "import time; print('ready', flush=True); time.sleep(60)", + ] + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + invocation = None + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=command, + ): + invocation = asyncio.create_task( + dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + ) + locator = None + for _ in range(100): + candidates = list(store.runs.glob("*/locator.json")) + if candidates: + candidate = candidates[0] + record = json.loads( + candidate.read_text(encoding="utf-8") + ) + output = Path(record["output_log"]) + if ( + output.is_file() + and "ready" in output.read_text(encoding="utf-8") + ): + locator = candidate + break + await asyncio.sleep(0.01) + self.assertIsNotNone(locator) + invocation.cancel() + with self.assertRaises(asyncio.CancelledError): + await invocation + finally: + if invocation is not None and not invocation.done(): + invocation.cancel() + await asyncio.gather(invocation, return_exceptions=True) + store.close() + + assert locator is not None + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "failed") + self.assertEqual(record["exit_code"], "cancelled") + self.assertEqual(record["failure_class"], "cancelled") + log = (workspace / dispatch.WORK_LOG_NAME).read_text(encoding="utf-8") + self.assertIn( + "| FINISH | test/PLAN-local-G05.md | 0 | worker | 0 | pi | " + "failed:cancelled |", + log, + ) + + async def test_pi_silent_awaiting_model_is_inspected_without_termination(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + session_id = "22222222-2222-2222-2222-222222222222" + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + native = attempt_dir / "pi-sessions" / f"session_{session_id}.jsonl" + child = ( + "from pathlib import Path\n" + "import sys,time\n" + "path = Path(sys.argv[1])\n" + "path.parent.mkdir(parents=True, exist_ok=True)\n" + "path.write_text(" + "'{\"type\":\"session\",\"version\":3,\"id\":\"test\"," + "\"timestamp\":\"2026-07-25T00:00:00.000Z\"," + "\"cwd\":\"/tmp/test\"}\\n" + "{\"type\":\"message\",\"id\":\"assistant-1\"," + "\"parentId\":null,\"message\":{\"role\":\"assistant\"," + "\"content\":[{\"type\":\"toolCall\",\"id\":\"call-1\"," + "\"name\":\"read\"}]}}\\n" + "{\"type\":\"message\",\"id\":\"result-1\"," + "\"parentId\":\"assistant-1\"," + "\"message\":{\"role\":\"toolResult\"," + "\"toolCallId\":\"call-1\",\"content\":[]}}\\n', " + "encoding='utf-8')\n" + "time.sleep(0.08)\n" + ) + return [sys.executable, "-c", child, str(native)] + + spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi", local_pi=True) + try: + with ( + mock.patch.object(dispatch, "build_command", side_effect=command_for), + mock.patch.object(dispatch.uuid, "uuid4", return_value=session_id), + mock.patch.object(dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01), + mock.patch.object( + dispatch, "PI_MODEL_RESPONSE_STALL_SECONDS", 0.03 + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "review", spec, "Reply briefly." + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "succeeded") + self.assertEqual(record["pi_session_phase"], "awaiting-model") + self.assertEqual( + record["pi_session_phase_reason"], + "all-tool-results-recorded", + ) + self.assertEqual(record["pi_expected_tool_call_ids"], ["call-1"]) + self.assertEqual(record["pi_completed_tool_call_ids"], ["call-1"]) + self.assertEqual(record["pi_pending_tool_call_ids"], []) + inspection = record["pi_silence_inspection"] + self.assertGreaterEqual(inspection["silence_seconds"], 0.03) + self.assertIn("stream_tail", inspection) + heartbeat = Path(record["heartbeat_log"]).read_text(encoding="utf-8") + self.assertIn("[silence-inspection]", heartbeat) + + async def test_pi_silent_starting_state_is_inspected_without_termination(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + session_id = "24242424-2424-2424-2424-242424242424" + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + native = attempt_dir / "pi-sessions" / f"session_{session_id}.jsonl" + child = ( + "from pathlib import Path\n" + "import sys,time\n" + "path = Path(sys.argv[1])\n" + "path.parent.mkdir(parents=True, exist_ok=True)\n" + "path.write_text(" + "'{\"type\":\"session\",\"version\":3,\"id\":\"test\"," + "\"timestamp\":\"2026-07-25T00:00:00.000Z\"," + "\"cwd\":\"/tmp/test\"}\\n', encoding='utf-8')\n" + "time.sleep(0.08)\n" + ) + return [sys.executable, "-c", child, str(native)] + + spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi", local_pi=True) + try: + with ( + mock.patch.object(dispatch, "build_command", side_effect=command_for), + mock.patch.object(dispatch.uuid, "uuid4", return_value=session_id), + mock.patch.object(dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01), + mock.patch.object( + dispatch, "PI_MODEL_RESPONSE_STALL_SECONDS", 0.03 + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "review", spec, "Reply briefly." + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "succeeded") + self.assertEqual(record["pi_session_phase"], "starting") + self.assertIsNone(record["pi_stall_timeout_seconds"]) + self.assertIn("pi_silence_inspection", record) + heartbeat = Path(record["heartbeat_log"]).read_text(encoding="utf-8") + self.assertNotIn("[session-stall]", heartbeat) + + async def test_pi_json_stream_progress_prevents_native_only_stall(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + session_id = "23232323-2323-2323-2323-232323232323" + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + native = attempt_dir / "pi-sessions" / f"session_{session_id}.jsonl" + child = ( + "from pathlib import Path\n" + "import json,sys,time\n" + "path = Path(sys.argv[1])\n" + "path.parent.mkdir(parents=True, exist_ok=True)\n" + "path.write_text(" + "'{\"type\":\"session\",\"version\":3,\"id\":\"test\"," + "\"timestamp\":\"2026-07-25T00:00:00.000Z\"," + "\"cwd\":\"/tmp/test\"}\\n', encoding='utf-8')\n" + "for _ in range(8):\n" + " print(json.dumps({'type': 'message_update'}), flush=True)\n" + " time.sleep(0.015)\n" + ) + return [sys.executable, "-c", child, str(native)] + + spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi", local_pi=True) + try: + with ( + mock.patch.object(dispatch, "build_command", side_effect=command_for), + mock.patch.object(dispatch.uuid, "uuid4", return_value=session_id), + mock.patch.object(dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01), + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "review", spec, "Reply briefly." + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "succeeded") + self.assertEqual(record["pi_activity_state"], "streaming") + stream = Path(record["stream_log"]).read_text(encoding="utf-8") + self.assertIn('[stdout] {"type": "message_update"}', stream) + + async def test_pi_incomplete_tool_batch_has_no_automatic_timeout(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + session_id = "33333333-3333-3333-3333-333333333333" + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + native = attempt_dir / "pi-sessions" / f"session_{session_id}.jsonl" + child = ( + "from pathlib import Path\n" + "import sys,time\n" + "path = Path(sys.argv[1])\n" + "path.parent.mkdir(parents=True, exist_ok=True)\n" + "path.write_text(" + "'{\"type\":\"session\",\"version\":3,\"id\":\"test\"," + "\"timestamp\":\"2026-07-25T00:00:00.000Z\"," + "\"cwd\":\"/tmp/test\"}\\n" + "{\"type\":\"message\",\"id\":\"assistant-1\"," + "\"parentId\":null,\"message\":{\"role\":\"assistant\"," + "\"content\":[{\"type\":\"toolCall\",\"id\":\"call-a\"," + "\"name\":\"read\"},{\"type\":\"toolCall\",\"id\":\"call-b\"," + "\"name\":\"bash\"}]}}\\n" + "{\"type\":\"message\",\"id\":\"result-a\"," + "\"parentId\":\"assistant-1\"," + "\"message\":{\"role\":\"toolResult\"," + "\"toolCallId\":\"call-a\",\"content\":[]}}\\n', " + "encoding='utf-8')\n" + "time.sleep(0.08)\n" + "with path.open('a', encoding='utf-8') as stream:\n" + " stream.write(" + "'{\"type\":\"message\",\"id\":\"result-b\"," + "\"parentId\":\"result-a\"," + "\"message\":{\"role\":\"toolResult\"," + "\"toolCallId\":\"call-b\",\"content\":[]}}\\n')\n" + "print('done', flush=True)\n" + ) + return [sys.executable, "-c", child, str(native)] + + spec = dispatch.AgentSpec( + "pi", "laguna-s:2.1", "pi", local_pi=True + ) + try: + with ( + mock.patch.object( + dispatch, "build_command", side_effect=command_for + ), + mock.patch.object( + dispatch.uuid, "uuid4", return_value=session_id + ), + mock.patch.object( + dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01 + ), + mock.patch.object( + dispatch, "PI_MODEL_RESPONSE_STALL_SECONDS", 0.02 + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "review", spec, "Reply briefly." + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertIsNone(record["pi_stall_timeout_seconds"]) + heartbeat = Path(record["heartbeat_log"]).read_text(encoding="utf-8") + self.assertNotIn("[session-stall]", heartbeat) + + async def test_resume_heartbeat_preserves_prior_native_session_path(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + prior_attempt = store.runs / "prior-attempt" + prior_attempt.mkdir() + native = prior_attempt / "prior-session.jsonl" + native.write_text(pi_session_jsonl([]), encoding="utf-8") + prior_locator = prior_attempt / "locator.json" + prior_locator.write_text( + json.dumps( + { + "workspace": str(workspace.resolve()), + "workspace_id": store.workspace_id, + "session_id": "resume-session", + "native_session_path": str(native), + } + ), + encoding="utf-8", + ) + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + self.assertEqual(pi_resume_session, native) + return [ + sys.executable, + "-c", + "import time; time.sleep(0.05); print('done', flush=True)", + ] + + spec = dispatch.AgentSpec( + "pi", "laguna-s:2.1", "pi", local_pi=True + ) + try: + with ( + mock.patch.object( + dispatch, "build_command", side_effect=command_for + ), + mock.patch.object( + dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01 + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "review", + spec, + "Continue.", + resume_locator=prior_locator, + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["native_session_path"], str(native)) + self.assertEqual( + record["resumed_from_locator"], str(prior_locator) + ) + + async def test_invoke_starts_fresh_session_for_foreign_pi_resume_locator(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) / "current" + workspace.mkdir() + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + foreign_attempt = Path(temporary) / "foreign-attempt" + foreign_attempt.mkdir() + foreign_native = foreign_attempt / "session.jsonl" + foreign_native.write_text(pi_session_jsonl([]), encoding="utf-8") + foreign_locator = foreign_attempt / "locator.json" + foreign_locator.write_text( + json.dumps( + { + "workspace": str((Path(temporary) / "foreign").resolve()), + "workspace_id": "foreign-workspace", + "session_id": "foreign-session", + "native_session_path": str(foreign_native), + } + ), + encoding="utf-8", + ) + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + self.assertIsNone(pi_resume_session) + self.assertNotEqual(actual_session_id, "foreign-session") + return [ + sys.executable, + "-c", + "print('fresh session', flush=True)", + ] + + spec = dispatch.AgentSpec( + "pi", + "laguna-s:2.1", + "pi", + local_pi=True, + ) + try: + with mock.patch.object( + dispatch, + "build_command", + side_effect=command_for, + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "review", + spec, + "Continue.", + resume_locator=foreign_locator, + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertIsNone(record["resumed_from_locator"]) + self.assertNotEqual( + record["native_session_path"], + str(foreign_native), + ) + + async def test_provider_stderr_requires_and_preserves_exact_evidence(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + provider_line = ( + "provider_tunnel_error: dial tcp 192.0.2.1:8001: " + "connect: connection refused" + ) + command = [ + sys.executable, + "-c", + "import sys; sys.stderr.write(sys.argv[1] + '\\n'); " + "raise SystemExit(1)", + provider_line, + ] + spec = dispatch.AgentSpec( + "pi", "ornith:35b", "pi", local_pi=True + ) + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=command, + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "worker", spec, "Read the plan." + ) + finally: + store.close() + + self.assertEqual(rc, 1) + self.assertEqual(failure, "provider-connection") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual( + record["failure_source"], "provider-terminal-diagnostic" + ) + self.assertTrue(record["provider_transport_failure_confirmed"]) + self.assertEqual(record["failure_evidence_source"], "pi:stderr") + self.assertEqual(record["failure_evidence_excerpt"], provider_line) + self.assertEqual(record["dispatcher_pid"], dispatch.os.getpid()) + self.assertEqual( + record["dispatcher_source_path"], str(dispatch.DISPATCHER_SOURCE_PATH) + ) + self.assertEqual( + record["dispatcher_source_sha256"], + dispatch.DISPATCHER_SOURCE_SHA256, + ) + self.assertEqual( + record["dispatcher_source_current_sha256"], + dispatch.DISPATCHER_SOURCE_SHA256, + ) + self.assertTrue(record["dispatcher_source_matches_loaded"]) + self.assertEqual( + dispatch.DISPATCHER_SOURCE_SHA256, + dispatch.sha256_file(SCRIPT), + ) + report = dispatch.failure_report_lines(failure, locator) + self.assertIn("provider_transport_failure_confirmed=true", report) + self.assertIn(f"dispatcher_pid={dispatch.os.getpid()}", report) + self.assertIn( + f"dispatcher_source_sha256={dispatch.DISPATCHER_SOURCE_SHA256}", + report, + ) + self.assertIn( + "dispatcher_source_matches_loaded=true", + report, + ) + self.assertIn(f"provider_evidence={provider_line}", report) + + async def test_claude_session_limit_stderr_records_provider_quota(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + diagnostic = ( + "You've hit your session limit ยท resets 9pm (Asia/Seoul)" + ) + command = [ + sys.executable, + "-c", + "import sys; sys.stderr.write(sys.argv[1] + '\\n'); " + "raise SystemExit(1)", + diagnostic, + ] + spec = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=command, + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 1) + self.assertEqual(failure, "provider-quota") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["failure_source"], "cli-terminal-diagnostic") + self.assertEqual(record["failure_evidence_source"], "claude:stderr") + self.assertEqual(record["failure_evidence_excerpt"], diagnostic) + self.assertEqual(record["reasoning_effort"], "xhigh") + self.assertFalse(record["provider_transport_failure_confirmed"]) + + async def test_claude_structured_rate_limit_stdout_records_provider_quota(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + rate_limit_event = json.dumps( + { + "type": "rate_limit_event", + "rate_limit_info": { + "status": "rejected", + "rateLimitType": "five_hour", + "overageStatus": "rejected", + }, + }, + ensure_ascii=False, + ) + result_event = json.dumps( + { + "type": "result", + "subtype": "success", + "is_error": True, + "terminal_reason": "api_error", + "api_error_status": 429, + "result": ( + "You've hit your session limit ยท " + "resets 9pm (Asia/Seoul)" + ), + }, + ensure_ascii=False, + ) + command = [ + sys.executable, + "-c", + "import sys; print(sys.argv[1]); print(sys.argv[2]); " + "raise SystemExit(1)", + rate_limit_event, + result_event, + ] + spec = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=command, + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 1) + self.assertEqual(failure, "provider-quota") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["failure_source"], "cli-terminal-diagnostic") + self.assertEqual(record["failure_evidence_source"], "claude:stdout") + self.assertIn('"api_error_status": 429', record["failure_evidence_excerpt"]) + self.assertFalse(record["provider_transport_failure_confirmed"]) + + async def test_agy_cli_log_quota_records_provider_quota(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + diagnostic = ( + "rpc failed: code=ResourceExhausted " + "status=RESOURCE_EXHAUSTED HTTP 429 quota exceeded" + ) + spec = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + + def build_agy_command( + _spec, + _prompt, + _workspace, + _session_id, + attempt_dir, + **_kwargs, + ): + return [ + sys.executable, + "-c", + ( + "from pathlib import Path; " + "Path(__import__('sys').argv[1]).write_text(" + "__import__('sys').argv[2] + '\\n', encoding='utf-8'); " + "raise SystemExit(1)" + ), + str(attempt_dir / "agy-cli.log"), + diagnostic, + ] + + try: + with ( + mock.patch.object( + dispatch, + "build_command", + side_effect=build_agy_command, + ), + mock.patch.object( + dispatch, + "agy_conversations", + return_value={}, + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 1) + self.assertEqual(failure, "provider-quota") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["failure_source"], "cli-terminal-diagnostic") + self.assertEqual( + record["failure_evidence_source"], + "agy:cli-log", + ) + self.assertEqual(record["failure_evidence_excerpt"], diagnostic) + + async def test_agy_cli_log_quota_with_zero_exit_records_provider_quota(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + diagnostic = ( + "agent executor error: model unreachable: " + "RESOURCE_EXHAUSTED (code 429): Individual quota reached" + ) + spec = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + + def build_agy_command( + _spec, + _prompt, + _workspace, + _session_id, + attempt_dir, + **_kwargs, + ): + return [ + sys.executable, + "-c", + ( + "from pathlib import Path; " + "Path(__import__('sys').argv[1]).write_text(" + "__import__('sys').argv[2] + '\\n', encoding='utf-8')" + ), + str(attempt_dir / "agy-cli.log"), + diagnostic, + ] + + try: + with ( + mock.patch.object( + dispatch, + "build_command", + side_effect=build_agy_command, + ), + mock.patch.object( + dispatch, + "agy_conversations", + return_value={}, + ), + ): + rc, failure, locator = await dispatch.invoke( + workspace, + store, + task, + "worker", + spec, + "Read the plan.", + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertEqual(failure, "provider-quota") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["status"], "failed") + self.assertEqual(record["exit_code"], 0) + self.assertEqual(record["failure_source"], "cli-terminal-diagnostic") + self.assertEqual( + record["failure_evidence_source"], + "agy:cli-log", + ) + self.assertEqual(record["failure_evidence_excerpt"], diagnostic) + self.assertFalse(record["provider_transport_failure_confirmed"]) + + async def test_exit_143_is_process_termination_not_provider_failure(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = TaskStageTest().make_task(workspace) + store = dispatch.StateStore(workspace) + provider_line = ( + "provider_tunnel_error: dial tcp 192.0.2.1:8001: " + "connect: connection refused" + ) + spec = dispatch.AgentSpec( + "pi", "ornith:35b", "pi", local_pi=True + ) + try: + with mock.patch.object( + dispatch, + "build_command", + return_value=[ + sys.executable, + "-c", + "import sys; sys.stderr.write(sys.argv[1] + '\\n'); " + "raise SystemExit(143)", + provider_line, + ], + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "worker", spec, "Read the plan." + ) + finally: + store.close() + + self.assertEqual(rc, 143) + self.assertEqual(failure, "process-terminated") + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertEqual(record["failure_source"], "process-termination") + self.assertFalse(record["provider_transport_failure_confirmed"]) + self.assertEqual(record["termination_signal"], "SIGTERM") + self.assertTrue(record["termination_signal_inferred"]) + self.assertEqual(record["termination_initiator"], "unknown") + + +class ReviewControlTest(unittest.TestCase): + def test_classifies_claude_session_limit_as_provider_quota(self): + diagnostic = ( + "You've hit your session limit ยท resets 9pm (Asia/Seoul)" + ) + self.assertEqual( + dispatch.classify_failure_with_evidence(diagnostic), + ("provider-quota", diagnostic), + ) + + def test_claude_assistant_text_is_not_a_terminal_diagnostic(self): + assistant_event = json.dumps( + { + "type": "assistant", + "message": { + "role": "assistant", + "content": [ + { + "type": "text", + "text": "You've hit your session limit", + } + ], + }, + } + ) + self.assertIsNone( + dispatch.terminal_diagnostic("claude", "stdout", assistant_event) + ) + + def test_claude_rejected_rate_limit_event_is_terminal_diagnostic(self): + event = json.dumps( + { + "type": "rate_limit_event", + "rate_limit_info": {"status": "rejected"}, + } + ) + diagnostic = dispatch.terminal_diagnostic("claude", "stdout", event) + self.assertIsNotNone(diagnostic) + self.assertEqual( + dispatch.classify_failure_with_evidence(diagnostic or ""), + ("provider-quota", diagnostic), + ) + + def test_agy_structured_resource_exhausted_is_terminal_diagnostic(self): + event = json.dumps( + { + "type": "error", + "error": { + "code": 429, + "status": "RESOURCE_EXHAUSTED", + "message": "Quota exceeded", + }, + } + ) + + diagnostic = dispatch.terminal_diagnostic("agy", "stdout", event) + + self.assertIsNotNone(diagnostic) + self.assertEqual( + dispatch.classify_failure_with_evidence(diagnostic or ""), + ("provider-quota", diagnostic), + ) + + def test_agy_assistant_quota_text_is_not_a_terminal_diagnostic(self): + event = json.dumps( + { + "type": "assistant", + "status": "rejected", + "error": {"code": 429}, + "content": "The quota exceeded message is handled in the code.", + } + ) + + self.assertIsNone( + dispatch.terminal_diagnostic("agy", "stdout", event) + ) + + def test_agy_log_diagnostic_requires_strong_quota_evidence(self): + with tempfile.TemporaryDirectory() as temporary: + log = Path(temporary) / "agy-cli.log" + log.write_text( + "quota configuration loaded\n" + "ERROR quota configuration refresh failed\n" + "status=RESOURCE_EXHAUSTED HTTP 429 quota exceeded\n", + encoding="utf-8", + ) + + self.assertEqual( + dispatch.agy_log_diagnostics(log), + ["status=RESOURCE_EXHAUSTED HTTP 429 quota exceeded"], + ) + + def test_claude_promotion_targets_terra_high(self): + claude = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + promoted = dispatch.promoted_spec(claude, recovery_count=0) + + self.assertEqual( + promoted, + dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ), + ) + assert promoted is not None + command = dispatch.build_command( + promoted, + "Read the plan.", + Path("/workspace"), + "session-id", + Path("/attempt"), + ) + self.assertIn("gpt-5.6-terra", command) + self.assertIn('model_reasoning_effort="high"', command) + self.assertEqual( + dispatch.effective_reasoning_effort(promoted), + "high", + ) + self.assertIs( + dispatch.promoted_spec(promoted, recovery_count=0), + promoted, + ) + + def test_regular_codex_and_claude_routes_keep_xhigh_effort(self): + codex = dispatch.AgentSpec( + "codex", + "gpt-5.6-sol", + "codex/gpt-5.6-sol xhigh", + ) + spark = dispatch.AgentSpec( + "codex", + "gpt-5.3-codex-spark", + "codex/gpt-5.3-codex-spark xhigh", + ) + claude = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + haiku = dispatch.AgentSpec( + "claude", + "claude-haiku-4-5", + "claude/claude-haiku-4-5 xhigh", + ) + self.assertEqual(dispatch.effective_reasoning_effort(codex), "xhigh") + self.assertEqual(dispatch.effective_reasoning_effort(spark), "xhigh") + self.assertEqual(dispatch.effective_reasoning_effort(claude), "xhigh") + self.assertEqual(dispatch.effective_reasoning_effort(haiku), "xhigh") + + def test_classifies_provider_tunnel_connection_refusal(self): + provider_line = ( + "provider_tunnel_error: dial tcp 192.0.2.1:8001: " + "connect: connection refused" + ) + self.assertEqual( + dispatch.classify_failure(provider_line), + "provider-connection", + ) + self.assertEqual( + dispatch.classify_failure_with_evidence( + f"unrelated warning\n{provider_line}" + ), + ("provider-connection", provider_line), + ) + + def test_generic_tool_stderr_is_not_provider_transport_evidence(self): + weak_lines = [ + "pytest setup failed: connection refused while opening fixture", + "dial tcp 127.0.0.1:9999: connect: connection refused", + "curl error: failure when receiving data from the peer", + ] + for line in weak_lines: + with self.subTest(line=line): + self.assertEqual( + dispatch.classify_failure_with_evidence(line), + ("generic-error", None), + ) + + def test_provider_stream_requires_strong_backend_or_sse_context(self): + line = ( + "Backend for model crashed before streaming started: " + "SSE stream before DONE" + ) + self.assertEqual( + dispatch.classify_failure_with_evidence(line), + ("provider-stream-disconnect", line), + ) + + def test_pi_stdout_provider_words_are_not_terminal_diagnostics(self): + line = "provider_tunnel_error: connection refused" + self.assertIsNone(dispatch.terminal_diagnostic("pi", "stdout", line)) + + def test_dispatcher_source_provenance_detects_hot_edit(self): + changed_sha256 = "f" * 64 + self.assertNotEqual(changed_sha256, dispatch.DISPATCHER_SOURCE_SHA256) + with mock.patch.object( + dispatch, + "sha256_file", + return_value=changed_sha256, + ): + provenance = dispatch.dispatcher_source_provenance() + self.assertEqual( + provenance["dispatcher_source_sha256"], + dispatch.DISPATCHER_SOURCE_SHA256, + ) + self.assertEqual( + provenance["dispatcher_source_current_sha256"], changed_sha256 + ) + self.assertFalse(provenance["dispatcher_source_matches_loaded"]) + + def test_pi_phase_reads_large_last_jsonl_event(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + path.write_text( + pi_session_jsonl( + [ + { + "type": "message", + "message": { + "role": "assistant", + "content": [ + { + "type": "toolCall", + "id": "large-result", + "name": "read", + } + ], + }, + }, + { + "type": "message", + "message": { + "role": "toolResult", + "toolCallId": "large-result", + "content": [ + {"type": "text", "text": "x" * 20000} + ], + }, + }, + ] + ), + encoding="utf-8", + ) + self.assertEqual( + dispatch.pi_native_session_phase(str(path)), + "awaiting-model", + ) + + def test_pi_phase_keeps_incomplete_sequential_batch_tool_running(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + events = [ + { + "type": "message", + "message": { + "role": "assistant", + "content": [ + {"type": "toolCall", "id": "call-a", "name": "read"}, + {"type": "toolCall", "id": "call-b", "name": "bash"}, + ], + }, + }, + { + "type": "message", + "message": { + "role": "toolResult", + "toolCallId": "call-a", + "content": [], + }, + }, + ] + path.write_text( + pi_session_jsonl(events), + encoding="utf-8", + ) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "tool-running") + self.assertEqual(state.expected_tool_call_ids, ("call-a", "call-b")) + self.assertEqual(state.completed_tool_call_ids, ("call-a",)) + self.assertEqual(state.pending_tool_call_ids, ("call-b",)) + + def test_pi_phase_waits_for_model_only_after_entire_batch_completes(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + events = [ + { + "type": "message", + "message": { + "role": "assistant", + "content": [ + {"type": "toolCall", "id": "call-a", "name": "read"}, + {"type": "toolCall", "id": "call-b", "name": "bash"}, + ], + }, + }, + { + "type": "message", + "message": { + "role": "toolResult", + "toolCallId": "call-a", + "content": [], + }, + }, + { + "type": "message", + "message": { + "role": "toolResult", + "toolCallId": "call-b", + "content": [], + }, + }, + ] + path.write_text( + pi_session_jsonl(events), + encoding="utf-8", + ) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "awaiting-model") + self.assertEqual(state.completed_tool_call_ids, ("call-a", "call-b")) + self.assertEqual(state.pending_tool_call_ids, ()) + + def test_pi_phase_marks_unknown_schema_without_assuming_tool_completion(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + path.write_text( + pi_session_jsonl( + [ + { + "type": "message", + "message": { + "role": "toolResult", + "content": [], + }, + }, + ] + ), + encoding="utf-8", + ) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "unknown") + self.assertEqual(state.reason, "tool-result-id-missing") + + def test_pi_phase_does_not_treat_unknown_assistant_content_as_final(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + path.write_text( + pi_session_jsonl( + [ + { + "type": "message", + "message": { + "role": "assistant", + "content": [ + { + "type": "futureToolCall", + "id": "unknown-call", + } + ], + }, + }, + ] + ), + encoding="utf-8", + ) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "unknown") + self.assertEqual(state.reason, "unsupported-assistant-content") + + def test_pi_phase_marks_future_session_version_unknown(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + path.write_text( + pi_session_jsonl( + [ + { + "type": "message", + "message": { + "role": "user", + "content": [], + }, + }, + ], + version=dispatch.PI_SESSION_SCHEMA_VERSION + 1, + ), + encoding="utf-8", + ) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "unknown") + self.assertEqual( + state.reason, + f"unsupported-session-version:" + f"{dispatch.PI_SESSION_SCHEMA_VERSION + 1}", + ) + + def test_pi_phase_marks_corrupt_session_entries_unknown(self): + header = pi_session_jsonl([]).encode() + cases = { + "missing-parent": header + + json.dumps( + { + "type": "message", + "id": "message-without-parent", + "message": { + "role": "user", + "content": [], + }, + } + ).encode() + + b"\n", + "invalid-utf8": header + b'{"type":"message","id":"bad-\\xff"}\n', + } + for name, content in cases.items(): + with self.subTest(name=name), tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + path.write_bytes(content) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "unknown") + + def test_pi_phase_follows_only_the_active_session_branch(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "session.jsonl" + path.write_text( + pi_session_jsonl( + [ + { + "type": "message", + "id": "root-user", + "parentId": None, + "message": { + "role": "user", + "content": [], + }, + }, + { + "type": "message", + "id": "abandoned-assistant", + "parentId": "root-user", + "message": { + "role": "assistant", + "content": [{"type": "text", "text": "done"}], + }, + }, + { + "type": "custom", + "id": "active-branch-marker", + "parentId": "root-user", + }, + ] + ), + encoding="utf-8", + ) + + state = dispatch.pi_native_session_state(str(path)) + + self.assertEqual(state.phase, "awaiting-model") + self.assertEqual(state.reason, "user-message") + + def test_detects_codex_collaboration_wait(self): + line = ( + '{"type":"item.started","item":{"type":"collab_tool_call",' + '"tool":"wait"}}' + ) + self.assertEqual(dispatch.codex_collaboration_tool(line), "wait") + + def test_ignores_completed_or_non_json_events(self): + self.assertIsNone( + dispatch.codex_collaboration_tool( + '{"type":"item.completed","item":{"type":"collab_tool_call",' + '"tool":"wait"}}' + ) + ) + self.assertIsNone(dispatch.codex_collaboration_tool("not json")) + + def test_prompts_keep_local_work_and_official_review_roles_separate(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = TaskStageTest().make_task(root) + pi = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + selfcheck = dispatch.base_prompt(task, "selfcheck", pi) + unchecked_retry = dispatch.base_prompt( + task, "selfcheck", pi, unchecked_items=True + ) + review = dispatch.base_prompt( + task, + "review", + dispatch.AgentSpec("codex", "gpt-5.6-sol", "codex"), + ) + self.assertNotIn("USER_REVIEW", selfcheck) + self.assertNotIn("user review", selfcheck.lower()) + self.assertEqual( + selfcheck, + f"{dispatch.SELF_CHECK_PROMPT_PREFIX} Read " + f"{task.plan.resolve()}; review all work once, fix omissions, " + f"and update {task.review.resolve()}. Keep files in English.", + ) + self.assertEqual( + unchecked_retry, + f"{dispatch.SELF_CHECK_PROMPT_PREFIX} Read " + f"{task.plan.resolve()}; complete every unchecked implementation " + f"item and update {task.review.resolve()}. Keep files in English.", + ) + self.assertIn(str(task.plan.resolve()), unchecked_retry) + self.assertNotIn("dispatcher child", selfcheck.lower()) + self.assertEqual( + dispatch.continuation_prompt( + task, + "selfcheck", + local_pi=True, + unchecked_items=True, + ), + unchecked_retry, + ) + self.assertEqual( + dispatch.continuation_prompt( + task, + "selfcheck", + local_pi=True, + resume_same_pi_session=True, + unchecked_items=True, + ), + unchecked_retry, + ) + self.assertEqual( + dispatch.continuation_prompt( + task, + "selfcheck", + local_pi=True, + resume_same_pi_session=True, + ), + f"{dispatch.SELF_CHECK_PROMPT_PREFIX} Continue. Keep files in " + "English.", + ) + self.assertEqual( + review, + dispatch.dispatcher_child_prompt( + f"Read {task.review.resolve()} and start the review. Keep " + "artifact content in English. Final in Korean." + ), + ) + + def test_local_review_stub_has_no_user_review_control_plane_content(self): + template = ( + Path(__file__).parents[3] + / "common" + / "plan" + / "templates" + / "review-stub-template.md" + ).read_text(encoding="utf-8") + self.assertNotIn("USER_REVIEW", template) + self.assertNotIn("์‚ฌ์šฉ์ž ๋ฆฌ๋ทฐ", template) + self.assertNotIn("user-review", template) + self.assertNotIn("## ์ž‘์—… ๋กœ๊ทธ ๊ณ„์•ฝ", template) + self.assertNotIn("WORK_LOG.md", template) + + def test_final_channel_contract_is_top_level_and_singular(self): + skill = ( + Path(__file__).parents[1] / "SKILL.md" + ).read_text(encoding="utf-8") + heading = "## ๐Ÿšจ ABSOLUTE PRIORITY โ€” NEVER SEND `final` EXCEPT IN THE TWO CASES BELOW" + self.assertEqual(skill.count(heading), 1) + priority, lower_contract = skill.split("\n## Purpose\n", 1) + self.assertEqual(priority.count("### `final` Permission"), 2) + self.assertEqual(priority.count("Allow `final`"), 2) + self.assertIn("unless at least one of the two titled permissions", priority) + self.assertNotIn("unless exactly one of the two titled permissions", priority) + self.assertNotIn("### `final` Permission", lower_contract) + self.assertNotIn("Allow `final`", lower_contract) + self.assertIn( + "### Every Other User-Visible Message Must Use `commentary`", + priority, + ) + self.assertIn( + "### Child Prompt Text Never Grants Caller `final` Permission", + priority, + ) + + def test_skill_narrative_is_english_except_exact_protocol_literals(self): + skill = ( + Path(__file__).parents[1] / "SKILL.md" + ).read_text(encoding="utf-8") + self.assertIn( + "Treat Korean text inside code spans or fenced examples as exact runtime or file-contract literals", + skill, + ) + in_fence = False + for line_number, line in enumerate(skill.splitlines(), start=1): + if line.strip().startswith("```"): + in_fence = not in_fence + continue + if in_fence: + continue + narrative = re.sub(r"`[^`]*`", "", line) + self.assertIsNone( + re.search(r"[๊ฐ€-ํžฃ]", narrative), + f"line {line_number} has non-literal Korean narrative: {line}", + ) + + def test_work_log_archive_ownership_stays_project_local(self): + skills_root = Path(__file__).parents[3] + dispatcher_skill = ( + Path(__file__).parents[1] / "SKILL.md" + ).read_text(encoding="utf-8") + review_skill = ( + skills_root / "common" / "code-review" / "SKILL.md" + ).read_text(encoding="utf-8") + plan_skill = ( + skills_root / "common" / "plan" / "SKILL.md" + ).read_text(encoding="utf-8") + + self.assertIn( + "append the final `FINISH` and move the generated `WORK_LOG.md`", + dispatcher_skill, + ) + self.assertIn("work_log_N.log", dispatcher_skill) + self.assertIn( + "append `FINISH` with `reconciled:verified-complete-archive`", + dispatcher_skill, + ) + self.assertIn( + "pidless stream/native evidence remains live", + dispatcher_skill, + ) + self.assertIn( + "Do not require the common code-review skill to preserve " + "`WORK_LOG.md`", + dispatcher_skill, + ) + self.assertNotIn("WORK_LOG", review_skill) + self.assertNotIn("work-log", review_skill) + self.assertNotIn("WORK_LOG", plan_skill) + self.assertNotIn("work-log", plan_skill) + + +class ProcessTerminationTest(unittest.IsolatedAsyncioTestCase): + async def test_terminates_the_exact_process_group(self): + class Process: + pid = 12345 + returncode = None + + async def wait(self): + self.returncode = -signal.SIGTERM + return self.returncode + + process = Process() + with mock.patch.object( + dispatch.os, + "killpg", + side_effect=[None, ProcessLookupError], + ) as killpg: + await dispatch.terminate_process_group(process) + + self.assertEqual(killpg.call_args_list[0], mock.call(12345, signal.SIGTERM)) + self.assertEqual(killpg.call_args_list[1], mock.call(12345, 0)) + + async def test_kills_sigterm_ignoring_descendant_and_closes_pipe(self): + child_script = ( + "import signal,time;" + "signal.signal(signal.SIGTERM,signal.SIG_IGN);" + "time.sleep(60)" + ) + parent_script = ( + "import signal,subprocess,sys,time;" + "signal.signal(signal.SIGTERM,signal.SIG_IGN);" + f"subprocess.Popen([sys.executable,'-c',{child_script!r}]);" + "print('ready',flush=True);" + "time.sleep(60)" + ) + process = await asyncio.create_subprocess_exec( + sys.executable, + "-c", + parent_script, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + start_new_session=True, + ) + try: + assert process.stdout is not None + self.assertEqual( + await asyncio.wait_for(process.stdout.readline(), timeout=1), + b"ready\n", + ) + await dispatch.terminate_process_group(process, grace_seconds=0.05) + self.assertEqual(process.returncode, -signal.SIGKILL) + self.assertEqual( + await asyncio.wait_for(process.stdout.read(), timeout=1), + b"", + ) + finally: + if process.returncode is None: + await dispatch.terminate_process_group(process, grace_seconds=0.05) + + +class ReviewRetryTest(unittest.IsolatedAsyncioTestCase): + def make_task(self, root: Path): + return TaskStageTest().make_task(root) + + async def test_claude_provider_quota_promotes_to_terra_high(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + claude = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + terra = dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + locators = [root / "locator-0.json", root / "locator-1.json"] + results = [ + (1, "provider-quota", locators[0]), + (0, None, locators[1]), + ] + with mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock(side_effect=results), + ) as invoke: + success, locator = await dispatch.run_escalating( + root, + mock.Mock(), + task, + "worker", + claude, + ) + + self.assertTrue(success) + self.assertEqual(locator, locators[1]) + self.assertEqual(invoke.await_count, 2) + self.assertEqual(invoke.await_args_list[0].args[4], claude) + self.assertEqual(invoke.await_args_list[1].args[4], terra) + + async def test_agy_and_claude_quota_chain_promotes_to_terra(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + agy = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + claude = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + terra = dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + locators = [ + root / "locator-0.json", + root / "locator-1.json", + root / "locator-2.json", + ] + with mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + side_effect=[ + (1, "provider-quota", locators[0]), + (1, "provider-quota", locators[1]), + (0, None, locators[2]), + ] + ), + ) as invoke: + success, locator = await dispatch.run_escalating( + root, + mock.Mock(), + task, + "worker", + agy, + ) + + self.assertTrue(success) + self.assertEqual(locator, locators[2]) + self.assertEqual( + [call.args[4] for call in invoke.await_args_list], + [agy, claude, terra], + ) + + async def test_process_termination_retries_same_claude_target(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + claude = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + locators = [root / "locator-0.json", root / "locator-1.json"] + with ( + mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + side_effect=[ + (1, "process-terminated", locators[0]), + (0, None, locators[1]), + ] + ), + ) as invoke, + mock.patch.object( + dispatch.asyncio, + "sleep", + new=mock.AsyncMock(), + ), + ): + success, locator = await dispatch.run_escalating( + root, + mock.Mock(), + task, + "worker", + claude, + ) + + self.assertTrue(success) + self.assertEqual(locator, locators[1]) + self.assertEqual( + [call.args[4] for call in invoke.await_args_list], + [claude, claude], + ) + + async def test_legacy_generic_quota_blocker_resumes_directly_on_terra(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = self.make_task(root) + store = dispatch.StateStore(root) + locator = write_legacy_quota_attempts( + store.runs, + task, + )[-1] + state = store.task_state(task) + state.update( + blocked=( + "worker recovery failure limit exhausted: 10/10 " + f"locator={locator}" + ), + recovery_failures={"worker": 10}, + ) + recovery = dispatch.legacy_promotion_recovery( + store.runs, + task, + state, + ) + self.assertIsNotNone(recovery) + initial_route = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + terra = dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + completed_locator = root / "completed-locator.json" + try: + with mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + return_value=(0, None, completed_locator) + ), + ) as invoke: + success, actual_locator = await dispatch.run_escalating( + root, + store, + task, + "worker", + initial_route, + initial_resume_locator=locator, + ) + recovered_state = store.task_state(task) + finally: + store.close() + + self.assertTrue(success) + self.assertEqual(actual_locator, completed_locator) + self.assertEqual(invoke.await_count, 1) + self.assertEqual(invoke.await_args.args[4], terra) + self.assertIsNone(recovered_state["blocked"]) + self.assertEqual( + recovered_state["recovery_failures"], + {}, + ) + self.assertEqual( + recovered_state["legacy_terminal_reclassification"][ + "failed_cli" + ], + "claude", + ) + + async def test_legacy_agy_quota_log_resumes_on_claude_then_terra(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = self.make_task(root) + store = dispatch.StateStore(root) + locator = write_legacy_quota_attempts( + store.runs, + task, + cli="agy", + model="Gemini 3.6 Flash (High)", + reasoning_effort=None, + )[-1] + state = store.task_state(task) + state.update( + blocked=( + "worker recovery failure limit exhausted: 10/10 " + f"locator={locator}" + ), + recovery_failures={"worker": 10}, + ) + recovery = dispatch.legacy_promotion_recovery( + store.runs, + task, + state, + ) + self.assertIsNotNone(recovery) + assert recovery is not None + self.assertEqual(recovery.failed_cli, "agy") + self.assertEqual(recovery.evidence_source, "agy:cli-log") + initial_route = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + claude = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + terra = dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + locators = [ + root / "claude-locator.json", + root / "terra-locator.json", + ] + try: + with mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + side_effect=[ + (1, "provider-quota", locators[0]), + (0, None, locators[1]), + ] + ), + ) as invoke: + success, actual_locator = await dispatch.run_escalating( + root, + store, + task, + "worker", + initial_route, + initial_resume_locator=locator, + ) + finally: + store.close() + + self.assertTrue(success) + self.assertEqual(actual_locator, locators[1]) + self.assertEqual( + [call.args[4] for call in invoke.await_args_list], + [claude, terra], + ) + + async def test_worker_persists_the_actual_promoted_target(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = self.make_task(root) + store = dispatch.StateStore(root) + initial_route = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + attempt = store.runs / "completed-attempt" + attempt.mkdir() + locator = attempt / "locator.json" + locator.write_text( + json.dumps( + { + "cli": "codex", + "model": "gpt-5.6-terra", + "reasoning_effort": "high", + } + ), + encoding="utf-8", + ) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "codex", + "target": "gpt-5.6-terra", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + } + try: + # Persist completing decision to execution_decisions["worker"] + store.task_state(task) + store.data["tasks"][task.name]["execution_decisions"]["worker"] = completing_decision + store.save() + + with ( + mock.patch.object( + dispatch, + "persisted_execution_decision", + return_value=(completing_decision, initial_route), + ), + mock.patch.object( + dispatch, + "run_escalating", + new=mock.AsyncMock(return_value=(True, locator)), + ), + ): + await dispatch.run_worker( + root, + store, + task, + ) + state = store.task_state(task) + finally: + store.close() + + self.assertTrue(state["worker_done"]) + self.assertEqual(state["worker_cli"], "codex") + self.assertEqual(state["worker_model"], "gpt-5.6-terra") + self.assertTrue(state["selfcheck_done"]) + self.assertEqual( + state["execution_class"], "cloud_model" + ) + self.assertEqual( + state["completing_decision"]["selected"]["adapter"], "codex" + ) + + async def test_retries_two_control_violations_then_succeeds(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("codex", "gpt-5.6-sol", "codex") + locators = [root / f"locator-{index}.json" for index in range(3)] + results = [ + (1, "review-control-violation", locators[0]), + (1, "review-control-violation", locators[1]), + (0, None, locators[2]), + ] + with mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock(side_effect=results) + ) as invoke: + success, locator = await dispatch.run_escalating( + root, mock.Mock(), task, "review", spec + ) + self.assertTrue(success) + self.assertEqual(locator, locators[2]) + self.assertEqual(invoke.await_count, 3) + + async def test_review_control_retries_do_not_create_a_blocker(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("codex", "gpt-5.6-sol", "codex") + locators = [root / f"locator-{index}.json" for index in range(4)] + results = [ + (1, "review-control-violation", locators[0]), + (1, "review-control-violation", locators[1]), + (1, "review-control-violation", locators[2]), + (0, None, locators[3]), + ] + with ( + mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock(side_effect=results) + ) as invoke, + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + success, locator = await dispatch.run_escalating( + root, mock.Mock(), task, "review", spec + ) + self.assertTrue(success) + self.assertEqual(locator, locators[3]) + self.assertEqual(invoke.await_count, 4) + + async def test_does_not_retry_obsolete_model_work_log_failure(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + locators = [root / "locator-0.json", root / "locator-1.json"] + results = [ + (0, "work-log-incomplete", locators[0]), + (0, None, locators[1]), + ] + with mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock(side_effect=results) + ) as invoke: + success, locator = await dispatch.run_escalating( + root, mock.Mock(), task, "worker", spec + ) + self.assertFalse(success) + self.assertEqual(locator, locators[0]) + self.assertEqual(invoke.await_count, 1) + + async def test_retries_pi_session_stall_twice_with_same_model(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi", local_pi=True) + locators = [root / f"locator-{index}.json" for index in range(3)] + results = [ + (1, "session-stall", locators[0]), + (1, "session-stall", locators[1]), + (0, None, locators[2]), + ] + with ( + mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock(side_effect=results) + ) as invoke, + mock.patch.object( + dispatch.asyncio, "sleep", new=mock.AsyncMock() + ), + ): + success, locator = await dispatch.run_escalating( + root, mock.Mock(), task, "worker", spec + ) + self.assertTrue(success) + self.assertEqual(locator, locators[2]) + self.assertEqual(invoke.await_count, 3) + self.assertTrue(all(call.args[4] == spec for call in invoke.await_args_list)) + self.assertEqual( + invoke.await_args_list[1].args[-1], + dispatch.dispatcher_child_prompt( + "Think in English. Keep artifact content in English. Final " + "in Korean. Continue this session and complete the current " + "task." + ), + ) + self.assertEqual( + invoke.await_args_list[1].kwargs["resume_locator"], + locators[0], + ) + self.assertEqual( + invoke.await_args_list[2].kwargs["resume_locator"], + locators[1], + ) + + def test_failed_laguna_locator_is_recovered_after_dispatcher_restart(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + native = root / "session.jsonl" + native.write_text("{}\n", encoding="utf-8") + locator = root / "locator.json" + locator.write_text( + json.dumps( + { + "cli": "pi", + "model": "laguna-s:2.1", + "status": "failed", + "failure_class": "session-stall", + "native_session_path": str(native), + } + ), + encoding="utf-8", + ) + self.assertEqual( + dispatch.laguna_resume_locator( + {"active_locator": str(locator)} + ), + locator, + ) + + async def test_retries_pi_connection_and_generic_failures(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + locators = [root / f"locator-{index}.json" for index in range(3)] + results = [ + (1, "provider-connection", locators[0]), + (1, "generic-error", locators[1]), + (0, None, locators[2]), + ] + with ( + mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock(side_effect=results) + ) as invoke, + mock.patch.object( + dispatch.asyncio, "sleep", new=mock.AsyncMock() + ) as sleep, + ): + success, locator = await dispatch.run_escalating( + root, mock.Mock(), task, "worker", spec + ) + self.assertTrue(success) + self.assertEqual(locator, locators[2]) + self.assertEqual(invoke.await_count, 3) + self.assertEqual(sleep.await_count, 2) + + async def test_pi_tenth_failure_blocks_without_cooldown(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + locators = [ + root / f"locator-{index}.json" + for index in range(dispatch.RECOVERY_FAILURE_LIMIT) + ] + results = [ + (1, "provider-stream-disconnect", locator) + for locator in locators + ] + sleep_observations = [] + + async def observe_sleep(delay): + sleep_observations.append(delay) + + with ( + mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock(side_effect=results) + ) as invoke, + mock.patch.object( + dispatch.asyncio, + "sleep", + new=mock.AsyncMock(side_effect=observe_sleep), + ), + ): + success, locator = await dispatch.run_escalating( + root, + mock.Mock(), + task, + "worker", + spec, + ) + + self.assertFalse(success) + self.assertEqual(locator, locators[-1]) + self.assertEqual(invoke.await_count, dispatch.RECOVERY_FAILURE_LIMIT) + self.assertEqual( + len(sleep_observations), dispatch.RECOVERY_FAILURE_LIMIT - 1 + ) + + async def test_recovery_failure_limit_survives_dispatcher_restart(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = self.make_task(root) + store = dispatch.StateStore(root) + locator = root / "locator.json" + store.update_task( + task, recovery_failures={"worker": dispatch.RECOVERY_FAILURE_LIMIT - 1} + ) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + try: + with mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + return_value=(1, "generic-error", locator) + ), + ) as invoke: + success, actual_locator = await dispatch.run_escalating( + root, store, task, "worker", spec + ) + self.assertFalse(success) + self.assertEqual(actual_locator, locator) + self.assertEqual(invoke.await_count, 1) + self.assertIn( + "recovery failure limit exhausted", + store.task_state(task)["blocked"], + ) + finally: + store.close() + + async def test_already_exhausted_recovery_budget_does_not_invoke_model(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = self.make_task(root) + store = dispatch.StateStore(root) + locator = root / "locator.json" + store.update_task( + task, recovery_failures={"worker": dispatch.RECOVERY_FAILURE_LIMIT} + ) + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + try: + with mock.patch.object( + dispatch, "invoke", new=mock.AsyncMock() + ) as invoke: + success, actual_locator = await dispatch.run_escalating( + root, + store, + task, + "worker", + spec, + initial_resume_locator=locator, + ) + self.assertFalse(success) + self.assertEqual(actual_locator, locator) + self.assertEqual(invoke.await_count, 0) + finally: + store.close() + + async def test_does_not_promote_work_log_infrastructure_failure(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + task = self.make_task(root) + spec = dispatch.AgentSpec("codex", "gpt-5.6-sol", "codex") + locator = root / "locator.json" + with mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + return_value=(1, "work-log-runtime-write", locator) + ), + ) as invoke: + success, actual_locator = await dispatch.run_escalating( + root, + mock.Mock(), + task, + "worker", + spec, + ) + self.assertFalse(success) + self.assertEqual(actual_locator, locator) + self.assertEqual(invoke.await_count, 1) + + +class RepetitionLimitTest(unittest.IsolatedAsyncioTestCase): + async def test_exhausted_selfcheck_budget_does_not_invoke_model(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task( + task, + selfcheck_incomplete=( + dispatch.SELF_CHECK_UNCHECKED_RETRY_LIMIT + 1 + ), + completing_decision=completing_decision, + ) + try: + with mock.patch.object( + dispatch, "run_escalating", new=mock.AsyncMock() + ) as run_escalating: + await dispatch.run_selfcheck(root, store, task) + self.assertEqual(run_escalating.await_count, 0) + self.assertIn( + "limit already exhausted", store.task_state(task)["blocked"] + ) + finally: + store.close() + + async def test_exhausted_review_budget_does_not_invoke_model(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + store.update_task( + task, review_no_progress=dispatch.REVIEW_NO_PROGRESS_LIMIT + ) + try: + with mock.patch.object( + dispatch, "run_escalating", new=mock.AsyncMock() + ) as run_escalating: + result = await dispatch.run_review(root, store, task) + self.assertIsNone(result) + self.assertEqual(run_escalating.await_count, 0) + self.assertIn( + "limit already exhausted", store.task_state(task)["blocked"] + ) + finally: + store.close() + + async def test_selfcheck_tenth_checklist_retry_blocks_task(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task( + task, + selfcheck_incomplete=dispatch.SELF_CHECK_UNCHECKED_RETRY_LIMIT, + completing_decision=completing_decision, + ) + locator = root / "locator.json" + try: + with ( + mock.patch.object( + dispatch, + "run_escalating", + new=mock.AsyncMock(return_value=(True, locator)), + ) as run_escalating, + mock.patch.object( + dispatch, + "implementation_review_errors", + return_value=["๊ฒ€์ฆ ๊ฒฐ๊ณผ ๋ฏธ์™„์„ฑ"], + ), + ): + await dispatch.run_selfcheck( + root, store, task, resume_locator=locator + ) + self.assertEqual(run_escalating.await_count, 1) + self.assertTrue( + run_escalating.await_args.kwargs["unchecked_items"] + ) + self.assertIn( + "selfcheck checklist remains incomplete", + store.task_state(task)["blocked"], + ) + finally: + store.close() + + async def test_selfcheck_switches_to_unchecked_items_after_full_pass(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task(task, completing_decision=completing_decision) + locators = [root / "full-locator.json", root / "retry-locator.json"] + try: + with ( + mock.patch.object( + dispatch, + "run_escalating", + new=mock.AsyncMock( + side_effect=[ + (True, locators[0]), + (True, locators[1]), + ] + ), + ) as run_escalating, + mock.patch.object( + dispatch, + "implementation_review_errors", + side_effect=[["๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ ๋ฏธ์™„๋ฃŒ"], []], + ), + ): + await dispatch.run_selfcheck(root, store, task) + + self.assertEqual(run_escalating.await_count, 2) + self.assertFalse( + run_escalating.await_args_list[0].kwargs["unchecked_items"] + ) + self.assertTrue( + run_escalating.await_args_list[1].kwargs["unchecked_items"] + ) + self.assertIsNone( + run_escalating.await_args_list[0].kwargs[ + "initial_resume_locator" + ] + ) + self.assertEqual( + run_escalating.await_args_list[1].kwargs[ + "initial_resume_locator" + ], + locators[0], + ) + state = store.task_state(task) + self.assertTrue(state["selfcheck_done"]) + self.assertEqual(state["selfcheck_incomplete"], 0) + self.assertIsNone(state["selfcheck_context_locator"]) + finally: + store.close() + + async def test_selfcheck_restart_resumes_persisted_context(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + attempt = store.runs / "prior-selfcheck" + attempt.mkdir() + native = attempt / "session.jsonl" + native.write_text("{}\n", encoding="utf-8") + locator = attempt / "locator.json" + locator.write_text( + json.dumps( + { + "workspace": str(root.resolve()), + "workspace_id": store.workspace_id, + "task": task.name, + "role": "selfcheck", + "cli": "pi", + "status": "succeeded", + "native_session_path": str(native), + } + ), + encoding="utf-8", + ) + store.update_task( + task, + completing_decision=completing_decision, + selfcheck_incomplete=1, + selfcheck_context_locator=str(locator), + ) + retry_locator = root / "retry-locator.json" + try: + with ( + mock.patch.object( + dispatch, + "run_escalating", + new=mock.AsyncMock(return_value=(True, retry_locator)), + ) as run_escalating, + mock.patch.object( + dispatch, "implementation_review_errors", return_value=[] + ), + ): + await dispatch.run_selfcheck(root, store, task) + + self.assertEqual(run_escalating.await_count, 1) + self.assertTrue( + run_escalating.await_args.kwargs["unchecked_items"] + ) + self.assertEqual( + run_escalating.await_args.kwargs["initial_resume_locator"], + locator, + ) + finally: + store.close() + + async def test_selfcheck_restart_blocks_without_persisted_context(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task( + task, + completing_decision=completing_decision, + selfcheck_incomplete=1, + ) + try: + with mock.patch.object( + dispatch, "run_escalating", new=mock.AsyncMock() + ) as run_escalating: + await dispatch.run_selfcheck(root, store, task) + + self.assertEqual(run_escalating.await_count, 0) + self.assertIn( + "selfcheck context resume ์‹คํŒจ", + store.task_state(task)["blocked"], + ) + finally: + store.close() + + async def test_selfcheck_allows_ten_unchecked_item_retries(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + completing_decision = { + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task(task, completing_decision=completing_decision) + locator = root / "locator.json" + try: + with ( + mock.patch.object( + dispatch, + "run_escalating", + new=mock.AsyncMock(return_value=(True, locator)), + ) as run_escalating, + mock.patch.object( + dispatch, + "implementation_review_errors", + return_value=["๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ ๋ฏธ์™„๋ฃŒ"], + ), + ): + await dispatch.run_selfcheck(root, store, task) + + self.assertEqual( + run_escalating.await_count, + 1 + dispatch.SELF_CHECK_UNCHECKED_RETRY_LIMIT, + ) + self.assertFalse( + run_escalating.await_args_list[0].kwargs["unchecked_items"] + ) + self.assertTrue( + all( + call.kwargs["unchecked_items"] + for call in run_escalating.await_args_list[1:] + ) + ) + self.assertTrue( + all( + call.kwargs["initial_resume_locator"] == locator + for call in run_escalating.await_args_list[1:] + ) + ) + state = store.task_state(task) + self.assertEqual( + state["selfcheck_incomplete"], + 1 + dispatch.SELF_CHECK_UNCHECKED_RETRY_LIMIT, + ) + self.assertIn( + "selfcheck checklist remains incomplete", + state["blocked"], + ) + finally: + store.close() + + async def test_review_tenth_no_progress_pass_blocks_task(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / ".git").mkdir() + task = TaskStageTest().make_task(root) + store = dispatch.StateStore(root) + store.update_task( + task, + review_no_progress=dispatch.REVIEW_NO_PROGRESS_LIMIT - 1, + ) + locator = root / "locator.json" + try: + with ( + mock.patch.object( + dispatch, + "run_escalating", + new=mock.AsyncMock(return_value=(True, locator)), + ), + mock.patch.object( + dispatch, "task_signature", return_value="unchanged" + ), + mock.patch.object( + dispatch, "review_fingerprints", return_value=set() + ), + ): + result = await dispatch.run_review(root, store, task) + self.assertIsNone(result) + self.assertIn( + "review made no progress", store.task_state(task)["blocked"] + ) + finally: + store.close() + + +class BlockerDrainTest(unittest.IsolatedAsyncioTestCase): + async def test_user_review_only_holds_its_dependency_closure(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + group = workspace / "agent-task" / "m-test" + gate_dir = group / "01_gate" + dependent_dir = group / "02+01_dependent" + independent_dir = group / "03_independent" + for directory in (gate_dir, dependent_dir, independent_dir): + directory.mkdir(parents=True) + + user_review = gate_dir / "USER_REVIEW.md" + user_review.write_text( + TaskStageTest.blocking_user_review_text(), encoding="utf-8" + ) + gate = dispatch.Task( + name="m-test/01_gate", + directory=gate_dir, + plan=None, + review=None, + user_review=user_review, + recovery=True, + index=1, + ) + + def runnable_task(name, directory, index, deps=()): + plan = directory / "PLAN-local-G05.md" + review = directory / "CODE_REVIEW-local-G05.md" + target = (workspace / "src" / f"task-{index}.py").resolve() + plan.write_text( + f"\n" + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + f"| `{target}` | TEST-1 |\n", + encoding="utf-8", + ) + review.write_text("", encoding="utf-8") + return dispatch.Task( + name=name, + directory=directory, + plan=plan, + review=review, + user_review=None, + recovery=False, + index=index, + deps=deps, + write_set={str(target)}, + write_set_known=True, + lane="local", + grade=5, + plan_hash=f"{name}-hash", + ) + + dependent = runnable_task( + "m-test/02+01_dependent", dependent_dir, 2, ("01",) + ) + independent = runnable_task( + "m-test/03_independent", independent_dir, 3 + ) + completed_archive = workspace / "completed-independent" + completed_archive.mkdir() + (completed_archive / "complete.log").write_text( + "complete\n", encoding="utf-8" + ) + args = SimpleNamespace( + task_group="m-test", retry_blocked=False, dry_run=False + ) + store = dispatch.StateStore(workspace) + try: + with ( + mock.patch.object( + dispatch, + "scan_tasks", + side_effect=[ + [gate, dependent, independent], + [gate, dependent], + ], + ), + mock.patch.object( + dispatch, + "run_worker", + new=mock.AsyncMock(return_value=str(completed_archive)), + ) as run_worker, + ): + result = await dispatch.dispatch_with_store( + args, workspace, store + ) + self.assertEqual(result, 2) + self.assertEqual(run_worker.await_count, 1) + self.assertEqual( + run_worker.await_args.args[2].name, independent.name + ) + orchestration = store.data["orchestrations"]["m-test"]["tasks"] + self.assertEqual(orchestration[gate.name]["status"], "blocked") + self.assertEqual( + orchestration[dependent.name]["status"], "waiting" + ) + self.assertEqual( + orchestration[independent.name]["status"], "complete" + ) + self.assertEqual( + store.data["orchestrations"]["m-test"]["status"], + "blocked", + ) + self.assertEqual( + orchestration[independent.name]["archive"], + str(completed_archive.resolve()), + ) + finally: + store.close() + + async def test_runtime_blocker_still_drains_independent_task(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + group = workspace / "agent-task" / "m-test" + gate_dir = group / "01_gate" + independent_dir = group / "02_independent" + gate_dir.mkdir(parents=True) + independent_dir.mkdir(parents=True) + + gate = TaskStageTest().make_task(gate_dir) + gate.name = "m-test/01_gate" + gate.index = 1 + gate.plan_hash = "gate-hash" + independent = TaskStageTest().make_task(independent_dir) + independent.name = "m-test/02_independent" + independent.index = 2 + independent.plan_hash = "independent-hash" + + completed_archive = workspace / "completed-independent" + completed_archive.mkdir() + (completed_archive / "complete.log").write_text( + "complete\n", encoding="utf-8" + ) + completed_tasks: list[str] = [] + + async def fake_worker(workspace_path, state_store, task, *args, **kwargs): + if task.name == gate.name: + state_store.update_task( + task, + blocked="worker recovery failure limit exhausted: 10/10", + ) + return None + completed_tasks.append(task.name) + return str(completed_archive) + + args = SimpleNamespace( + task_group="m-test", retry_blocked=False, dry_run=False + ) + store = dispatch.StateStore(workspace) + try: + with ( + mock.patch.object( + dispatch, + "scan_tasks", + side_effect=[[gate, independent], [gate]], + ), + mock.patch.object(dispatch, "run_worker", new=fake_worker), + ): + result = await dispatch.dispatch_with_store( + args, workspace, store + ) + self.assertEqual(result, 2) + self.assertEqual(completed_tasks, [independent.name]) + group_state = store.data["orchestrations"]["m-test"] + orchestration = group_state["tasks"] + self.assertEqual(group_state["status"], "blocked") + self.assertEqual(orchestration[gate.name]["status"], "blocked") + self.assertEqual( + orchestration[independent.name]["status"], "complete" + ) + store.prepare_orchestration("m-test", [gate], workspace) + group_state = store.data["orchestrations"]["m-test"] + self.assertEqual(group_state["status"], "running") + self.assertEqual( + group_state["tasks"][gate.name]["status"], "active" + ) + self.assertNotIn( + "reason", group_state["tasks"][gate.name] + ) + finally: + store.close() + + async def test_unexpected_agent_exception_drains_sibling_then_returns_three(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + group = workspace / "agent-task" / "m-test" + failed_dir = group / "01_failed" + sibling_dir = group / "02_sibling" + failed_dir.mkdir(parents=True) + sibling_dir.mkdir(parents=True) + + failed = TaskStageTest().make_task(failed_dir) + failed.name = "m-test/01_failed" + failed.index = 1 + failed.plan_hash = "failed-hash" + sibling = TaskStageTest().make_task(sibling_dir) + sibling.name = "m-test/02_sibling" + sibling.index = 2 + sibling.plan_hash = "sibling-hash" + + completed_archive = workspace / "completed-sibling" + completed_archive.mkdir() + (completed_archive / "complete.log").write_text( + "complete\n", encoding="utf-8" + ) + events: list[str] = [] + + async def fake_worker(workspace_path, state_store, task, *args, **kwargs): + if task.name == failed.name: + raise RuntimeError("unexpected control failure") + events.append("sibling-started") + await asyncio.sleep(0.02) + events.append("sibling-finished") + return str(completed_archive) + + args = SimpleNamespace( + task_group="m-test", retry_blocked=False, dry_run=False + ) + store = dispatch.StateStore(workspace) + try: + with ( + mock.patch.object( + dispatch, + "scan_tasks", + side_effect=[[failed, sibling], [failed]], + ), + mock.patch.object(dispatch, "run_worker", new=fake_worker), + ): + result = await dispatch.dispatch_with_store( + args, workspace, store + ) + + self.assertEqual(result, 3) + self.assertEqual(events, ["sibling-started", "sibling-finished"]) + group_state = store.data["orchestrations"]["m-test"] + self.assertEqual(group_state["status"], "running") + self.assertNotEqual( + group_state["tasks"][failed.name]["status"], + "blocked", + ) + self.assertEqual( + group_state["tasks"][sibling.name]["status"], + "complete", + ) + finally: + store.close() + + async def test_review_preflight_failure_still_drains_independent_worker(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + review_dir = workspace / "agent-task" / "m-test" / "01_review" + worker_dir = workspace / "agent-task" / "m-test" / "02_worker" + review_dir.mkdir(parents=True) + worker_dir.mkdir(parents=True) + + review = TaskStageTest().make_task(review_dir) + review.name = "m-test/01_review" + review.index = 1 + review.plan_hash = "review-hash" + worker = TaskStageTest().make_task(worker_dir) + worker.name = "m-test/02_worker" + worker.index = 2 + worker.plan_hash = "worker-hash" + + completed_archive = workspace / "completed-worker" + completed_archive.mkdir() + (completed_archive / "complete.log").write_text( + "complete\n", encoding="utf-8" + ) + args = SimpleNamespace( + task_group="m-test", retry_blocked=False, dry_run=False + ) + store = dispatch.StateStore(workspace) + store.update_task( + review, + worker_done=True, + selfcheck_done=True, + completing_decision={ + "work_unit_id": "test::plan-0::tag-TEST", + "stage": "worker", + "selected": { + "adapter": "codex", + "target": "gpt-5.6-sol", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + }, + execution_class="cloud_model", + ) + try: + with ( + mock.patch.object( + dispatch, + "scan_tasks", + side_effect=[[review, worker], [review]], + ), + mock.patch.object( + dispatch, + "ensure_review_shared_state", + side_effect=RuntimeError("shared helper unavailable"), + ), + mock.patch.object( + dispatch, + "run_worker", + new=mock.AsyncMock(return_value=str(completed_archive)), + ) as run_worker, + mock.patch.object( + dispatch, "run_review", new=mock.AsyncMock() + ) as run_review, + ): + result = await dispatch.dispatch_with_store( + args, workspace, store + ) + self.assertEqual(result, 2) + self.assertEqual(run_worker.await_count, 1) + self.assertEqual(run_review.await_count, 0) + orchestration = store.data["orchestrations"]["m-test"]["tasks"] + self.assertEqual(orchestration[review.name]["status"], "blocked") + self.assertEqual(orchestration[worker.name]["status"], "complete") + finally: + store.close() + + async def test_invalidated_complete_archive_cannot_end_with_success(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task_dir = workspace / "agent-task" / "m-test" / "01_task" + task_dir.mkdir(parents=True) + task = TaskStageTest().make_task(task_dir) + task.name = "m-test/01_task" + task.index = 1 + task.plan_hash = "task-hash" + + archive = workspace / "completed-task" + archive.mkdir() + complete_log = archive / "complete.log" + complete_log.write_text("complete\n", encoding="utf-8") + scan_count = 0 + + def scan_side_effect(*args, **kwargs): + nonlocal scan_count + scan_count += 1 + if scan_count == 1: + return [task] + complete_log.unlink() + return [] + + args = SimpleNamespace( + task_group="m-test", retry_blocked=False, dry_run=False + ) + store = dispatch.StateStore(workspace) + try: + with ( + mock.patch.object( + dispatch, "scan_tasks", side_effect=scan_side_effect + ), + mock.patch.object( + dispatch, + "run_worker", + new=mock.AsyncMock(return_value=str(archive)), + ), + ): + result = await dispatch.dispatch_with_store( + args, workspace, store + ) + self.assertEqual(result, 2) + self.assertEqual( + store.data["orchestrations"]["m-test"]["status"], + "blocked", + ) + finally: + store.close() + + + async def test_external_active_task_returns_non_terminal_exit_three(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + directory = workspace / "agent-task" / "m-test" / "01_active" + directory.mkdir(parents=True) + task = TaskStageTest().make_task(directory) + task.name = "m-test/01_active" + task.index = 1 + task.plan_hash = "active-hash" + locator = workspace / "locator.json" + args = SimpleNamespace( + task_group="m-test", retry_blocked=False, dry_run=False + ) + store = dispatch.StateStore(workspace) + store.mark_active(task, "worker", locator) + try: + with ( + mock.patch.object( + dispatch, "scan_tasks", return_value=[task] + ), + mock.patch.object( + dispatch, + "external_active_is_live", + return_value=(True, "pid=123"), + ), + mock.patch.object( + dispatch, "run_worker", new=mock.AsyncMock() + ) as run_worker, + ): + result = await dispatch.dispatch_with_store( + args, workspace, store + ) + self.assertEqual(result, 3) + self.assertEqual(run_worker.await_count, 0) + finally: + store.close() + + async def test_foreign_failed_laguna_locator_is_not_resumed(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) / "current" + workspace.mkdir() + (workspace / ".git").mkdir() + directory = workspace / "agent-task" / "group" / "01_task" + directory.mkdir(parents=True) + task = TaskStageTest().make_task(directory) + task.name = "group/01_task" + task.index = 1 + task.plan_hash = "foreign-laguna" + + foreign_attempt = Path(temporary) / "foreign-attempt" + foreign_attempt.mkdir() + foreign_native = foreign_attempt / "session.jsonl" + foreign_native.write_text("{}\n", encoding="utf-8") + foreign_locator = foreign_attempt / "locator.json" + foreign_locator.write_text( + json.dumps( + { + "status": "failed", + "workspace": str((Path(temporary) / "foreign").resolve()), + "workspace_id": "foreign-workspace", + "cli": "pi", + "model": "laguna-s:2.1", + "failure_class": "session-stall", + "native_session_path": str(foreign_native), + } + ), + encoding="utf-8", + ) + observed_resume_locators: list[Path | None] = [] + + async def fake_worker( + workspace_path, + state_store, + selected_task, + *args, + **kwargs, + ): + observed_resume_locators.append(kwargs.get("resume_locator")) + state_store.update_task(selected_task, blocked="test-stop") + return None + + args = SimpleNamespace( + task_group="group", + retry_blocked=False, + dry_run=False, + ) + store = dispatch.StateStore(workspace) + store.mark_active(task, "worker", foreign_locator) + try: + with ( + mock.patch.object( + dispatch, + "scan_tasks", + return_value=[task], + ), + mock.patch.object( + dispatch, + "run_worker", + new=fake_worker, + ), + ): + result = await dispatch.dispatch_with_store( + args, + workspace, + store, + ) + self.assertEqual(result, 2) + self.assertEqual(observed_resume_locators, [None]) + finally: + store.close() + + +class ReviewSchedulingTest(unittest.TestCase): + def test_all_ready_reviews_are_selected_without_numeric_cap(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + tasks = [ + dispatch.Task( + name=f"group/{index:02d}_task", + directory=workspace / f"task-{index}", + plan=None, + review=None, + user_review=None, + recovery=False, + write_set={ + str((workspace / "src" / f"task-{index}.py").resolve()) + }, + write_set_known=True, + plan_hash=f"hash-{index}", + ) + for index in range(4) + ] + ready = [ + (tasks[0], "review"), + (tasks[1], "review"), + (tasks[2], "review"), + (tasks[3], "worker"), + ] + store = dispatch.StateStore(workspace) + try: + selected, deferred, reason = dispatch.select_dispatch_candidates( + store, + ready, + persist=False, + ) + finally: + store.close() + self.assertEqual(selected, ready) + self.assertEqual(deferred, []) + self.assertEqual(reason, "") + + def test_new_reviews_join_already_running_review_phase(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + tasks = [ + dispatch.Task( + name=f"group/{index:02d}_task", + directory=workspace / f"task-{index}", + plan=None, + review=None, + user_review=None, + recovery=False, + write_set={ + str((workspace / "src" / f"task-{index}.py").resolve()) + }, + write_set_known=True, + plan_hash=f"hash-{index}", + ) + for index in range(3) + ] + ready = [ + (tasks[0], "review"), + (tasks[1], "review"), + (tasks[2], "selfcheck"), + ] + store = dispatch.StateStore(workspace) + try: + selected, deferred, reason = dispatch.select_dispatch_candidates( + store, + ready, + persist=False, + ) + finally: + store.close() + self.assertEqual(selected, ready) + self.assertEqual(deferred, []) + self.assertEqual(reason, "") + + def test_only_declared_same_group_live_predecessors_delay_task(self): + task = dispatch.Task( + name="group/03+01,02_join", + directory=Path("/tmp/group/03+01,02_join"), + plan=None, + review=None, + user_review=None, + recovery=False, + deps=("01", "02"), + ) + + live = dispatch.live_predecessors( + task, + { + "group/01_core", + "other/02_unrelated", + "group/04_parallel", + }, + ) + + self.assertEqual(live, ["01"]) + + def test_task_without_declared_dependency_ignores_live_siblings(self): + task = dispatch.Task( + name="group/04_parallel", + directory=Path("/tmp/group/04_parallel"), + plan=None, + review=None, + user_review=None, + recovery=False, + ) + + self.assertEqual( + dispatch.live_predecessors( + task, + {"group/01_core", "group/02_other"}, + ), + [], + ) + + +class WriteSetTest(unittest.TestCase): + def make_claim_task( + self, + workspace: Path, + name: str, + *paths: Path, + plan_hash: str = "plan-0", + ) -> dispatch.Task: + return dispatch.Task( + name=name, + directory=workspace / "agent-task" / name, + plan=None, + review=None, + user_review=None, + recovery=False, + write_set={str(path.resolve()) for path in paths}, + write_set_known=True, + plan_hash=plan_hash, + ) + + def test_normalizes_relative_and_absolute_aliases(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + source = workspace / "src" / "shared.go" + source.parent.mkdir() + source.write_text("package src\n", encoding="utf-8") + relative_plan = workspace / "relative.md" + absolute_plan = workspace / "absolute.md" + relative_plan.write_text( + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + "| `./src/shared.go` | TEST-1 |\n", + encoding="utf-8", + ) + absolute_plan.write_text( + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + f"| `{source}` | TEST-1 |\n", + encoding="utf-8", + ) + relative, relative_known = dispatch.extract_write_set( + relative_plan, workspace + ) + absolute, absolute_known = dispatch.extract_write_set( + absolute_plan, workspace + ) + self.assertTrue(relative_known) + self.assertTrue(absolute_known) + self.assertEqual(relative, absolute) + self.assertEqual(relative, {str(source.resolve())}) + + def test_recovery_restores_write_set_from_matching_archived_plan(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + task = workspace / "agent-task" / "recovery" + task.mkdir(parents=True) + header = "\n" + (task / "plan_local_G05_2.log").write_text( + header + + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + "| `src/recovery.go` | TEST-1 |\n", + encoding="utf-8", + ) + (task / "code_review_local_G05_2.log").write_text( + header + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n- ์ข…ํ•ฉ ํŒ์ •: WARN\n", + encoding="utf-8", + ) + (task / "plan_cloud_G09_3.log").write_text( + "\n" + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + "| `src/unrelated.go` | OTHER-1 |\n", + encoding="utf-8", + ) + [scanned] = dispatch.scan_tasks(workspace, None) + self.assertTrue(scanned.write_set_known) + self.assertEqual( + scanned.write_set, + {str((workspace / "src" / "recovery.go").resolve())}, + ) + self.assertEqual(scanned.errors, []) + + def test_recovery_without_matching_plan_fails_closed(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + task = workspace / "agent-task" / "recovery" + task.mkdir(parents=True) + (task / "code_review_local_G05_2.log").write_text( + "\n" + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n" + "- ์ข…ํ•ฉ ํŒ์ •: WARN\n", + encoding="utf-8", + ) + [scanned] = dispatch.scan_tasks(workspace, None) + self.assertFalse(scanned.write_set_known) + self.assertEqual( + scanned.errors, + ["PLAN Modified Files Summary๋ฅผ ๋ณต๊ตฌํ•  matching PLAN log๊ฐ€ ์—†๋‹ค"], + ) + + def test_rejects_broad_or_outside_workspace_write_sets(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / "src").mkdir() + plan = workspace / "unsafe.md" + plan.write_text( + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + "| `src/exact.go` | TEST-0 |\n" + "| `src/` | TEST-1 |\n" + "| `../outside.go` | TEST-2 |\n" + "| `src/*.go` | TEST-3 |\n" + "| | TEST-4 |\n" + "| `` | TEST-5 |\n" + "| `src\\windows.go` | TEST-6 |\n", + encoding="utf-8", + ) + write_set, known = dispatch.extract_write_set(plan, workspace) + inspected, diagnostics = dispatch.inspect_write_set(plan, workspace) + self.assertFalse(known) + self.assertEqual(write_set, set()) + self.assertEqual(inspected, {str((workspace / "src/exact.go").resolve())}) + self.assertIn( + "๋””๋ ‰ํ„ฐ๋ฆฌ claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: src/", + diagnostics, + ) + self.assertIn( + "workspace ๋ฐ– claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: ../outside.go", + diagnostics, + ) + self.assertIn( + "glob ๋˜๋Š” broad path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: src/*.go", + diagnostics, + ) + self.assertIn( + "์ •ํ™•ํ•œ backtick workspace ํŒŒ์ผ ๊ฒฝ๋กœ๊ฐ€ ์—†๋Š” claim ํ–‰: " + "", + diagnostics, + ) + self.assertIn( + "placeholder ๋˜๋Š” malformed path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: ", + diagnostics, + ) + self.assertIn( + r"malformed path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: src\windows.go", + diagnostics, + ) + + def test_active_plan_without_valid_modified_files_summary_fails_closed(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + directory = workspace / "agent-task" / "missing-write-set" + directory.mkdir(parents=True) + header = "\n" + (directory / "PLAN-local-G05.md").write_text( + header + "## Background\n\nNo file table.\n", + encoding="utf-8", + ) + (directory / "CODE_REVIEW-local-G05.md").write_text( + header, + encoding="utf-8", + ) + + [task] = dispatch.scan_tasks(workspace, None) + + self.assertFalse(task.write_set_known) + self.assertEqual( + task.errors, + [ + "PLAN Modified Files Summary๊ฐ€ ์œ ํšจํ•˜์ง€ ์•Š๋‹ค: " + "Modified Files Summary ์„น์…˜์ด ์—†๋‹ค" + ], + ) + + def test_validate_plan_mode_reports_precise_invalid_claim(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + plan = workspace / "PLAN-cloud-G10.md" + plan.write_text( + "## Modified Files Summary\n\n" + "| File | Items |\n|---|---|\n" + "| `agent-test/runs/output-filter-recovery/**` | TEST-1 |\n", + encoding="utf-8", + ) + with mock.patch.object( + sys, + "argv", + [ + str(SCRIPT), + "--workspace", + str(workspace), + "--validate-plan", + str(plan), + ], + ), mock.patch("sys.stderr", new_callable=io.StringIO) as stderr: + self.assertEqual(dispatch.main(), 2) + self.assertIn( + "glob ๋˜๋Š” broad path claim์€ ํ—ˆ์šฉ๋˜์ง€ ์•Š๋Š”๋‹ค: " + "agent-test/runs/output-filter-recovery/**", + stderr.getvalue(), + ) + + def test_workspace_claims_persist_replace_wait_and_release_on_completion(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + shared = workspace / "src" / "shared.py" + disjoint = workspace / "src" / "disjoint.py" + expansion = workspace / "src" / "expansion.py" + alpha = self.make_claim_task( + workspace, + "alpha/01_task", + shared, + ) + beta = self.make_claim_task( + workspace, + "beta/01_task", + shared, + ) + gamma = self.make_claim_task( + workspace, + "gamma/01_task", + disjoint, + ) + + store = dispatch.StateStore(workspace) + try: + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, + [(alpha, "worker"), (beta, "worker"), (gamma, "review")], + persist=True, + ) + self.assertEqual(selected, [(gamma, "review"), (alpha, "worker")]) + self.assertEqual( + deferred, + [ + ( + beta, + "worker", + "write claim ์ถฉ๋Œ ๋Œ€๊ธฐ: " + f"owner=alpha/01_task; path={shared.resolve()}", + ) + ], + ) + alpha_acquired_at = store.data["write_claims"][alpha.name][ + "acquired_at" + ] + finally: + store.close() + + reopened = dispatch.StateStore(workspace) + try: + self.assertEqual( + set(reopened.data["write_claims"]), + {alpha.name, gamma.name}, + ) + alpha_followup = self.make_claim_task( + workspace, + alpha.name, + shared, + expansion, + plan_hash="plan-1", + ) + selected, deferred, _ = dispatch.select_dispatch_candidates( + reopened, + [(alpha_followup, "worker")], + persist=True, + ) + self.assertEqual(selected, [(alpha_followup, "worker")]) + self.assertEqual(deferred, []) + self.assertEqual( + reopened.data["write_claims"][alpha.name]["acquired_at"], + alpha_acquired_at, + ) + self.assertEqual( + reopened.data["write_claims"][alpha.name]["paths"], + sorted([str(shared.resolve()), str(expansion.resolve())]), + ) + + conflicting_followup = self.make_claim_task( + workspace, + alpha.name, + shared, + disjoint, + plan_hash="plan-2", + ) + selected, deferred, _ = dispatch.select_dispatch_candidates( + reopened, + [(conflicting_followup, "worker")], + persist=True, + ) + self.assertEqual(selected, []) + self.assertIn(f"owner={gamma.name}", deferred[0][2]) + self.assertEqual( + reopened.data["write_claims"][alpha.name]["plan_hash"], + "plan-1", + ) + + archive = workspace / "archive-alpha" + archive.mkdir() + (archive / "complete.log").write_text( + "complete\n", + encoding="utf-8", + ) + reopened.mark_orchestration_task_complete( + "alpha", + alpha.name, + archive, + ) + self.assertNotIn(alpha.name, reopened.data["write_claims"]) + + selected, deferred, _ = dispatch.select_dispatch_candidates( + reopened, + [(beta, "worker")], + persist=True, + ) + self.assertEqual(selected, [(beta, "worker")]) + self.assertEqual(deferred, []) + finally: + reopened.close() + + def test_unknown_write_set_cannot_acquire_claim(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_claim_task( + workspace, + "group/01_unknown", + workspace / "src" / "unknown.py", + ) + task.write_set_known = False + task.write_set = set() + store = dispatch.StateStore(workspace) + try: + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, + [(task, "worker")], + persist=True, + ) + self.assertEqual(selected, []) + self.assertIn("valid non-empty", deferred[0][2]) + self.assertEqual(store.data["write_claims"], {}) + finally: + store.close() + + def test_claim_preview_is_stateless(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_claim_task( + workspace, + "group/01_preview", + workspace / "src" / "preview.py", + ) + store = dispatch.StateStore(workspace) + try: + before = json.loads(json.dumps(store.data)) + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, + [(task, "worker")], + persist=False, + ) + self.assertEqual(selected, [(task, "worker")]) + self.assertEqual(deferred, []) + self.assertEqual(store.data, before) + self.assertFalse(store.path.exists()) + finally: + store.close() + + def test_active_legacy_task_adopts_exclusive_workspace_claim(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + legacy = self.make_claim_task( + workspace, + "legacy/01_active", + workspace / "src" / "unknown.py", + ) + legacy.write_set_known = False + legacy.write_set = set() + candidate = self.make_claim_task( + workspace, + "other/01_candidate", + workspace / "src" / "disjoint.py", + ) + store = dispatch.StateStore(workspace) + try: + store.adopt_active_write_claim(legacy) + claim = store.data["write_claims"][legacy.name] + self.assertTrue(claim["exclusive"]) + self.assertEqual(claim["paths"], []) + + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, + [(candidate, "worker")], + persist=True, + ) + self.assertEqual(selected, []) + self.assertIn(f"owner={legacy.name}", deferred[0][2]) + self.assertIn("", deferred[0][2]) + finally: + store.close() + + def test_state_workspace_identity_is_persisted_and_validated(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + state_path = store.path + expected_id = store.workspace_id + try: + store.save() + finally: + store.close() + + state = json.loads(state_path.read_text(encoding="utf-8")) + self.assertEqual( + state["workspace_identity"], + {"id": expected_id, "root": str(workspace.resolve())}, + ) + state["workspace_identity"]["root"] = str( + (workspace / "foreign").resolve() + ) + state_path.write_text(json.dumps(state), encoding="utf-8") + + with self.assertRaises(dispatch.DispatcherTerminalStateError): + dispatch.StateStore(workspace) + + def test_review_progress_signature_ignores_dispatcher_work_log(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + task = TaskStageTest().make_task(workspace) + before = dispatch.task_signature(workspace, task) + + (workspace / dispatch.WORK_LOG_NAME).write_text( + "| FINISH | test | review |\n", encoding="utf-8" + ) + after_work_log = dispatch.task_signature(workspace, task) + self.assertEqual(after_work_log, before) + + assert task.review is not None + task.review.write_text( + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n- ์ข…ํ•ฉ ํŒ์ •: WARN\n", + encoding="utf-8", + ) + after_review = dispatch.task_signature(workspace, task) + self.assertNotEqual(after_review, before) + + +class WorkLogArchiveTest(unittest.TestCase): + def complete_archive( + self, + workspace: Path, + task_name: str, + month: str = "07", + suffix: str = "", + ) -> Path: + archive = ( + workspace + / "agent-task" + / "archive" + / "2026" + / month + / f"{task_name}{suffix}" + ) + archive.mkdir(parents=True) + (archive / "complete.log").write_text( + f"complete {task_name}\n", + encoding="utf-8", + ) + return archive + + def test_archives_split_group_log_with_next_cross_month_number(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active_group = workspace / "agent-task" / "group" + active_group.mkdir(parents=True) + source = active_group / dispatch.WORK_LOG_NAME + source.write_text("final timeline\n", encoding="utf-8") + (active_group / "01_done").mkdir() + old_group = ( + workspace + / "agent-task" + / "archive" + / "2026" + / "06" + / "group" + ) + old_group.mkdir(parents=True) + (old_group / "work_log_0.log").write_text( + "old timeline\n", + encoding="utf-8", + ) + archive = self.complete_archive( + workspace, + "group/01_done", + ) + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"group/01_done"}, + {"group/01_done": str(archive)}, + set(), + ) + + destination = archive.parent / "work_log_1.log" + self.assertEqual(errors, {}) + self.assertEqual( + archived, + {"group": str(destination.resolve())}, + ) + self.assertEqual( + destination.read_text(encoding="utf-8"), + "final timeline\n", + ) + self.assertFalse(source.exists()) + self.assertFalse(active_group.exists()) + + def test_does_not_archive_until_every_group_task_is_complete_and_idle(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active_group = workspace / "agent-task" / "group" + active_group.mkdir(parents=True) + source = active_group / dispatch.WORK_LOG_NAME + source.write_text("in progress\n", encoding="utf-8") + archive = self.complete_archive( + workspace, + "group/01_done", + ) + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"group/01_done", "group/02_open"}, + {"group/01_done": str(archive)}, + {"group/02_open"}, + ) + + self.assertEqual(archived, {}) + self.assertEqual(errors, {}) + self.assertEqual( + source.read_text(encoding="utf-8"), + "in progress\n", + ) + self.assertFalse((archive.parent / "work_log_0.log").exists()) + + def test_archives_single_task_log_inside_its_suffixed_archive(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active_group = workspace / "agent-task" / "single" + active_group.mkdir(parents=True) + source = active_group / dispatch.WORK_LOG_NAME + source.write_text("single timeline\n", encoding="utf-8") + archive = self.complete_archive( + workspace, + "single", + suffix="_1", + ) + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"single"}, + {"single": str(archive)}, + set(), + ) + + destination = archive / "work_log_0.log" + self.assertEqual(errors, {}) + self.assertEqual( + archived, + {"single": str(destination.resolve())}, + ) + self.assertEqual( + destination.read_text(encoding="utf-8"), + "single timeline\n", + ) + self.assertFalse(active_group.exists()) + + def test_closes_unmatched_start_before_archiving_completed_group(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + source = ( + workspace + / "agent-task" + / "group" + / dispatch.WORK_LOG_NAME + ) + locator = workspace / "runs" / "attempt" / "locator.json" + dispatch.append_work_log_event( + source, + task_name="group/01_done", + loop=0, + event="START", + execution_id="group__01_done__p0__review__a00", + role="review", + attempt=0, + model="codex/gpt-5.6-sol xhigh", + result="running", + locator=locator, + ) + archive = self.complete_archive( + workspace, + "group/01_done", + ) + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"group/01_done"}, + {"group/01_done": str(archive)}, + set(), + ) + + destination = archive.parent / "work_log_0.log" + text = destination.read_text(encoding="utf-8") + self.assertEqual(errors, {}) + self.assertEqual( + archived, + {"group": str(destination.resolve())}, + ) + self.assertEqual(text.count("| START |"), 1) + self.assertEqual(text.count("| FINISH |"), 1) + self.assertIn( + "reconciled:verified-complete-archive", + text, + ) + self.assertEqual( + dispatch.unfinished_work_log_attempts(destination), + [], + ) + + def test_normalizes_single_task_work_log_moved_by_generic_review(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + archive = self.complete_archive(workspace, "single") + legacy = archive / dispatch.WORK_LOG_NAME + legacy.write_text("legacy timeline\n", encoding="utf-8") + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"single"}, + {"single": str(archive)}, + set(), + ) + + destination = archive / "work_log_0.log" + self.assertEqual(errors, {}) + self.assertEqual( + archived, + {"single": str(destination.resolve())}, + ) + self.assertFalse(legacy.exists()) + self.assertEqual( + destination.read_text(encoding="utf-8"), + "legacy timeline\n", + ) + + def test_merges_active_and_archived_work_logs_after_review_move(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active = workspace / "agent-task" / "single" / dispatch.WORK_LOG_NAME + archive = self.complete_archive(workspace, "single") + legacy = archive / dispatch.WORK_LOG_NAME + locator = workspace / "runs" / "review" / "locator.json" + locator_text = str(locator.resolve()) + legacy.parent.mkdir(parents=True, exist_ok=True) + legacy.write_text( + "# Milestone Work Log\n\n" + "> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n" + "| seq | time | event | task | role | attempt | model | result | locator |\n" + "|---:|---|---|---|---|---:|---|---|---|\n" + f"| 1 | 26-07-30 06:34:00 | START | single | review | 0 | codex | running | {locator_text} |\n", + encoding="utf-8", + ) + active.parent.mkdir(parents=True, exist_ok=True) + active.write_text( + "# Milestone Work Log\n\n" + "> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n" + "| seq | time | event | task | role | attempt | model | result | locator |\n" + "|---:|---|---|---|---|---:|---|---|---|\n" + f"| 1 | 26-07-30 06:43:00 | FINISH | single | review | 0 | codex | succeeded:0 | {locator_text} |\n", + encoding="utf-8", + ) + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"single"}, + {"single": str(archive)}, + set(), + ) + + destination = archive / "work_log_0.log" + self.assertEqual(errors, {}) + self.assertEqual( + archived, + {"single": str(destination.resolve())}, + ) + self.assertFalse(active.exists()) + self.assertFalse(legacy.exists()) + self.assertEqual( + len( + [ + line + for line in destination.read_text(encoding="utf-8").splitlines() + if (cells := dispatch.work_log_event_cells(line)) + and cells[2] in {"START", "FINISH"} + ] + ), + 2, + ) + self.assertEqual(dispatch.unfinished_work_log_attempts(destination), []) + + def test_preserves_sources_when_work_log_merge_rows_conflict(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active = workspace / "agent-task" / "single" / dispatch.WORK_LOG_NAME + archive = self.complete_archive(workspace, "single") + legacy = archive / dispatch.WORK_LOG_NAME + locator = workspace / "runs" / "worker" / "locator.json" + locator_text = str(locator.resolve()) + header = ( + "# Milestone Work Log\n\n" + "> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n" + "| seq | time | event | task | role | attempt | model | result | locator |\n" + "|---:|---|---|---|---|---:|---|---|---|\n" + ) + legacy.write_text( + header + + f"| 1 | 26-07-30 06:34:00 | START | single | worker | 0 | agy | running | {locator_text} |\n", + encoding="utf-8", + ) + active.parent.mkdir(parents=True, exist_ok=True) + active.write_text( + header + + f"| 1 | 26-07-30 06:35:00 | START | single | worker | 0 | agy | running | {locator_text} |\n", + encoding="utf-8", + ) + + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"single"}, + {"single": str(archive)}, + set(), + ) + + self.assertEqual(archived, {}) + self.assertIn("WORK_LOG ๋ณ‘ํ•ฉ ์ถฉ๋Œ", errors["single"]) + self.assertTrue(active.exists()) + self.assertTrue(legacy.exists()) + self.assertFalse((archive / "work_log_0.log").exists()) + + def test_archive_failure_preserves_source_and_reports_retryable_error(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active_group = workspace / "agent-task" / "group" + active_group.mkdir(parents=True) + source = active_group / dispatch.WORK_LOG_NAME + source.write_text("keep me\n", encoding="utf-8") + archive = self.complete_archive( + workspace, + "group/01_done", + ) + + with mock.patch.object( + Path, + "replace", + side_effect=OSError("disk full"), + ): + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"group/01_done"}, + {"group/01_done": str(archive)}, + set(), + ) + + self.assertEqual(archived, {}) + self.assertIn("group", errors) + self.assertIn("disk full", errors["group"]) + self.assertEqual( + source.read_text(encoding="utf-8"), + "keep me\n", + ) + + def test_existing_destination_is_never_overwritten(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + active_group = workspace / "agent-task" / "group" + active_group.mkdir(parents=True) + source = active_group / dispatch.WORK_LOG_NAME + source.write_text("new timeline\n", encoding="utf-8") + archive = self.complete_archive( + workspace, + "group/01_done", + ) + destination = archive.parent / "work_log_0.log" + destination.write_text("existing timeline\n", encoding="utf-8") + + with mock.patch.object( + dispatch, + "next_work_log_archive_number", + return_value=0, + ): + archived, errors = dispatch.archive_completed_group_work_logs( + workspace, + {"group/01_done"}, + {"group/01_done": str(archive)}, + set(), + ) + + self.assertEqual(archived, {}) + self.assertIn("์ด๋ฏธ ์กด์žฌํ•œ๋‹ค", errors["group"]) + self.assertEqual( + destination.read_text(encoding="utf-8"), + "existing timeline\n", + ) + self.assertEqual( + source.read_text(encoding="utf-8"), + "new timeline\n", + ) + + def test_process_marker_recovers_liveness_when_pid_write_was_lost(self): + with tempfile.TemporaryDirectory() as temporary: + locator = Path(temporary) / "locator.json" + locator.write_text( + json.dumps( + { + "status": "running", + "agent_process_marker": "attempt-marker", + } + ), + encoding="utf-8", + ) + + with mock.patch.object( + dispatch, + "marked_agent_process_pids", + return_value=[123, 456], + ): + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + self.assertTrue(live) + self.assertIn("123,456", detail) + + with mock.patch.object( + dispatch, + "marked_agent_process_pids", + return_value=[], + ): + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + self.assertFalse(live) + self.assertIn("absent from the process table", detail) + + def test_process_marker_finds_spawned_agent_process(self): + marker = f"test-marker-{uuid.uuid4()}" + environment = { + **os.environ, + dispatch.AGENT_PROCESS_MARKER_ENV: marker, + } + process = subprocess.Popen( + [ + sys.executable, + "-c", + "import time; time.sleep(5)", + ], + env=environment, + ) + try: + found: list[int] = [] + for _ in range(50): + found = dispatch.marked_agent_process_pids(marker) + if process.pid in found: + break + time.sleep(0.01) + self.assertIn(process.pid, found) + finally: + process.terminate() + process.wait(timeout=5) + + def test_workspace_bound_liveness_rejects_foreign_and_accepts_current_locators(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) / "current" + workspace.mkdir() + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + try: + foreign_attempt = Path(temporary) / "foreign" / "attempt" + foreign_attempt.mkdir(parents=True) + foreign_locator = foreign_attempt / "locator.json" + foreign_locator.write_text( + json.dumps( + { + "status": "running", + "workspace": str((Path(temporary) / "foreign").resolve()), + "workspace_id": "foreign-workspace", + "agent_pid": os.getpid(), + } + ), + encoding="utf-8", + ) + with mock.patch.object( + dispatch, + "process_is_alive", + side_effect=AssertionError( + "foreign locator must be rejected before PID inspection" + ), + ): + live, detail = dispatch.external_active_is_live( + {"active_locator": str(foreign_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + self.assertFalse(live) + self.assertIn("foreign workspace locator path", detail) + + current_attempt = store.runs / "current-attempt" + current_attempt.mkdir() + current_locator = current_attempt / "locator.json" + current_locator.write_text( + json.dumps( + { + "status": "running", + "workspace": str(store.workspace), + "workspace_id": store.workspace_id, + "agent_pid": os.getpid(), + } + ), + encoding="utf-8", + ) + live, detail = dispatch.external_active_is_live( + {"active_locator": str(current_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + self.assertTrue(live) + self.assertIn("agent_pid", detail) + + legacy_attempt = store.runs / "legacy-attempt" + legacy_attempt.mkdir() + legacy_locator = legacy_attempt / "locator.json" + legacy_locator.write_text( + json.dumps( + { + "status": "running", + "agent_pid": os.getpid(), + } + ), + encoding="utf-8", + ) + live, detail = dispatch.external_active_is_live( + {"active_locator": str(legacy_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + self.assertTrue(live) + self.assertIn("agent_pid", detail) + + foreign_stream = foreign_attempt / "stream.log" + foreign_stream.write_text("foreign output\n", encoding="utf-8") + legacy_locator.write_text( + json.dumps( + { + "status": "running", + "agent_pid": os.getpid(), + "stream_log": str(foreign_stream), + } + ), + encoding="utf-8", + ) + with mock.patch.object( + dispatch, + "process_is_alive", + side_effect=AssertionError( + "foreign stream evidence must fail before PID inspection" + ), + ): + live, detail = dispatch.external_active_is_live( + {"active_locator": str(legacy_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + self.assertFalse(live) + self.assertIn("foreign workspace locator evidence", detail) + finally: + store.close() + + def test_workspace_bound_liveness_rejects_mismatched_identity_inside_runs(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + try: + attempt = store.runs / "foreign-identity" + attempt.mkdir() + locator = attempt / "locator.json" + locator.write_text( + json.dumps( + { + "status": "running", + "workspace": str((workspace / "other").resolve()), + "workspace_id": "foreign-workspace", + "agent_pid": os.getpid(), + } + ), + encoding="utf-8", + ) + with mock.patch.object( + dispatch, + "process_is_alive", + side_effect=AssertionError( + "mismatched workspace must fail before PID inspection" + ), + ): + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + self.assertFalse(live) + self.assertIn("foreign workspace locator id", detail) + finally: + store.close() + + def test_workspace_bound_laguna_resume_rejects_foreign_locator_and_native_session(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) / "current" + workspace.mkdir() + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + try: + foreign_attempt = Path(temporary) / "foreign-attempt" + foreign_attempt.mkdir() + foreign_native = foreign_attempt / "session.jsonl" + foreign_native.write_text("{}\n", encoding="utf-8") + foreign_locator = foreign_attempt / "locator.json" + foreign_locator.write_text( + json.dumps( + { + "status": "failed", + "workspace": str((Path(temporary) / "foreign").resolve()), + "workspace_id": "foreign-workspace", + "cli": "pi", + "model": "laguna-s:2.1", + "failure_class": "session-stall", + "native_session_path": str(foreign_native), + } + ), + encoding="utf-8", + ) + self.assertIsNone( + dispatch.laguna_resume_locator( + {"active_locator": str(foreign_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + ) + + current_attempt = store.runs / "current-laguna" + current_attempt.mkdir() + current_native = current_attempt / "session.jsonl" + current_native.write_text("{}\n", encoding="utf-8") + current_locator = current_attempt / "locator.json" + record = { + "status": "failed", + "workspace": str(store.workspace), + "workspace_id": store.workspace_id, + "cli": "pi", + "model": "laguna-s:2.1", + "failure_class": "session-stall", + "native_session_path": str(current_native), + } + current_locator.write_text( + json.dumps(record), + encoding="utf-8", + ) + self.assertEqual( + dispatch.laguna_resume_locator( + {"active_locator": str(current_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ), + current_locator, + ) + + record["native_session_path"] = str(foreign_native) + current_locator.write_text( + json.dumps(record), + encoding="utf-8", + ) + self.assertIsNone( + dispatch.laguna_resume_locator( + {"active_locator": str(current_locator)}, + expected_workspace=store.workspace, + expected_workspace_id=store.workspace_id, + expected_runs_root=store.runs, + ) + ) + finally: + store.close() + + def test_orchestration_keeps_pidless_stream_evidence_active(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + try: + attempt = store.runs / "attempt" + attempt.mkdir() + stream = attempt / "stream.log" + stream.write_text("reasoning\n", encoding="utf-8") + locator = attempt / "locator.json" + locator.write_text( + json.dumps( + { + "status": "running", + "workspace": str(workspace.resolve()), + "workspace_id": store.workspace_id, + "cli": "codex", + "stream_log": str(stream), + } + ), + encoding="utf-8", + ) + store.data["orchestrations"] = { + "group": { + "status": "running", + "tasks": { + "group/01_done": { + "status": "active", + "archive": None, + } + }, + } + } + store.data["tasks"] = { + "group/01_done": { + "active_locator": str(locator), + } + } + + live = dispatch.orchestration_live_agent_processes( + store, + "group", + ) + + self.assertIn("group/01_done", live) + self.assertIn( + "time-based duplicate recovery is disabled", + live["group/01_done"], + ) + finally: + store.close() + + def test_restart_waits_for_live_writer_then_reconciles_and_archives(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task_directory = workspace / "agent-task" / "group" / "01_done" + task_directory.mkdir(parents=True) + plan = task_directory / "PLAN-local-G05.md" + review = task_directory / "CODE_REVIEW-local-G05.md" + plan.write_text( + "\n", + encoding="utf-8", + ) + review.write_text( + "\n", + encoding="utf-8", + ) + task = dispatch.Task( + name="group/01_done", + directory=task_directory, + plan=plan, + review=review, + user_review=None, + recovery=False, + plan_hash=dispatch.sha256_file(plan), + lane="local", + grade=5, + ) + source = dispatch.append_milestone_event( + task, + event="START", + execution_id="group__01_done__p0__review__a00", + role="review", + attempt=0, + model="codex/gpt-5.6-sol xhigh", + result="running", + locator=workspace / "runs" / "attempt" / "locator.json", + ) + store = dispatch.StateStore(workspace) + try: + store.prepare_orchestration("group", [task], workspace) + archive = ( + workspace + / "agent-task" + / "archive" + / "2026" + / "07" + / "group" + / "01_done" + ) + archive.parent.mkdir(parents=True) + (task_directory / "complete.log").write_text( + "complete\n", + encoding="utf-8", + ) + task_directory.rename(archive) + destination = archive.parent / "work_log_0.log" + wait_observations: list[bool] = [] + + async def observe_wait(seconds): + self.assertEqual( + seconds, + dispatch.STREAM_HEARTBEAT_SECONDS, + ) + wait_observations.append( + source.is_file() and not destination.exists() + ) + + with ( + mock.patch.object( + dispatch, + "orchestration_live_agent_processes", + side_effect=[ + {"group/01_done": "agent_pid=123 alive"}, + {}, + ], + ), + mock.patch.object( + dispatch.asyncio, + "sleep", + new=observe_wait, + ), + ): + result = asyncio.run( + dispatch.dispatch_with_store( + SimpleNamespace( + task_group="group", + retry_blocked=False, + dry_run=False, + ), + workspace, + store, + ) + ) + + self.assertEqual(result, 0) + self.assertEqual(wait_observations, [True]) + self.assertFalse(source.exists()) + self.assertTrue(destination.is_file()) + text = destination.read_text(encoding="utf-8") + self.assertIn("| FINISH |", text) + self.assertIn( + "reconciled:verified-complete-archive", + text, + ) + self.assertEqual( + store.data["orchestrations"]["group"]["status"], + "complete", + ) + finally: + store.close() + + def test_dispatcher_returns_three_when_completed_log_archive_needs_retry(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + (workspace / "agent-task").mkdir() + archive = self.complete_archive( + workspace, + "group/01_done", + ) + store = dispatch.StateStore(workspace) + try: + store.mark_orchestration_task_complete( + "group", + "group/01_done", + archive, + ) + args = SimpleNamespace( + task_group="group", + retry_blocked=False, + dry_run=False, + ) + with ( + mock.patch.object(dispatch, "scan_tasks", return_value=[]), + mock.patch.object( + dispatch, + "archive_completed_group_work_logs", + return_value=({}, {"group": "disk full"}), + ), + ): + result = asyncio.run( + dispatch.dispatch_with_store( + args, + workspace, + store, + ) + ) + + self.assertEqual(result, 3) + self.assertEqual( + store.data["orchestrations"]["group"]["status"], + "running", + ) + finally: + store.close() + + +class OrchestrationPersistenceTest(unittest.TestCase): + def make_task(self, workspace: Path, name: str = "task"): + directory = workspace / "agent-task" / name + directory.mkdir(parents=True) + plan = directory / "PLAN-local-G05.md" + review = directory / "CODE_REVIEW-local-G05.md" + plan.write_text( + f"\n" + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + "| `src/task.go` | TEST-1 |\n", + encoding="utf-8", + ) + review.write_text( + f"\n", + encoding="utf-8", + ) + return dispatch.scan_tasks(workspace, None)[0] + + def test_complete_archive_removes_only_its_task_attempt_logs(self): + with tempfile.TemporaryDirectory() as temporary: + runs = Path(temporary) / "runs" + completed = runs / "completed-attempt" + other = runs / "other-attempt" + for attempt, task_name in ((completed, "group/01_done"), (other, "group/02_open")): + attempt.mkdir(parents=True) + (attempt / "locator.json").write_text( + json.dumps({"task": task_name}), encoding="utf-8" + ) + for name in ("stream.log", "heartbeat.log", "session.jsonl"): + (attempt / name).write_text("evidence\n", encoding="utf-8") + + removed = dispatch.cleanup_completed_task_attempt_logs( + runs, "group/01_done" + ) + + self.assertEqual(removed, 1) + self.assertFalse(completed.exists()) + self.assertTrue(other.is_dir()) + self.assertTrue((other / "stream.log").is_file()) + + def test_mark_complete_removes_task_attempt_logs(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + try: + attempt = store.runs / "attempt" + attempt.mkdir() + (attempt / "locator.json").write_text( + json.dumps({"task": "group/01_done"}), encoding="utf-8" + ) + (attempt / "stream.log").write_text("stream\n", encoding="utf-8") + archive = workspace / "archive" + archive.mkdir() + (archive / "complete.log").write_text("complete\n", encoding="utf-8") + + store.mark_orchestration_task_complete( + "group", "group/01_done", archive + ) + + self.assertFalse(attempt.exists()) + finally: + store.close() + + def test_reconcile_keeps_attempt_logs_until_active_writer_exits(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + archive = workspace / "archive" + archive.mkdir() + (archive / "complete.log").write_text( + "complete\n", + encoding="utf-8", + ) + store = dispatch.StateStore(workspace) + try: + store.mark_orchestration_task_complete( + "group", + "group/01_done", + archive, + ) + attempt = store.runs / "live-attempt" + attempt.mkdir() + (attempt / "locator.json").write_text( + json.dumps({"task": "group/01_done"}), + encoding="utf-8", + ) + (attempt / "stream.log").write_text( + "still running\n", + encoding="utf-8", + ) + + completed, errors = store.reconcile_orchestration( + "group", + workspace, + {"group/01_done"}, + ) + + self.assertEqual(errors, {}) + self.assertIn("group/01_done", completed) + self.assertTrue(attempt.is_dir()) + + store.reconcile_orchestration("group", workspace, set()) + self.assertFalse(attempt.exists()) + finally: + store.close() + + def test_attempt_log_cleanup_failure_does_not_revoke_completion(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + store = dispatch.StateStore(workspace) + try: + attempt = store.runs / "attempt" + attempt.mkdir() + (attempt / "locator.json").write_text( + json.dumps({"task": "group/01_done"}), encoding="utf-8" + ) + archive = workspace / "archive" + archive.mkdir() + (archive / "complete.log").write_text( + "complete\n", encoding="utf-8" + ) + + with mock.patch.object( + dispatch.shutil, + "rmtree", + side_effect=OSError("transient cleanup failure"), + ): + store.mark_orchestration_task_complete( + "group", "group/01_done", archive + ) + with mock.patch.object( + dispatch, + "scan_tasks", + return_value=[], + ): + result = asyncio.run( + dispatch.dispatch_with_store( + SimpleNamespace( + task_group="group", + retry_blocked=False, + dry_run=False, + ), + workspace, + store, + ) + ) + + record = store.data["orchestrations"]["group"]["tasks"][ + "group/01_done" + ] + self.assertEqual(result, 3) + self.assertEqual(record["status"], "complete") + self.assertTrue(attempt.is_dir()) + self.assertEqual( + dispatch.cleanup_completed_task_attempt_logs( + store.runs, "group/01_done" + ), + 1, + ) + self.assertFalse(attempt.exists()) + with mock.patch.object(dispatch, "scan_tasks", return_value=[]): + result = asyncio.run( + dispatch.dispatch_with_store( + SimpleNamespace( + task_group="group", + retry_blocked=False, + dry_run=False, + ), + workspace, + store, + ) + ) + self.assertEqual(result, 0) + finally: + store.close() + + def test_attempt_log_cleanup_pending_precedes_other_terminal_blocker(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + pending_dir = workspace / "agent-task" / "group" / "02_pending" + pending_dir.mkdir(parents=True) + pending = TaskStageTest().make_task(pending_dir) + pending.name = "group/02_pending" + pending.index = 2 + pending.plan_hash = "pending-hash" + + store = dispatch.StateStore(workspace) + try: + store.prepare_orchestration("group", [pending], workspace) + attempt = store.runs / "attempt" + attempt.mkdir() + (attempt / "locator.json").write_text( + json.dumps({"task": "group/01_done"}), encoding="utf-8" + ) + archive = workspace / "archive" + archive.mkdir() + (archive / "complete.log").write_text( + "complete\n", encoding="utf-8" + ) + + with ( + mock.patch.object( + dispatch.shutil, + "rmtree", + side_effect=OSError("transient cleanup failure"), + ), + mock.patch.object(dispatch, "scan_tasks", return_value=[]), + ): + store.mark_orchestration_task_complete( + "group", "group/01_done", archive + ) + result = asyncio.run( + dispatch.dispatch_with_store( + SimpleNamespace( + task_group="group", + retry_blocked=False, + dry_run=False, + ), + workspace, + store, + ) + ) + + self.assertEqual(result, 3) + self.assertEqual( + store.data["orchestrations"]["group"]["status"], + "running", + ) + self.assertTrue(attempt.is_dir()) + finally: + store.close() + + def test_external_liveness_ignores_heartbeat_mtime(self): + with tempfile.TemporaryDirectory() as temporary: + attempt = Path(temporary) + stream = attempt / "stream.log" + heartbeat = attempt / "heartbeat.log" + locator = attempt / "locator.json" + stream.write_text("old model output\n", encoding="utf-8") + heartbeat.write_text("fresh heartbeat\n", encoding="utf-8") + stale_at = time.time() - dispatch.CODEX_STREAM_STALL_SECONDS - 1 + os.utime(stream, (stale_at, stale_at)) + locator.write_text( + json.dumps( + { + "status": "running", + "cli": "agy", + "stream_log": str(stream), + "heartbeat_log": str(heartbeat), + } + ), + encoding="utf-8", + ) + + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + self.assertTrue(live) + self.assertIn("stream inactive=", detail) + self.assertIn("time-based duplicate recovery is disabled", detail) + + record = json.loads(locator.read_text(encoding="utf-8")) + record["agent_pid"] = 999_999_999 + locator.write_text(json.dumps(record), encoding="utf-8") + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + self.assertFalse(live) + self.assertIn("recorded agent process identity", detail) + + record.pop("agent_pid") + locator.write_text(json.dumps(record), encoding="utf-8") + stream.touch() + live, _ = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + self.assertTrue(live) + + def test_external_liveness_keeps_silent_live_agent_process(self): + with tempfile.TemporaryDirectory() as temporary: + attempt = Path(temporary) + stream = attempt / "stream.log" + locator = attempt / "locator.json" + stream.write_text("old model output\n", encoding="utf-8") + stale_at = time.time() - (60 * 60) + os.utime(stream, (stale_at, stale_at)) + locator.write_text( + json.dumps( + { + "status": "running", + "cli": "pi", + "agent_pid": os.getpid(), + "stream_log": str(stream), + } + ), + encoding="utf-8", + ) + + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + + self.assertTrue(live) + self.assertIn("agent_pid=", detail) + + def test_external_liveness_never_times_out_pidless_exact_tool_execution(self): + with tempfile.TemporaryDirectory() as temporary: + attempt = Path(temporary) + stream = attempt / "stream.log" + native = attempt / "session.jsonl" + locator = attempt / "locator.json" + stream.write_text("old model output\n", encoding="utf-8") + native.write_text( + pi_session_jsonl( + [ + { + "type": "message", + "message": { + "role": "assistant", + "content": [ + { + "type": "toolCall", + "id": "slow-tool", + "name": "bash", + } + ], + }, + } + ] + ), + encoding="utf-8", + ) + stale_at = time.time() - (24 * 60 * 60) + os.utime(stream, (stale_at, stale_at)) + os.utime(native, (stale_at, stale_at)) + locator.write_text( + json.dumps( + { + "status": "running", + "cli": "pi", + "stream_log": str(stream), + "native_session_path": str(native), + } + ), + encoding="utf-8", + ) + + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + + self.assertTrue(live) + self.assertIn("time-based duplicate recovery is disabled", detail) + + record = json.loads(locator.read_text(encoding="utf-8")) + record["agent_pid"] = 999_999_999 + locator.write_text(json.dumps(record), encoding="utf-8") + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + self.assertFalse(live) + self.assertIn("recorded agent process identity", detail) + + def test_external_liveness_rejects_reused_pid_identity(self): + with tempfile.TemporaryDirectory() as temporary: + attempt = Path(temporary) + stream = attempt / "stream.log" + locator = attempt / "locator.json" + stream.write_text("old model output\n", encoding="utf-8") + stale_at = time.time() - dispatch.CODEX_STREAM_STALL_SECONDS - 1 + os.utime(stream, (stale_at, stale_at)) + locator.write_text( + json.dumps( + { + "status": "running", + "cli": "agy", + "agent_pid": os.getpid(), + "agent_process_start_token": "not-the-current-process", + "stream_log": str(stream), + } + ), + encoding="utf-8", + ) + + live, detail = dispatch.external_active_is_live( + {"active_locator": str(locator)} + ) + + self.assertFalse(live) + self.assertIn("recorded agent process identity", detail) + + def test_retry_blocked_only_clears_selected_task_group(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + self.make_task(workspace, "alpha/01_task") + self.make_task(workspace, "beta/01_task") + tasks = { + task.name: task for task in dispatch.scan_tasks(workspace, None) + } + alpha = tasks["alpha/01_task"] + beta = tasks["beta/01_task"] + store = dispatch.StateStore(workspace) + try: + for task in (alpha, beta): + store.update_task( + task, + blocked="recovery failure limit exhausted: 10/10", + review_no_progress=10, + selfcheck_incomplete=10, + recovery_failures={"worker": 10}, + ) + + store.clear_blocked("alpha") + + alpha_state = store.task_state(alpha) + self.assertIsNone(alpha_state["blocked"]) + self.assertEqual(alpha_state["review_no_progress"], 0) + self.assertEqual(alpha_state["selfcheck_incomplete"], 0) + self.assertEqual(alpha_state["recovery_failures"], {}) + + beta_state = store.task_state(beta) + self.assertIsNotNone(beta_state["blocked"]) + self.assertEqual(beta_state["review_no_progress"], 10) + self.assertEqual(beta_state["selfcheck_incomplete"], 10) + self.assertEqual(beta_state["recovery_failures"], {"worker": 10}) + finally: + store.close() + + def test_legacy_promotion_recovery_requires_older_source_and_typed_event(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + self.make_task(workspace, "alpha/01_task") + task = dispatch.scan_tasks(workspace, None)[0] + runs = workspace / ".git" / "agent-task-dispatcher" / "runs" + locators = write_legacy_quota_attempts(runs, task) + locator = locators[-1] + state = { + "blocked": ( + "worker recovery failure limit exhausted: 10/10 " + f"locator={locator}" + ), + "recovery_failures": {"worker": 10}, + } + + recovery = dispatch.legacy_promotion_recovery( + runs, + task, + state, + ) + self.assertIsNotNone(recovery) + assert recovery is not None + self.assertEqual(recovery.failure_class, "provider-quota") + self.assertEqual(recovery.evidence_source, "claude:stdout") + + record = json.loads(locator.read_text(encoding="utf-8")) + record["dispatcher_source_sha256"] = ( + dispatch.DISPATCHER_SOURCE_SHA256 + ) + locator.write_text(json.dumps(record), encoding="utf-8") + self.assertIsNone( + dispatch.legacy_promotion_recovery(runs, task, state) + ) + + def test_legacy_promotion_recovery_rejects_mixed_attempt_history(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + self.make_task(workspace, "alpha/01_task") + task = dispatch.scan_tasks(workspace, None)[0] + runs = workspace / ".git" / "agent-task-dispatcher" / "runs" + locators = write_legacy_quota_attempts(runs, task) + mixed_stream = locators[4].parent / "stream.log" + mixed_stream.write_text( + "[stdout] " + + json.dumps( + { + "type": "assistant", + "message": { + "content": "You've hit your session limit" + }, + } + ) + + "\n", + encoding="utf-8", + ) + state = { + "blocked": ( + "worker recovery failure limit exhausted: 10/10 " + f"locator={locators[-1]}" + ), + "recovery_failures": {"worker": 10}, + } + + self.assertIsNone( + dispatch.legacy_promotion_recovery(runs, task, state) + ) + + def test_persisted_legacy_promotion_survives_dispatcher_restart(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + self.make_task(workspace, "alpha/01_task") + task = dispatch.scan_tasks(workspace, None)[0] + runs = workspace / ".git" / "agent-task-dispatcher" / "runs" + locator = write_legacy_quota_attempts(runs, task)[-1] + blocked_state = { + "blocked": ( + "worker recovery failure limit exhausted: 10/10 " + f"locator={locator}" + ), + "recovery_failures": {"worker": 10}, + } + recovery = dispatch.legacy_promotion_recovery( + runs, + task, + blocked_state, + ) + assert recovery is not None + restarted_state = { + "blocked": None, + "recovery_failures": {"worker": 1}, + "legacy_terminal_reclassification": { + "role": recovery.role, + "failure_class": recovery.failure_class, + "evidence_source": recovery.evidence_source, + "prior_dispatcher_sha256": + recovery.prior_dispatcher_sha256, + "current_dispatcher_sha256": + dispatch.DISPATCHER_SOURCE_SHA256, + "locator": str(recovery.locator), + "failed_cli": recovery.failed_cli, + "failed_model": recovery.failed_model, + "failed_reasoning_effort": + recovery.failed_reasoning_effort, + }, + } + + restored = ( + dispatch.pending_persisted_legacy_promotion_recovery( + task, + restarted_state, + ) + ) + + self.assertIsNotNone(restored) + assert restored is not None + self.assertEqual(restored.failed_cli, "claude") + self.assertEqual( + dispatch.promoted_spec( + dispatch.failed_spec_from_recovery(restored), + 0, + ).model, + "gpt-5.6-terra", + ) + + def test_persisted_legacy_agy_promotion_survives_dispatcher_restart(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + self.make_task(workspace, "alpha/01_task") + task = dispatch.scan_tasks(workspace, None)[0] + runs = workspace / ".git" / "agent-task-dispatcher" / "runs" + locator = write_legacy_quota_attempts( + runs, + task, + cli="agy", + model="Gemini 3.6 Flash (High)", + reasoning_effort=None, + )[-1] + blocked_state = { + "blocked": ( + "worker recovery failure limit exhausted: 10/10 " + f"locator={locator}" + ), + "recovery_failures": {"worker": 10}, + } + recovery = dispatch.legacy_promotion_recovery( + runs, + task, + blocked_state, + ) + assert recovery is not None + restarted_state = { + "blocked": None, + "recovery_failures": {"worker": 1}, + "legacy_terminal_reclassification": { + "role": recovery.role, + "failure_class": recovery.failure_class, + "evidence_source": recovery.evidence_source, + "prior_dispatcher_sha256": + recovery.prior_dispatcher_sha256, + "current_dispatcher_sha256": + dispatch.DISPATCHER_SOURCE_SHA256, + "locator": str(recovery.locator), + "failed_cli": recovery.failed_cli, + "failed_model": recovery.failed_model, + "failed_reasoning_effort": + recovery.failed_reasoning_effort, + }, + } + + restored = ( + dispatch.pending_persisted_legacy_promotion_recovery( + task, + restarted_state, + ) + ) + + self.assertIsNotNone(restored) + assert restored is not None + self.assertEqual(restored.failed_cli, "agy") + self.assertEqual(restored.evidence_source, "agy:cli-log") + self.assertEqual( + dispatch.promoted_spec( + dispatch.failed_spec_from_recovery(restored), + 0, + ).cli, + "claude", + ) + + def test_corrupt_persistent_state_fails_closed_and_releases_lock(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + state_root = workspace / ".git" / "agent-task-dispatcher" + state_root.mkdir(parents=True) + state_path = state_root / "state.json" + state_path.write_text("{broken", encoding="utf-8") + + with self.assertRaisesRegex( + RuntimeError, "dispatcher state๋ฅผ ์ฝ์„ ์ˆ˜ ์—†๋‹ค" + ): + dispatch.StateStore(workspace) + + state_path.write_text("{}\n", encoding="utf-8") + reopened = dispatch.StateStore(workspace) + reopened.close() + + def test_live_workspace_lock_is_non_terminal_exit_three(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + (workspace / "agent-task").mkdir() + owner = dispatch.StateStore(workspace) + args = SimpleNamespace( + workspace=str(workspace), + task_group=None, + dry_run=False, + retry_blocked=False, + ) + try: + with mock.patch.object(dispatch, "parse_args", return_value=args): + result = dispatch.main() + self.assertEqual(result, 3) + finally: + owner.close() + + def test_dispatcher_child_rejects_nested_orchestration_before_lock(self): + args = SimpleNamespace( + workspace=".", + task_group="m-test", + dry_run=True, + retry_blocked=False, + validate_plan=None, + ) + with ( + mock.patch.object(dispatch, "parse_args", return_value=args), + mock.patch.dict( + os.environ, + {dispatch.AGENT_PROCESS_MARKER_ENV: "owned-worker"}, + ), + mock.patch.object(dispatch.asyncio, "run") as run, + mock.patch("sys.stderr", new_callable=io.StringIO) as stderr, + ): + result = dispatch.main() + + self.assertEqual(result, 4) + run.assert_not_called() + self.assertIn("nested dispatcher invocation rejected", stderr.getvalue()) + self.assertIn("do not wait for the parent dispatcher", stderr.getvalue()) + + def test_dispatcher_child_can_validate_one_plan(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + plan = workspace / "PLAN-cloud-G10.md" + target = workspace / "src" / "target.go" + plan.write_text( + "## Modified Files Summary\n\n" + "| File | Items |\n|---|---|\n" + f"| `{target}` | TEST-1 |\n", + encoding="utf-8", + ) + args = SimpleNamespace( + workspace=str(workspace), + task_group=None, + dry_run=False, + retry_blocked=False, + validate_plan=str(plan), + ) + with ( + mock.patch.object(dispatch, "parse_args", return_value=args), + mock.patch.dict( + os.environ, + {dispatch.AGENT_PROCESS_MARKER_ENV: "owned-review"}, + ), + ): + result = dispatch.main() + + self.assertEqual(result, 0) + + def test_unexpected_dispatcher_exception_is_non_terminal_exit_three(self): + args = SimpleNamespace( + workspace=".", + task_group=None, + dry_run=False, + retry_blocked=False, + ) + with ( + mock.patch.object(dispatch, "parse_args", return_value=args), + mock.patch.object( + dispatch, + "dispatch", + new=mock.AsyncMock(side_effect=RuntimeError("transient failure")), + ), + ): + result = dispatch.main() + self.assertEqual(result, 3) + + def test_scheduler_exception_waits_for_running_agent_tasks(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + (workspace / "agent-task").mkdir() + args = SimpleNamespace( + workspace=str(workspace), + task_group=None, + dry_run=False, + retry_blocked=False, + ) + events: list[str] = [] + + async def failing_scheduler(args, workspace, store): + async def running_agent(): + events.append("started") + await asyncio.sleep(0.02) + events.append("finished") + + asyncio.create_task(running_agent()) + await asyncio.sleep(0) + raise dispatch.DispatcherTerminalStateError("scheduler failed") + + with mock.patch.object( + dispatch, + "dispatch_with_store", + new=failing_scheduler, + ): + with self.assertRaisesRegex( + dispatch.DispatcherInterruptedWithActiveWork, + "scheduler failed", + ): + asyncio.run(dispatch.dispatch(args)) + + self.assertEqual(events, ["started", "finished"]) + + def test_dry_run_retry_blocked_does_not_clear_persistent_limits(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + store.update_task( + task, + blocked="recovery failure limit exhausted: 10/10", + review_no_progress=10, + selfcheck_incomplete=10, + recovery_failures={"review": 10}, + ) + store.close() + + args = SimpleNamespace( + workspace=str(workspace), + task_group=None, + dry_run=True, + retry_blocked=True, + ) + result = asyncio.run(dispatch.dispatch(args)) + self.assertEqual(result, 2) + + reopened = dispatch.StateStore(workspace) + try: + state = reopened.task_state(task) + self.assertEqual( + state["blocked"], + "recovery failure limit exhausted: 10/10", + ) + self.assertEqual(state["review_no_progress"], 10) + self.assertEqual(state["selfcheck_incomplete"], 10) + self.assertEqual( + state["recovery_failures"], {"review": 10} + ) + finally: + reopened.close() + + def test_child_restart_detects_task_that_disappeared_without_complete_log(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + first = dispatch.StateStore(workspace) + first.prepare_orchestration("__all__", [task], workspace) + first.close() + task.plan.unlink() + task.review.unlink() + task.directory.rmdir() + + restarted = dispatch.StateStore(workspace) + restarted.prepare_orchestration("__all__", [], workspace) + completed, errors = restarted.reconcile_orchestration( + "__all__", workspace, set() + ) + self.assertEqual(completed, {}) + self.assertIn(task.name, errors) + self.assertIn("์ƒˆ complete.log archive ๋ชจ๋‘์—์„œ ์‚ฌ๋ผ์กŒ๋‹ค", errors[task.name]) + restarted.close() + + def test_child_restart_recovers_new_complete_archive_not_baseline_archive(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + old_archive = workspace / "agent-task" / "archive" / "2026" / "06" / "task" + old_archive.mkdir(parents=True) + (old_archive / "complete.log").write_text("old\n", encoding="utf-8") + task = self.make_task(workspace) + first = dispatch.StateStore(workspace) + first.prepare_orchestration("__all__", [task], workspace) + first.close() + + new_archive = workspace / "agent-task" / "archive" / "2026" / "07" / "task_1" + new_archive.parent.mkdir(parents=True) + (task.directory / "complete.log").write_text("new\n", encoding="utf-8") + task.directory.rename(new_archive) + + restarted = dispatch.StateStore(workspace) + restarted.prepare_orchestration("__all__", [], workspace) + completed, errors = restarted.reconcile_orchestration( + "__all__", workspace, set() + ) + self.assertEqual(errors, {}) + self.assertEqual(completed, {"task": str(new_archive.resolve())}) + restarted.close() + + def test_preexisting_incomplete_archive_cannot_become_false_new_completion(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + incomplete_archive = ( + workspace / "agent-task" / "archive" / "2026" / "06" / "task" + ) + incomplete_archive.mkdir(parents=True) + task = self.make_task(workspace) + first = dispatch.StateStore(workspace) + first.prepare_orchestration("__all__", [task], workspace) + first.close() + + task.plan.unlink() + task.review.unlink() + task.directory.rmdir() + (incomplete_archive / "complete.log").write_text( + "late unrelated completion\n", + encoding="utf-8", + ) + + restarted = dispatch.StateStore(workspace) + restarted.prepare_orchestration("__all__", [], workspace) + completed, errors = restarted.reconcile_orchestration( + "__all__", workspace, set() + ) + self.assertEqual(completed, {}) + self.assertIn(task.name, errors) + restarted.close() + + def test_dry_run_does_not_create_persistent_orchestration_state(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + self.make_task(workspace) + args = SimpleNamespace( + workspace=str(workspace), + task_group=None, + dry_run=True, + retry_blocked=False, + ) + result = asyncio.run(dispatch.dispatch(args)) + self.assertEqual(result, 0) + state_path = workspace / ".git" / "agent-task-dispatcher" / "state.json" + self.assertFalse(state_path.exists()) + + def test_explicit_unobserved_task_group_cannot_report_success(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + (workspace / "agent-task").mkdir() + args = SimpleNamespace( + workspace=str(workspace), + task_group="missing-group", + dry_run=False, + retry_blocked=False, + ) + + result = asyncio.run(dispatch.dispatch(args)) + + self.assertEqual(result, 2) + state = json.loads( + ( + workspace + / ".git" + / "agent-task-dispatcher" + / "state.json" + ).read_text(encoding="utf-8") + ) + self.assertEqual( + state["orchestrations"]["missing-group"]["status"], + "blocked", + ) + + +class RouteDecisionPersistenceTest(unittest.TestCase): + def make_task( + self, + workspace: Path, + name: str = "route/01_unit", + *, + lane: str = "local", + grade: int = 5, + ): + directory = workspace / "agent-task" / name + directory.mkdir(parents=True) + header = f"\n" + (directory / f"PLAN-{lane}-G{grade:02d}.md").write_text( + header + + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + + "| src/route.py | ROUTE-1 |\n", + encoding="utf-8", + ) + (directory / f"CODE_REVIEW-{lane}-G{grade:02d}.md").write_text( + header, + encoding="utf-8", + ) + return next(task for task in dispatch.scan_tasks(workspace, None) if task.name == name) + + def test_reopen_body_edit_and_generation_reset_preserve_or_reset_pin(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + first = dispatch.StateStore(workspace) + try: + initial, worker_spec = dispatch.persisted_execution_decision( + first, task, stage="worker" + ) + review, review_spec = dispatch.persisted_execution_decision( + first, task, stage="review" + ) + self.assertEqual(initial["transition"]["trigger"], "initial") + self.assertEqual(worker_spec.cli, "pi") + self.assertEqual(review_spec.cli, "codex") + self.assertEqual(review["transition"]["trigger"], "initial") + self.assertEqual( + [entry["stage"] for entry in first.task_state(task)["route_transition_history"]], + ["worker", "review"], + ) + finally: + first.close() + + reopened = dispatch.StateStore(workspace) + try: + reopened_task = dispatch.scan_tasks(workspace, None)[0] + resumed, resumed_spec = dispatch.persisted_execution_decision( + reopened, reopened_task, stage="worker" + ) + self.assertEqual(resumed["transition"]["trigger"], "resume") + self.assertEqual(resumed_spec.display, "pi/iop/ornith:35b") + + assert reopened_task.plan is not None + reopened_task.plan.write_text( + reopened_task.plan.read_text(encoding="utf-8") + "\n๋ณธ๋ฌธ๋งŒ ๋ณ€๊ฒฝ\n", + encoding="utf-8", + ) + body_edited = dispatch.scan_tasks(workspace, None)[0] + self.assertEqual(body_edited.plan_hash, reopened_task.plan_hash) + body_resume, _ = dispatch.persisted_execution_decision( + reopened, body_edited, stage="worker" + ) + self.assertEqual(body_resume["transition"]["trigger"], "resume") + + assert body_edited.plan is not None and body_edited.review is not None + for path in (body_edited.plan, body_edited.review): + path.write_text( + path.read_text(encoding="utf-8").replace("plan=0", "plan=1"), + encoding="utf-8", + ) + next_generation = dispatch.scan_tasks(workspace, None)[0] + reset, _ = dispatch.persisted_execution_decision( + reopened, next_generation, stage="worker" + ) + self.assertEqual(reset["transition"]["trigger"], "initial") + reset_history = reopened.task_state(next_generation)[ + "route_transition_history" + ] + self.assertEqual(len(reset_history), 1) + self.assertEqual(reset_history[0]["stage"], "worker") + self.assertEqual(reset_history[0]["transition"], "initial") + self.assertEqual( + reset_history[0]["work_unit_id"], reset["work_unit_id"] + ) + self.assertEqual(reset_history[0]["selected"], reset["selected"]) + self.assertEqual(reset_history[0]["decision"], reset["decision"]) + self.assertEqual(reset_history[0]["quota"], reset["quota"]) + self.assertNotIn("rule_id", reset_history[0]) + self.assertNotIn("priority", reset_history[0]) + self.assertNotIn("quota_snapshot", reset_history[0]) + finally: + reopened.close() + + def test_resume_keeps_pin_across_kst_boundary(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + task = self.make_task(workspace) + initial = dispatch.select_execution_decision( + task, stage="worker", + evaluated_at=datetime(2026, 7, 25, 6, 59, tzinfo=dispatch.KST), + ) + resumed = dispatch.select_execution_decision( + task, stage="worker", prior_decision=initial, + evaluated_at=datetime(2026, 7, 25, 7, 0, tzinfo=dispatch.KST), + ) + self.assertEqual(resumed["transition"]["trigger"], "resume") + self.assertEqual(resumed["selected"], initial["selected"]) + self.assertTrue(resumed["decision"]["pinned"]) + + def test_rejects_tampered_canonical_target_and_selector_load_error(self): + with tempfile.TemporaryDirectory() as temporary: + task = self.make_task(Path(temporary)) + decision = dispatch.select_execution_decision(task, stage="worker") + tampered = json.loads(json.dumps(decision)) + tampered["selected"] = { + "adapter": "agy", + "target": "untrusted-target", + "execution_class": "local_model", + "selfcheck_required": False, + } + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch.agent_spec_from_decision(tampered) + for failure in (OSError("load failed"), SyntaxError("broken selector"), RuntimeError("loader crashed")): + with self.subTest(failure=type(failure).__name__): + with mock.patch.object(dispatch, "_selector_module", side_effect=failure): + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch.select_execution_decision(task, stage="worker") + + def test_malformed_or_exhausted_state_blocks_only_its_task(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + exhausted = self.make_task( + workspace, "route/01_exhausted", lane="cloud", grade=7 + ) + healthy = self.make_task(workspace, "route/02_healthy") + store = dispatch.StateStore(workspace) + try: + store.update_task( + exhausted, + quota_snapshot={ + "source": "test", + "targets": [{ + "adapter": "claude", + "target": "claude-opus-4-8", + "status": "exhausted", + }], + }, + ) + asyncio.run(dispatch.run_worker(workspace, store, exhausted)) + self.assertIn("no_eligible_target", store.task_state(exhausted)["blocked"]) + _, healthy_spec = dispatch.persisted_execution_decision( + store, healthy, stage="worker" + ) + self.assertEqual(healthy_spec.cli, "pi") + + store.update_task( + healthy, execution_decisions={"worker": {"malformed": True}} + ) + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch.persisted_execution_decision(store, healthy, stage="worker") + finally: + store.close() + + +class DispatcherConvergenceSimulationTest(unittest.IsolatedAsyncioTestCase): + def write_task( + self, + workspace: Path, + task_name: str, + source_path: str, + ) -> None: + directory = workspace / "agent-task" / task_name + directory.mkdir(parents=True) + header = f"\n" + (directory / "PLAN-local-G05.md").write_text( + header + + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + + f"| `{source_path}` | SIM-1 |\n", + encoding="utf-8", + ) + (directory / "CODE_REVIEW-local-G05.md").write_text( + header, + encoding="utf-8", + ) + + async def test_parallel_multi_task_followup_dependency_and_terminal_completion(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + (workspace / "agent-task").mkdir() + self.write_task(workspace, "sim/01_alpha", "src/alpha.go") + self.write_task(workspace, "sim/02_beta", "src/beta.go") + self.write_task(workspace, "sim/03+01,02_join", "src/join.go") + self.write_task(workspace, "sim/04_conflict", "./src/alpha.go") + work_log = workspace / "agent-task" / "sim" / dispatch.WORK_LOG_NAME + work_log.write_text("final timeline\n", encoding="utf-8") + + active = {"worker": set(), "selfcheck": set(), "review": set()} + active_tasks: set[str] = set() + maximum = {"worker": 0, "selfcheck": 0, "review": 0} + overlap_violations: list[tuple[str, set[str]]] = [] + review_attempts: dict[str, int] = {} + + def enter(stage: str, task_name: str) -> None: + active[stage].add(task_name) + active_tasks.add(task_name) + maximum[stage] = max(maximum[stage], len(active[stage])) + if {"sim/01_alpha", "sim/04_conflict"} <= active_tasks: + overlap_violations.append((stage, set(active_tasks))) + + def leave(stage: str, task_name: str) -> None: + active[stage].remove(task_name) + active_tasks.remove(task_name) + + async def fake_worker(workspace_path, store, task, *args, **kwargs): + enter("worker", task.name) + try: + await asyncio.sleep(0.005) + completing_decision = { + "work_unit_id": dispatch.work_unit_id_from_file(task.plan), + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/ornith:35b", + "execution_class": "local_model", + "selfcheck_required": True, + }, + } + store.update_task( + task, + worker_done=True, + worker_cli="pi", + worker_model="ornith:35b", + completing_decision=completing_decision, + execution_class="local_model", + selfcheck_done=False, + blocked=None, + ) + finally: + leave("worker", task.name) + + async def fake_selfcheck(workspace_path, store, task, *args, **kwargs): + enter("selfcheck", task.name) + try: + await asyncio.sleep(0.005) + store.update_task(task, selfcheck_done=True, blocked=None) + finally: + leave("selfcheck", task.name) + + alpha_in_review = asyncio.Event() + beta_review_finished = asyncio.Event() + completion_scan_observed = asyncio.Event() + original_scan_tasks = dispatch.scan_tasks + + def observed_scan_tasks(*args, **kwargs): + scanned = original_scan_tasks(*args, **kwargs) + if ( + beta_review_finished.is_set() + and "sim/01_alpha" in set(kwargs.get("exclude_names") or ()) + ): + completion_scan_observed.set() + return scanned + + async def fake_review(workspace_path, store, task, *args, **kwargs): + enter("review", task.name) + try: + attempt = review_attempts.get(task.name, 0) + 1 + review_attempts[task.name] = attempt + if task.name == "sim/01_alpha" and attempt == 1: + alpha_in_review.set() + await beta_review_finished.wait() + # Released by the dispatcher's own completion-triggered + # scan, not by elapsed time. + await completion_scan_observed.wait() + for path in (task.plan, task.review): + assert path is not None + path.write_text( + path.read_text(encoding="utf-8").replace( + "plan=0", "plan=1" + ), + encoding="utf-8", + ) + return None + elif task.name == "sim/02_beta": + await alpha_in_review.wait() + beta_review_finished.set() + else: + await asyncio.sleep(0.005) + archive = ( + workspace_path + / "agent-task" + / "archive" + / "2026" + / "07" + / task.name + ) + archive.parent.mkdir(parents=True, exist_ok=True) + (task.directory / "complete.log").write_text( + "simulation complete\n", encoding="utf-8" + ) + task.directory.rename(archive) + return str(archive) + finally: + leave("review", task.name) + + args = SimpleNamespace( + workspace=str(workspace), + task_group="sim", + dry_run=False, + retry_blocked=False, + ) + with ( + mock.patch.object(dispatch, "run_worker", new=fake_worker), + mock.patch.object(dispatch, "run_selfcheck", new=fake_selfcheck), + mock.patch.object(dispatch, "run_review", new=fake_review), + mock.patch.object(dispatch, "ensure_review_shared_state"), + mock.patch.object( + dispatch, "scan_tasks", wraps=observed_scan_tasks + ) as scan_tasks, + ): + result = await asyncio.wait_for(dispatch.dispatch(args), timeout=2) + + self.assertEqual(result, 0) + self.assertGreaterEqual(maximum["worker"], 2) + self.assertGreaterEqual(maximum["selfcheck"], 2) + self.assertGreaterEqual(maximum["review"], 2) + self.assertEqual(overlap_violations, []) + self.assertGreater(scan_tasks.call_count, 1) + self.assertLessEqual(scan_tasks.call_count, 5) + self.assertTrue( + any( + call.kwargs.get("exclude_names") + for call in scan_tasks.call_args_list + ), + "completion-triggered scans must exclude still-running tasks", + ) + self.assertTrue( + completion_scan_observed.is_set(), + "alpha must be released by an observed completion-triggered scan", + ) + self.assertEqual(review_attempts["sim/01_alpha"], 2) + self.assertEqual(review_attempts["sim/02_beta"], 1) + self.assertEqual(review_attempts["sim/03+01,02_join"], 1) + self.assertEqual(review_attempts["sim/04_conflict"], 1) + archive_root = workspace / "agent-task" / "archive" / "2026" / "07" / "sim" + for subtask in ("01_alpha", "02_beta", "03+01,02_join", "04_conflict"): + self.assertTrue((archive_root / subtask / "complete.log").is_file()) + self.assertFalse(work_log.exists()) + self.assertEqual( + (archive_root / "work_log_0.log").read_text(encoding="utf-8"), + "final timeline\n", + ) + + + +class DynamicFailoverBudgetTest(unittest.TestCase): + def make_task(self, workspace: Path): + directory = workspace / "agent-task" / "budget/01_unit" + directory.mkdir(parents=True) + header = "\n" + (directory / "PLAN-local-G07.md").write_text( + header + + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + "| `src/budget.py` | TEST-1 |\n", + encoding="utf-8", + ) + (directory / "CODE_REVIEW-local-G07.md").write_text(header, encoding="utf-8") + return dispatch.scan_tasks(workspace, None)[0] + + def test_context_package_keeps_artifacts_and_blocks_cross_adapter_native_session(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + attempt = workspace / "attempt" + attempt.mkdir() + raw, normalized, native = attempt / "stream.log", attempt / "normalized-output.log", attempt / "session.jsonl" + raw.write_text("raw\n", encoding="utf-8") + normalized.write_text("normalized\n", encoding="utf-8") + native.write_text("{}\n", encoding="utf-8") + locator = attempt / "locator.json" + record = {"task": task.name, "workspace": str(workspace), "plan_path": str(task.plan), "stream_log": str(raw), "normalized_output_log": str(normalized), "native_session_path": str(native)} + locator.write_text(json.dumps(record), encoding="utf-8") + pi = dispatch.AgentSpec("pi", "ornith:35b", "pi/iop/ornith:35b", local_pi=True) + codex = dispatch.AgentSpec("codex", "gpt-5.6-sol", "codex/gpt-5.6-sol xhigh") + logical = dispatch.build_context_package(workspace, task, locator, previous_spec=pi, next_spec=codex) + self.assertEqual(logical["resume_mode"], "logical") + self.assertNotIn("native_session_path", logical) + native_package = dispatch.build_context_package(workspace, task, locator, previous_spec=pi, next_spec=pi) + self.assertEqual(native_package["native_session_path"], str(native.resolve())) + external = workspace / "external.log" + external.write_text("outside\n", encoding="utf-8") + record["stream_log"] = str(external) + locator.write_text(json.dumps(record), encoding="utf-8") + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch.build_context_package(workspace, task, locator, previous_spec=pi, next_spec=codex) + record["stream_log"] = str(raw) + record["workspace"] = "" + locator.write_text(json.dumps(record), encoding="utf-8") + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch.build_context_package(workspace, task, locator, previous_spec=pi, next_spec=codex) + record["workspace"] = str(workspace) + locator.write_text(json.dumps(record), encoding="utf-8") + normalized.unlink() + with self.assertRaises(dispatch.ExecutionDecisionError): + dispatch.build_context_package(workspace, task, locator, previous_spec=pi, next_spec=codex) + + def test_primary_and_alternate_share_budget_across_reopen(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + locator_gemini = self.make_attempt_locator(workspace, task, gemini_spec) + laguna_spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + locator_laguna = self.make_attempt_locator(workspace, task, laguna_spec) + + initial_store = dispatch.StateStore(workspace) + try: + dispatch.persisted_execution_decision(initial_store, task, stage="worker", evaluated_at=daytime) + finally: + initial_store.close() + + store = dispatch.StateStore(workspace) + try: + invoked_specs = [] + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "agy": + return (1, "provider-quota", locator_gemini) + if len(invoked_specs) == 2: + return (1, "generic-error", locator_laguna) + raise asyncio.CancelledError() + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + try: + asyncio.run(dispatch.run_escalating(workspace, store, task, "worker", gemini_spec)) + except asyncio.CancelledError: + pass + + state1 = store.task_state(task) + decisions1 = state1["execution_decisions"] + history1 = state1["route_transition_history"] + worker_budget1 = dispatch.StageFailureBudget.from_decision(store, task, decisions1["worker"]) + self.assertEqual([s.cli for s in invoked_specs[:2]], ["agy", "pi"]) + self.assertEqual([h["transition"] for h in history1], ["initial", "provider-quota"]) + self.assertEqual(worker_budget1.count(), 2) + finally: + store.close() + + reopened = dispatch.StateStore(workspace) + try: + worker_budget_reopened = dispatch.StageFailureBudget.from_decision(reopened, task, decisions1["worker"]) + current_count = worker_budget_reopened.count() + needed_failures = 10 - current_count + locators = [workspace / f"failure-{i}.json" for i in range(needed_failures)] + + with ( + mock.patch.object( + dispatch, + "invoke", + new=mock.AsyncMock( + side_effect=[(1, "generic-error", loc) for loc in locators] + ), + ) as invoke, + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + success, final_loc = asyncio.run( + dispatch.run_escalating(workspace, reopened, task, "worker", laguna_spec) + ) + self.assertFalse(success) + self.assertEqual(invoke.await_count, 8) + self.assertTrue(all(call.args[4] == laguna_spec for call in invoke.await_args_list)) + self.assertEqual(final_loc, locators[-1]) + self.assertIn("recovery failure limit exhausted", reopened.task_state(task)["blocked"]) + + state2 = reopened.task_state(task) + worker_budget2 = dispatch.StageFailureBudget.from_decision(reopened, task, decisions1["worker"]) + review_decision = dispatch.select_execution_decision(task, stage="review", evaluated_at=daytime) + review_budget2 = dispatch.StageFailureBudget.from_decision(reopened, task, review_decision) + self.assertEqual(worker_budget2.count(), 10) + self.assertEqual(review_budget2.count(), 0) + raw_entry = state2.get("stage_failure_budgets", {}).get(worker_budget2.key, {}) + self.assertEqual(raw_entry.get("last_target"), {"adapter": "pi", "target": "iop/laguna-s:2.1"}) + self.assertEqual(raw_entry.get("last_transition"), "provider-quota") + self.assertEqual(state2["execution_decisions"], decisions1) + self.assertEqual([h["transition"] for h in state2["route_transition_history"]], ["initial", "provider-quota"]) + finally: + reopened.close() + + def test_success_resets_only_current_stage_budget(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + locator = self.make_attempt_locator(workspace, task, gemini_spec) + + worker_decision = dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime)[0] + review_decision = dispatch.select_execution_decision(task, stage="review", evaluated_at=daytime) + + worker_budget = dispatch.StageFailureBudget.from_decision(store, task, worker_decision) + review_budget = dispatch.StageFailureBudget.from_decision(store, task, review_decision) + + worker_budget.record_failure(target={"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, transition="generic-error") + review_budget.record_failure(target={"adapter": "codex", "target": "gpt-5.6-sol"}, transition="generic-error") + + self.assertEqual(worker_budget.count(), 1) + self.assertEqual(review_budget.count(), 1) + + async def mock_invoke(*args, **kwargs): + return (0, None, locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + success, final_loc = asyncio.run( + dispatch.run_escalating(workspace, store, task, "worker", gemini_spec) + ) + + self.assertTrue(success) + self.assertEqual(final_loc, locator) + self.assertEqual(worker_budget.count(), 0) + self.assertEqual(review_budget.count(), 1) + finally: + store.close() + + def make_attempt_locator(self, workspace: Path, task: dispatch.Task, spec: dispatch.AgentSpec) -> Path: + attempt = workspace / f"attempt-{spec.cli}" + attempt.mkdir(parents=True, exist_ok=True) + raw, normalized = attempt / "stream.log", attempt / "normalized-output.log" + raw.write_text("raw log\n", encoding="utf-8") + normalized.write_text("normalized output\n", encoding="utf-8") + locator = attempt / "locator.json" + record = { + "task": task.name, + "workspace": str(workspace), + "plan_path": str(task.plan), + "stream_log": str(raw), + "normalized_output_log": str(normalized), + "cli": spec.cli, + "model": spec.model, + "spec": {"adapter": spec.cli, "target": spec.model}, + } + locator.write_text(json.dumps(record), encoding="utf-8") + return locator + + +class DispatcherCanonicalFailoverIntegrationTest(unittest.IsolatedAsyncioTestCase): + def make_task(self, workspace: Path, lane: str = "local", grade: int = 8) -> dispatch.Task: + directory = workspace / "agent-task" / "failover/01_unit" + directory.mkdir(parents=True, exist_ok=True) + header = "\n" + (directory / f"PLAN-{lane}-G{grade:02d}.md").write_text( + header + + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + "| `src/failover.py` | TEST-1 |\n", + encoding="utf-8", + ) + (directory / f"CODE_REVIEW-{lane}-G{grade:02d}.md").write_text(header, encoding="utf-8") + tasks = dispatch.scan_tasks(workspace, None) + return tasks[0] + + def make_attempt_locator(self, workspace: Path, task: dispatch.Task, spec: dispatch.AgentSpec) -> Path: + attempt = workspace / f"attempt-{spec.cli}" + attempt.mkdir(parents=True, exist_ok=True) + raw, normalized = attempt / "stream.log", attempt / "normalized-output.log" + raw.write_text("raw log\n", encoding="utf-8") + normalized.write_text("normalized output\n", encoding="utf-8") + locator = attempt / "locator.json" + record = { + "task": task.name, + "workspace": str(workspace), + "plan_path": str(task.plan), + "stream_log": str(raw), + "normalized_output_log": str(normalized), + "cli": spec.cli, + "model": spec.model, + "spec": {"adapter": spec.cli, "target": spec.model}, + } + locator.write_text(json.dumps(record), encoding="utf-8") + return locator + + async def test_cloud_g01_g02_quota_failover_runs_spark_gemini_haiku(self): + daytime = datetime( + 2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9)) + ) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="cloud", grade=1) + store = dispatch.StateStore(workspace) + selector = dispatch._selector_module() + + def unknown_quota_probe(*args, **kwargs): + adapter = kwargs["adapter"] + target = kwargs["target"] + return { + "schema_version": "1.0", + "snapshot_id": f"unknown-{adapter}-{target}", + "source": "iop-node quota-probe", + "checked_at": kwargs["checked_at"].isoformat(), + "targets": [ + { + "adapter": adapter, + "target": target, + "status": "unknown", + } + ], + "required_caps": [], + "reason_codes": ["checker_error"], + } + + try: + with mock.patch.object( + selector, + "probe_candidate_quota", + side_effect=unknown_quota_probe, + ): + _, initial_spec = dispatch.persisted_execution_decision( + store, + task, + stage="worker", + evaluated_at=daytime, + ) + specs = { + "codex": initial_spec, + "agy": dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (Low)", + "agy/Gemini 3.6 Flash (Low)", + ), + "claude": dispatch.AgentSpec( + "claude", + "claude-haiku-4-5", + "claude/claude-haiku-4-5 xhigh", + ), + } + locators = { + cli: self.make_attempt_locator(workspace, task, spec) + for cli, spec in specs.items() + } + invoked_specs = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "claude": + return 0, None, locators[spec.cli] + return 1, "provider-quota", locators[spec.cli] + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object( + dispatch.asyncio, + "sleep", + new=mock.AsyncMock(), + ), + ): + success, final_locator = await dispatch.run_escalating( + workspace, + store, + task, + "worker", + initial_spec, + ) + + self.assertTrue(success) + self.assertEqual(final_locator, locators["claude"]) + self.assertEqual( + [(spec.cli, spec.model) for spec in invoked_specs], + [ + ("codex", "gpt-5.3-codex-spark"), + ("agy", "Gemini 3.6 Flash (Low)"), + ("claude", "claude-haiku-4-5"), + ], + ) + decision = store.task_state(task)["execution_decisions"]["worker"] + self.assertEqual( + decision["used_candidates"], + [ + {"adapter": "codex", "target": "gpt-5.3-codex-spark"}, + {"adapter": "agy", "target": "Gemini 3.6 Flash (Low)"}, + {"adapter": "claude", "target": "claude-haiku-4-5"}, + ], + ) + finally: + store.close() + + async def test_invalid_logical_context_does_not_commit_or_promote(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + + cases = [ + ("no_locator", lambda loc, ws, tk: ws / "nonexistent.json"), + ("invalid_json", lambda loc, ws, tk: (loc.write_text("invalid json", encoding="utf-8"), loc)[1]), + ("workspace_mismatch", lambda loc, ws, tk: ( + loc.write_text(json.dumps({ + "task": tk.name, "workspace": str(ws / "other"), "plan_path": str(tk.plan), + "stream_log": str(loc.parent / "stream.log"), + "normalized_output_log": str(loc.parent / "normalized-output.log"), + }), encoding="utf-8"), loc + )[1]), + ("task_mismatch", lambda loc, ws, tk: ( + loc.write_text(json.dumps({ + "task": "other/task", "workspace": str(ws), "plan_path": str(tk.plan), + "stream_log": str(loc.parent / "stream.log"), + "normalized_output_log": str(loc.parent / "normalized-output.log"), + }), encoding="utf-8"), loc + )[1]), + ("plan_mismatch", lambda loc, ws, tk: ( + loc.write_text(json.dumps({ + "task": tk.name, "workspace": str(ws), "plan_path": str(ws / "other.md"), + "stream_log": str(loc.parent / "stream.log"), + "normalized_output_log": str(loc.parent / "normalized-output.log"), + }), encoding="utf-8"), loc + )[1]), + ("missing_raw_artifact", lambda loc, ws, tk: ( + (loc.parent / "stream.log").unlink(), loc + )[1]), + ("missing_normalized_artifact", lambda loc, ws, tk: ( + (loc.parent / "normalized-output.log").unlink(), loc + )[1]), + ] + + for name, modifier in cases: + with self.subTest(variant=name), tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + base_locator = self.make_attempt_locator(workspace, task, gemini_spec) + target_locator = modifier(base_locator, workspace, task) + + invoked_specs = [] + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + return (1, "provider-quota", target_locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + initial_decisions = store.task_state(task)["execution_decisions"]["worker"] + initial_history = list(store.task_state(task)["route_transition_history"]) + + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", gemini_spec) + + self.assertFalse(success) + self.assertEqual(len(invoked_specs), 1) + self.assertEqual(invoked_specs[0].cli, "agy") + + state = store.task_state(task) + self.assertIn("worker selector decision ์‹คํŒจ", state.get("blocked", "")) + self.assertEqual(state["execution_decisions"]["worker"]["selected"], initial_decisions["selected"]) + self.assertEqual(state["route_transition_history"], initial_history) + finally: + store.close() + + async def test_day_gemini_zero_exit_quota_continues_on_laguna_with_logical_context(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + locator = self.make_attempt_locator(workspace, task, gemini_spec) + invoked_specs = [] + invoked_prompts = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + prompt = args[5] + invoked_specs.append(spec) + invoked_prompts.append(prompt) + if spec.cli == "agy": + return (0, "provider-quota", locator) + return (0, None, locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", gemini_spec) + + self.assertTrue(success) + self.assertEqual(len(invoked_specs), 2) + self.assertEqual(invoked_specs[0].cli, "agy") + self.assertEqual(invoked_specs[1].cli, "pi") + self.assertTrue(invoked_specs[1].local_pi) + laguna_prompt = invoked_prompts[1] + self.assertIn(str(task.plan.resolve()), laguna_prompt) + self.assertIn(str(locator.resolve()), laguna_prompt) + self.assertIn(str(workspace.resolve()), laguna_prompt) + self.assertIn(str((locator.parent / "stream.log").resolve()), laguna_prompt) + self.assertIn(str((locator.parent / "normalized-output.log").resolve()), laguna_prompt) + state = store.task_state(task) + decisions = state["execution_decisions"]["worker"] + self.assertEqual(decisions["selected"]["adapter"], "pi") + self.assertEqual(decisions["transition"]["trigger"], "provider-quota") + finally: + store.close() + + async def test_cloud_g07_provider_quota_promotes_claude_to_codex_without_no_failover_block(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="cloud", grade=7) + store = dispatch.StateStore(workspace) + try: + claude_spec = dispatch.AgentSpec("claude", "claude-opus-4-8", "claude/claude-opus-4-8 xhigh") + terra_spec = dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + claude_locator = self.make_attempt_locator( + workspace, task, claude_spec + ) + terra_locator = self.make_attempt_locator( + workspace, task, terra_spec + ) + invoked_specs = [] + invoked_prompts = [] + transition_budget_counts = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + invoked_prompts.append(args[5]) + if spec.cli == "claude": + return (1, "provider-quota", claude_locator) + state = store.task_state(task) + decision = state["execution_decisions"]["worker"] + budget = dispatch.StageFailureBudget.from_decision( + store, task, decision + ) + transition_budget_counts.append(budget.count()) + return (0, None, terra_locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", claude_spec) + + self.assertTrue(success) + self.assertEqual(final_loc, terra_locator) + self.assertEqual(len(invoked_specs), 2) + self.assertEqual(invoked_specs, [claude_spec, terra_spec]) + self.assertEqual(transition_budget_counts, [1]) + continuation = invoked_prompts[1] + self.assertIn(str(task.plan.resolve()), continuation) + self.assertIn(str(claude_locator.resolve()), continuation) + self.assertIn(str(workspace.resolve()), continuation) + self.assertIn( + str((claude_locator.parent / "stream.log").resolve()), + continuation, + ) + self.assertIn( + str( + (claude_locator.parent / "normalized-output.log").resolve() + ), + continuation, + ) + state = store.task_state(task) + decision = state["execution_decisions"]["worker"] + self.assertEqual(decision["selected"]["adapter"], "codex") + self.assertEqual(decision["selected"]["target"], "gpt-5.6-terra") + self.assertEqual(decision["transition"]["kind"], "promotion") + self.assertEqual( + decision["transition"]["trigger"], "provider-quota" + ) + self.assertEqual( + [entry["transition"] for entry in state["route_transition_history"]], + ["initial", "provider-quota"], + ) + budget = dispatch.StageFailureBudget.from_decision( + store, task, decision + ) + self.assertEqual(budget.count(), 0) + self.assertNotIn("no_failover_candidate", state.get("blocked") or "") + finally: + store.close() + + async def test_cloud_agy_promotion_chain_commits_each_transition(self): + daytime = datetime( + 2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9)) + ) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="cloud", grade=5) + store = dispatch.StateStore(workspace) + try: + agy_spec = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (High)", + "agy/Gemini 3.6 Flash (High)", + ) + claude_spec = dispatch.AgentSpec( + "claude", + "claude-opus-4-8", + "claude/claude-opus-4-8 xhigh", + ) + terra_spec = dispatch.AgentSpec( + "codex", + "gpt-5.6-terra", + "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + locators = { + spec.cli: self.make_attempt_locator(workspace, task, spec) + for spec in (agy_spec, claude_spec, terra_spec) + } + invoked_specs = [] + invoked_prompts = [] + transition_budget_counts = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + invoked_prompts.append(args[5]) + if spec.cli == "agy": + return (1, "provider-quota", locators["agy"]) + state = store.task_state(task) + decision = state["execution_decisions"]["worker"] + budget = dispatch.StageFailureBudget.from_decision( + store, task, decision + ) + transition_budget_counts.append(budget.count()) + if spec.cli == "claude": + return (1, "context-limit", locators["claude"]) + return (0, None, locators["codex"]) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object( + dispatch.asyncio, "sleep", new=mock.AsyncMock() + ), + ): + dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=daytime + ) + success, final_locator = await dispatch.run_escalating( + workspace, store, task, "worker", agy_spec + ) + + self.assertTrue(success) + self.assertEqual(final_locator, locators["codex"]) + self.assertEqual( + invoked_specs, [agy_spec, claude_spec, terra_spec] + ) + self.assertEqual(transition_budget_counts, [1, 2]) + self.assertIn( + str(locators["agy"].resolve()), invoked_prompts[1] + ) + self.assertIn( + str(locators["claude"].resolve()), invoked_prompts[2] + ) + state = store.task_state(task) + decision = state["execution_decisions"]["worker"] + self.assertEqual( + decision["promotion_path"], + [ + { + "adapter": "agy", + "target": "Gemini 3.6 Flash (High)", + }, + {"adapter": "claude", "target": "claude-opus-4-8"}, + {"adapter": "codex", "target": "gpt-5.6-terra"}, + ], + ) + self.assertEqual( + [entry["transition"] for entry in state["route_transition_history"]], + ["initial", "provider-quota", "context-limit"], + ) + self.assertEqual( + dispatch.StageFailureBudget.from_decision( + store, task, decision + ).count(), + 0, + ) + finally: + store.close() + + async def test_night_laguna_failure_continues_on_available_gemini(self): + nighttime = datetime(2026, 7, 26, 1, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + laguna_spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + locator = self.make_attempt_locator(workspace, task, laguna_spec) + invoked_specs = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "pi": + return (1, "provider-stream-disconnect", locator) + return (0, None, locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=nighttime) + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", laguna_spec) + + self.assertTrue(success) + self.assertEqual(len(invoked_specs), 2) + self.assertEqual(invoked_specs[0].cli, "pi") + self.assertEqual(invoked_specs[1].cli, "agy") + state = store.task_state(task) + decisions = state["execution_decisions"]["worker"] + self.assertEqual(decisions["selected"]["adapter"], "agy") + self.assertEqual(decisions["transition"]["trigger"], "provider-stream-disconnect") + finally: + store.close() + + async def test_night_gemini_quota_exhaustion_blocks_without_bounce(self): + nighttime = datetime(2026, 7, 26, 1, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + laguna_spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + locator = self.make_attempt_locator(workspace, task, laguna_spec) + invoked_specs = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + return (1, "provider-quota", locator) + + quota_snap = { + "targets": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)", "status": "exhausted"} + ] + } + store.update_task(task, quota_snapshot=quota_snap) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=nighttime) + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", laguna_spec) + + self.assertFalse(success) + state = store.task_state(task) + self.assertIn("no_failover_candidate", state.get("blocked", "")) + finally: + store.close() + + async def test_recovered_primary_quota_does_not_reverse_failover(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + store.update_task( + task, + quota_snapshot={ + "snapshot_id": "gemini-exhausted", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)", "status": "exhausted"} + ], + }, + ) + + laguna_spec = dispatch.AgentSpec("pi", "iop/laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + decision, spec = dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + self.assertEqual(spec.cli, "pi") + + locator = self.make_attempt_locator(workspace, task, laguna_spec) + + store.update_task( + task, + quota_snapshot={ + "snapshot_id": "gemini-recovered", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T04:00:00+09:00", + "targets": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)", "status": "available"} + ], + }, + ) + + invoked_specs = [] + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + return (1, "provider-quota", locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", laguna_spec) + + self.assertFalse(success) + self.assertEqual(len(invoked_specs), 1) + self.assertEqual(invoked_specs[0].cli, "pi") + + state = store.task_state(task) + self.assertIn("no_failover_candidate", state.get("blocked", "")) + finally: + store.close() + + async def test_generic_failure_stays_on_same_target(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + locator = self.make_attempt_locator(workspace, task, gemini_spec) + invoked_specs = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if len(invoked_specs) == 1: + return (1, "generic-error", locator) + return (0, None, locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + success, final_loc = await dispatch.run_escalating(workspace, store, task, "worker", gemini_spec) + + self.assertTrue(success) + self.assertEqual(len(invoked_specs), 2) + self.assertEqual(invoked_specs[0].cli, "agy") + self.assertEqual(invoked_specs[1].cli, "agy") + finally: + store.close() + + async def test_no_promotion_target_keeps_same_target_and_persists_state(self): + """local-G08 daytime: provider-connection x2 โ†’ success. Same AGY target, delay [2, 4], budget reset. + + The selector-backed worker must NOT fall through to legacy promoted_spec(). + Invocation target, persisted selected, history, and terminal recovery backoff + must all agree on AGY with bounded exponential backoff. + """ + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace) + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec( + "agy", + "Gemini 3.6 Flash (Medium)", + "agy/Gemini 3.6 Flash (Medium)", + ) + locator = self.make_attempt_locator(workspace, task, gemini_spec) + invoked_specs = [] + sleep_delays = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if len(invoked_specs) <= 2: + return (1, "provider-connection", locator) + return (0, None, locator) + + async def observe_sleep(delay): + sleep_delays.append(delay) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object( + dispatch.asyncio, + "sleep", + new=mock.AsyncMock(side_effect=observe_sleep), + ), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + initial_state = store.task_state(task) + initial_selected = dict(initial_state["execution_decisions"]["worker"]["selected"]) + initial_history = list(initial_state["route_transition_history"]) + + success, final_loc = await dispatch.run_escalating( + workspace, store, task, "worker", gemini_spec + ) + + self.assertTrue(success) + # Two failures then one success = 3 invocations + self.assertEqual(len(invoked_specs), 3) + # All invocations must be on AGY โ€” no legacy AGYโ†’Claude fallthrough + for i, spec in enumerate(invoked_specs): + self.assertEqual(spec.cli, "agy", f"invocation {i} target mismatch") + self.assertEqual(spec.model, "Gemini 3.6 Flash (Medium)", f"invocation {i} model mismatch") + + # Terminal backoff: retries go 0โ†’1โ†’2, delays = [2**1, 2**2] = [2, 4] + self.assertEqual(len(sleep_delays), 2, f"expected 2 sleep calls, got {len(sleep_delays)}") + self.assertEqual(sleep_delays[0], 2) + self.assertEqual(sleep_delays[1], 4) + + state = store.task_state(task) + # Persisted selected must NOT change from initial AGY + self.assertEqual( + state["execution_decisions"]["worker"]["selected"], + initial_selected, + ) + # History must NOT have a promotion entry + self.assertEqual( + state["route_transition_history"], + initial_history, + ) + # No block should be set after success + self.assertIsNone(state.get("blocked")) + # After success, stage failure budget count must be 0 + worker_decision = state["execution_decisions"]["worker"] + worker_budget = dispatch.StageFailureBudget.from_decision(store, task, worker_decision) + self.assertEqual(worker_budget.count(), 0) + finally: + store.close() + + async def test_promotion_chain_exhaustion_stays_on_last_target(self): + """Cloud promotion chain: AGYโ†’Claudeโ†’Terra exhausted. + + When the last canonical target (Terra) fails with a promotable failure + and no promotion target remains, the worker must stay on Terra for + same-target recovery rather than falling through to legacy promoted_spec(). + """ + daytime = datetime( + 2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9)) + ) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="cloud", grade=5) + store = dispatch.StateStore(workspace) + try: + agy_spec = dispatch.AgentSpec( + "agy", "Gemini 3.6 Flash (High)", "agy/Gemini 3.6 Flash (High)" + ) + claude_spec = dispatch.AgentSpec( + "claude", "claude-opus-4-8", "claude/claude-opus-4-8 xhigh" + ) + terra_spec = dispatch.AgentSpec( + "codex", "gpt-5.6-terra", "codex/gpt-5.6-terra high", + reasoning_effort="high", + ) + loc_agy = self.make_attempt_locator(workspace, task, agy_spec) + loc_claude = self.make_attempt_locator(workspace, task, claude_spec) + loc_terra = self.make_attempt_locator(workspace, task, terra_spec) + invoked_specs = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "agy": + return (1, "provider-quota", loc_agy) + if spec.cli == "claude": + return (1, "context-limit", loc_claude) + # Terra fails once, then succeeds โ€” chain exhaustion keeps it on Terra + terra_count = sum(1 for s in invoked_specs if s.cli == "codex") + if terra_count == 1: + return (1, "provider-quota", loc_terra) + return (0, None, loc_terra) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=daytime + ) + success, final_loc = await dispatch.run_escalating( + workspace, store, task, "worker", agy_spec + ) + + # Chain: AGY โ†’ Claude โ†’ Terra, then Terra retries on same target (no legacy fallthrough) + self.assertEqual( + [s.cli for s in invoked_specs], + ["agy", "claude", "codex", "codex"], + ) + # The third and fourth invocations are both Terra (same-target recovery) + self.assertEqual(invoked_specs[2].cli, "codex") + self.assertEqual(invoked_specs[2].model, "gpt-5.6-terra") + self.assertEqual(invoked_specs[3].cli, "codex") + self.assertEqual(invoked_specs[3].model, "gpt-5.6-terra") + + state = store.task_state(task) + decision = state["execution_decisions"]["worker"] + # Promotion path should include all three transitions + self.assertEqual( + len(decision["promotion_path"]), 3, + ) + # History should show the promotions but no legacy recovery + transitions = [h["transition"] for h in state["route_transition_history"]] + self.assertIn("provider-quota", transitions) + self.assertIn("context-limit", transitions) + # No legacy promoted_spec() fallthrough: selected stays on Terra + self.assertEqual( + decision["selected"]["adapter"], "codex" + ) + self.assertEqual( + decision["selected"]["target"], "gpt-5.6-terra" + ) + finally: + store.close() + + async def test_legacy_promoted_spec_still_works_for_non_selector_worker(self): + """Ensure legacy promoted_spec() path is preserved for non-selector workers.""" + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + # Use a task that does NOT have a persisted selector decision + # so the selector promotion block is skipped entirely. + directory = workspace / "agent-task" / "legacy_recovery_test" + directory.mkdir(parents=True, exist_ok=True) + header = "\n" + (directory / "PLAN-local-G08.md").write_text(header, encoding="utf-8") + (directory / "CODE_REVIEW-local-G08.md").write_text(header, encoding="utf-8") + tasks = dispatch.scan_tasks(workspace, None) + task = tasks[0] + store = dispatch.StateStore(workspace) + try: + agy_spec = dispatch.AgentSpec( + "agy", "Gemini 3.6 Flash (High)", "agy/Gemini 3.6 Flash (High)" + ) + claude_spec = dispatch.AgentSpec( + "claude", "claude-opus-4-8", "claude/claude-opus-4-8 xhigh" + ) + locator = self.make_attempt_locator(workspace, task, agy_spec) + invoked_specs = [] + + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "agy": + return (1, "provider-quota", locator) + return (0, None, locator) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + # Do NOT persist a selector decision โ€” legacy path only + success, final_loc = await dispatch.run_escalating( + workspace, store, task, "worker", agy_spec + ) + + self.assertTrue(success) + # Legacy path: AGY โ†’ Claude (via promoted_spec) + self.assertEqual(len(invoked_specs), 2) + self.assertEqual(invoked_specs[0].cli, "agy") + self.assertEqual(invoked_specs[1].cli, "claude") + finally: + store.close() + + +class SelectorDispatcherIntegrationTest(unittest.IsolatedAsyncioTestCase): + async def asyncSetUp(self): + await super().asyncSetUp() + invoke_patcher = mock.patch.object( + dispatch, + "invoke", + side_effect=AssertionError("Real provider invocation forbidden in test simulation"), + ) + build_cmd_patcher = mock.patch.object( + dispatch, + "build_command", + side_effect=AssertionError("Real provider command construction forbidden in test simulation"), + ) + self.invoke_deny_guard = invoke_patcher.start() + self.build_cmd_deny_guard = build_cmd_patcher.start() + self.addCleanup(invoke_patcher.stop) + self.addCleanup(build_cmd_patcher.stop) + + def make_task( + self, workspace: Path, lane: str = "local", grade: int = 8, unit: str = "01_unit" + ) -> dispatch.Task: + directory = workspace / "agent-task" / "selector_dispatch_integration" / unit + directory.mkdir(parents=True, exist_ok=True) + header = f"\n" + (directory / f"PLAN-{lane}-G{grade:02d}.md").write_text( + header + + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + f"| `src/{unit}.py` | TEST-1 |\n", + encoding="utf-8", + ) + (directory / f"CODE_REVIEW-{lane}-G{grade:02d}.md").write_text(header, encoding="utf-8") + tasks = dispatch.scan_tasks(workspace, None) + for task in tasks: + if task.name.endswith(unit): + return task + return tasks[0] + + def make_attempt_locator( + self, workspace: Path, task: dispatch.Task, spec: dispatch.AgentSpec + ) -> Path: + attempt = workspace / f"attempt-{spec.cli}" + attempt.mkdir(parents=True, exist_ok=True) + raw, normalized = attempt / "stream.log", attempt / "normalized-output.log" + raw.write_text("raw log\n", encoding="utf-8") + normalized.write_text("normalized output\n", encoding="utf-8") + locator = attempt / "locator.json" + record = { + "task": task.name, + "workspace": str(workspace), + "plan_path": str(task.plan), + "stream_log": str(raw), + "normalized_output_log": str(normalized), + "cli": spec.cli, + "model": spec.model, + "spec": {"adapter": spec.cli, "target": spec.model}, + } + locator.write_text(json.dumps(record), encoding="utf-8") + return locator + + async def test_worker_and_review_initial_invocation_uses_selector(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + with mock.patch.object(dispatch, "run_escalating") as run_escalating_mock, \ + mock.patch.object(dispatch, "datetime") as datetime_mock: + datetime_mock.now.return_value = daytime + run_escalating_mock.side_effect = lambda ws, st, t, stage, spec, **kwargs: ( + True, self.make_attempt_locator(ws, t, spec) + ) + + await dispatch.run_worker(workspace, store, task) + self.assertEqual(run_escalating_mock.call_count, 1) + call_args = run_escalating_mock.call_args[0] + self.assertEqual(call_args[3], "worker") + spec_worker = call_args[4] + self.assertEqual(spec_worker.cli, "agy") + self.assertEqual(spec_worker.model, "Gemini 3.6 Flash (Medium)") + + state_after_worker = store.task_state(task) + self.assertTrue(state_after_worker.get("worker_done")) + self.assertIn("worker", state_after_worker.get("execution_decisions", {})) + self.assertEqual( + state_after_worker["execution_decisions"]["worker"]["selected"]["target"], + "Gemini 3.6 Flash (Medium)", + ) + + await dispatch.run_review(workspace, store, task) + self.assertEqual(run_escalating_mock.call_count, 2) + call_args2 = run_escalating_mock.call_args[0] + self.assertEqual(call_args2[3], "review") + spec_review = call_args2[4] + self.assertEqual(spec_review.cli, "codex") + self.assertEqual(spec_review.model, "gpt-5.6-sol") + self.assertEqual(spec_review.display, "codex/gpt-5.6-sol xhigh") + + state_after_review = store.task_state(task) + self.assertIn("review", state_after_review.get("execution_decisions", {})) + self.assertNotEqual( + state_after_review["execution_decisions"]["worker"]["selected"], + state_after_review["execution_decisions"]["review"]["selected"], + ) + finally: + store.close() + + async def test_dry_run_statelessness_initial_and_resume_previews(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + # 1. Non-persisted dry-run (initial preview) + args = dispatch.argparse.Namespace( + workspace=str(workspace), + task_group=None, + retry_blocked=False, + dry_run=True, + ) + with mock.patch.object(dispatch, "datetime") as datetime_mock: + datetime_mock.now.return_value = daytime + result = await dispatch.dispatch_with_store(args, workspace, store) + + self.assertEqual(result, 0) + state_initial = store.task_state(task) + self.assertEqual(state_initial.get("execution_decisions"), {}) + self.assertEqual(state_initial.get("route_transition_history"), []) + + # 2. Persist decision and test dry-run (read-only resume preview) + dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=daytime + ) + state_persisted = store.task_state(task) + history_before = list(state_persisted.get("route_transition_history", [])) + self.assertEqual(len(history_before), 1) + + with mock.patch.object(dispatch, "datetime") as datetime_mock: + datetime_mock.now.return_value = daytime + result2 = await dispatch.dispatch_with_store(args, workspace, store) + + self.assertEqual(result2, 0) + state_after_dry_run = store.task_state(task) + history_after = list(state_after_dry_run.get("route_transition_history", [])) + self.assertEqual(history_before, history_after) + finally: + store.close() + + async def test_dry_run_multiple_ready_tasks_isolation_and_statelessness(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + nighttime = datetime(2026, 7, 26, 23, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task1 = self.make_task(workspace, lane="local", grade=8, unit="01_unit1") + task2 = self.make_task(workspace, lane="local", grade=8, unit="02_unit2") + store = dispatch.StateStore(workspace) + try: + # Task 1 is pinned during daytime + dec1, spec1 = dispatch.persisted_execution_decision( + store, task1, stage="worker", evaluated_at=daytime + ) + self.assertEqual(spec1.cli, "agy") + + state1_before = dict(store.task_state(task1)) + state2_before = dict(store.task_state(task2)) + + banners = [] + def capture_banner(event, name, lines): + banners.append((event, name, lines)) + + args = dispatch.argparse.Namespace( + workspace=str(workspace), + task_group=None, + retry_blocked=False, + dry_run=True, + ) + with mock.patch.object(dispatch, "datetime") as datetime_mock, \ + mock.patch.object(dispatch, "banner", side_effect=capture_banner): + datetime_mock.now.return_value = nighttime + result = await dispatch.dispatch_with_store(args, workspace, store) + + self.assertEqual(result, 0) + + task1_banners = [b for b in banners if b[1] == task1.name] + task2_banners = [b for b in banners if b[1] == task2.name] + self.assertTrue(any("model=agy/" in line for b in task1_banners for line in b[2])) + self.assertTrue(any("model=pi/" in line for b in task2_banners for line in b[2])) + + state1_after = store.task_state(task1) + state2_after = store.task_state(task2) + + self.assertEqual( + state1_before.get("route_transition_history"), + state1_after.get("route_transition_history"), + ) + self.assertEqual( + state2_before.get("route_transition_history"), + state2_after.get("route_transition_history"), + ) + self.assertEqual(state2_after.get("execution_decisions"), {}) + finally: + store.close() + + async def test_resume_pins_target_across_time_and_body_changes_and_resets_on_new_generation(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + nighttime = datetime(2026, 7, 26, 23, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + # 1. Initial decision daytime (KST 14:00) -> agy Gemini Medium + dec1, spec1 = dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=daytime + ) + self.assertEqual(spec1.cli, "agy") + self.assertEqual(spec1.model, "Gemini 3.6 Flash (Medium)") + + # 2. Resuming at nighttime (KST 23:00) keeps pinned Gemini Medium + dec2, spec2 = dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=nighttime + ) + self.assertEqual(spec2.cli, "agy") + self.assertEqual(spec2.model, "Gemini 3.6 Flash (Medium)") + + # 3. Body edit (header intact) keeps pinned Gemini Medium + plan_file = task.plan + header = f"\n" + plan_file.write_text(header + "\n# Modified Body Content\n", encoding="utf-8") + dec3, spec3 = dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=nighttime + ) + self.assertEqual(spec3.cli, "agy") + + # 4. New generation header (plan=1) re-evaluates initial decision at nighttime -> pi Laguna + plan_file.write_text("\n\n# New Plan\n", encoding="utf-8") + task_new = dispatch.scan_tasks(workspace, None)[0] + dec4, spec4 = dispatch.persisted_execution_decision( + store, task_new, stage="worker", evaluated_at=nighttime + ) + self.assertEqual(spec4.cli, "pi") + self.assertEqual(spec4.model, "laguna-s:2.1") + finally: + store.close() + + async def test_qualified_failover_and_blocker_scenarios(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + # 1. Initial decision local G08 -> agy Gemini Medium (primary) & pi Laguna (fallback) + dec1, spec1 = dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=daytime + ) + self.assertEqual(spec1.cli, "agy") + + # 2. Qualified failover (provider-quota) -> transitions to pi Laguna + dec2 = dispatch.select_execution_decision( + task, stage="worker", prior_decision=dec1, + evaluated_at=daytime, transition="failover", failure_class="provider-quota" + ) + self.assertEqual(dec2["transition"]["trigger"], "provider-quota") + self.assertEqual(dec2["selected"]["adapter"], "pi") + + # 3. Subsequent failover when no candidate remains -> raises no_failover_candidate + with self.assertRaises(dispatch.ExecutionDecisionError) as ctx: + dispatch.select_execution_decision( + task, stage="worker", prior_decision=dec2, + evaluated_at=daytime, transition="failover", failure_class="provider-quota" + ) + self.assertIn("no_failover_candidate", str(ctx.exception)) + finally: + store.close() + + async def test_context_budget_and_retry_blocked_lifecycle(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + init_snap = { + "schema_version": "1.0", + "snapshot_id": "snap-init", + "source": "fake_probe", + "checked_at": daytime.isoformat(), + "targets": [ + { + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + "status": "available", + "reason_codes": [], + } + ], + "required_caps": [], + "reason_codes": [], + } + # 1. Primary initial execution (agy/Gemini Medium) + dec1, spec1 = dispatch.persisted_execution_decision( + store, task, stage="worker", evaluated_at=daytime, quota_snapshot=init_snap + ) + self.assertEqual(spec1.cli, "agy") + + # 2. Record primary failure (count=1) -> failover to alternate (pi/laguna) + budget = dispatch.StageFailureBudget.from_decision(store, task, dec1) + count1 = budget.record_failure(target=dec1["selected"], transition="provider-quota") + self.assertEqual(count1, 1) + + dec2 = dispatch.select_execution_decision( + task, stage="worker", prior_decision=dec1, + evaluated_at=daytime, transition="failover", failure_class="provider-quota" + ) + dispatch.commit_execution_decision(store, task, "worker", dec2) + self.assertEqual(dec2["selected"]["adapter"], "pi") + + # 3. Alternate fails 9 times -> budget count reaches 10, task is blocked + budget2 = dispatch.StageFailureBudget.from_decision(store, task, dec2) + for _ in range(9): + c = budget2.record_failure(target=dec2["selected"], transition="generic-failure") + self.assertEqual(c, 10) + + store.update_task( + task, + blocked="worker recovery failure limit exhausted: 10/10 locator=/tmp/loc.json", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": "/tmp/loc.json", + "selected": dec2["selected"], + "work_unit_id": dec2["work_unit_id"], + } + ) + self.assertIsNotNone(store.task_state(task).get("blocked")) + + # 4. Retry blocked clears blocked & budget, preserves decision & transition history + args = dispatch.argparse.Namespace( + workspace=str(workspace), + task_group=None, + retry_blocked=True, + dry_run=False, + ) + selector = dispatch._selector_module() + with mock.patch.object(dispatch, "scan_tasks", return_value=[]), \ + mock.patch.object(dispatch, "datetime") as datetime_mock, \ + mock.patch.object(selector.subprocess, "run", side_effect=AssertionError("unexpected subprocess")) as mock_sub: + datetime_mock.now.return_value = daytime + result = await dispatch.dispatch_with_store(args, workspace, store) + + self.assertEqual(result, 0) + self.invoke_deny_guard.assert_not_called() + self.build_cmd_deny_guard.assert_not_called() + mock_sub.assert_not_called() + + state_after_retry = store.task_state(task) + self.assertIsNone(state_after_retry.get("blocked")) + self.assertEqual(state_after_retry.get("stage_failure_budgets"), {}) + self.assertIn("worker", state_after_retry.get("execution_decisions", {})) + self.assertTrue(len(state_after_retry.get("route_transition_history", [])) >= 2) + + # 5. Success resets stage failure budget + budget3 = dispatch.StageFailureBudget.from_decision(store, task, dec2) + budget3.reset_on_success() + self.assertEqual(store.task_state(task).get("stage_failure_budgets"), {}) + finally: + store.close() + + async def test_review_recovery_and_runtime_audit_evidence(self): + daytime = datetime( + 2026, 7, 26, 14, 0, 0, + tzinfo=timezone(timedelta(hours=9)), + ) + nighttime = datetime( + 2026, 7, 26, 23, 0, 0, + tzinfo=timezone(timedelta(hours=9)), + ) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + historical_header = ( + f"\n" + ) + (task.directory / "plan_local_G07_9.log").write_text( + historical_header, encoding="utf-8" + ) + (task.directory / "code_review_cloud_G07_9.log").write_text( + historical_header + + "\n## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n- ์ข…ํ•ฉ ํŒ์ •: FAIL\n", + encoding="utf-8", + ) + assert task.review is not None + task.review.rename(task.directory / "CODE_REVIEW-cloud-G09.md") + task = next( + item for item in dispatch.scan_tasks(workspace, None) + if item.name == task.name + ) + self.assertTrue(task.recovery) + store = dispatch.StateStore(workspace) + try: + # 1. Active review uses the PLAN generation/route but the fixed + # official-review policy and a complete canonical schema. + dec_rev, spec_rev = dispatch.persisted_execution_decision( + store, task, stage="review", evaluated_at=daytime + ) + self.assertEqual(spec_rev.cli, "codex") + self.assertEqual(spec_rev.model, "gpt-5.6-sol") + self.assertEqual(dec_rev["lane"], "local") + self.assertEqual(dec_rev["grade"], 8) + self.assertEqual( + dec_rev["decision"]["rule_id"], "official-review-codex" + ) + self.assertEqual(dec_rev["decision"]["policy_priority"], 10) + self.assertEqual( + dec_rev["decision"]["reason_codes"], + ["official_review_fixed"], + ) + self.assertEqual(dec_rev["decision"]["timezone"], "Asia/Seoul") + self.assertFalse(dec_rev["decision"]["pinned"]) + self.assertEqual( + dec_rev["quota"], + { + "snapshot_id": None, + "mode": "bounded", + "status": "unknown", + "source": "official_review_fixed_policy", + "checked_at": None, + "targets": [], + }, + ) + self.assertEqual( + dec_rev["transition"], + { + "previous_target": None, + "next_target": None, + "trigger": "initial", + "context_transfer": "none", + }, + ) + self.assertNotIn("rule_id", dec_rev) + self.assertNotIn("priority", dec_rev) + self.assertNotIn("quota_snapshot", dec_rev) + self.assertEqual( + dispatch.agent_spec_from_decision(dec_rev), spec_rev + ) + + # 2. Persisted canonical review decisions are reused after + # policy/identity validation rather than being reselected. + reused, reused_spec = dispatch.persisted_execution_decision( + store, + task, + stage="review", + evaluated_at=nighttime, + ) + self.assertEqual(reused, dec_rev) + self.assertEqual(reused_spec, spec_rev) + + # A qualified cloud failure restarts the same fixed Codex target + # without selector failover, promotion, quota probe, or local CLI. + retry_locator = self.make_attempt_locator( + workspace, task, spec_rev + ) + invoked_specs = [] + + async def fake_review_invoke(*args, **kwargs): + invoked_specs.append(args[4]) + if len(invoked_specs) == 1: + return 1, "provider-quota", retry_locator + return 0, None, retry_locator + + with ( + mock.patch.object( + dispatch, "invoke", new=fake_review_invoke + ), + mock.patch.object( + dispatch, + "select_execution_decision", + side_effect=AssertionError( + "fixed review recovery must not reselect" + ), + ) as selector_mock, + mock.patch.object( + dispatch.asyncio, "sleep", new=mock.AsyncMock() + ), + ): + success, final_locator = await dispatch.run_escalating( + workspace, + store, + task, + "review", + spec_rev, + ) + self.assertTrue(success) + self.assertEqual(final_locator, retry_locator) + self.assertEqual(invoked_specs, [spec_rev, spec_rev]) + selector_mock.assert_not_called() + self.assertEqual( + store.task_state(task)["execution_decisions"]["review"], + dec_rev, + ) + + # 3. Review failure budget is independent from worker budget. + budget_worker = dispatch.StageFailureBudget( + store, task, dec_rev["work_unit_id"], "worker" + ) + budget_worker.record_failure( + target={ + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + }, + transition="initial", + ) + budget_review = dispatch.StageFailureBudget.from_decision( + store, task, dec_rev + ) + self.assertEqual(budget_review.count(), 0) + + # 4. Audit consumers read canonical nested decision/quota and + # only expose legacy flat fields through read-only fallback. + evidence = dispatch.selector_evidence_lines(dec_rev) + self.assertIn("rule_id=official-review-codex", evidence) + self.assertIn("priority=10", evidence) + self.assertIn("transition=initial", evidence) + self.assertIn("quota_status=unknown", evidence) + status = dispatch.status_lines( + task, "review", "ready", decision=dec_rev + ) + self.assertIn("rule_id=official-review-codex", status) + runtime_evidence = dispatch.selector_runtime_evidence(dec_rev) + self.assertIn("decision", runtime_evidence) + self.assertIn("quota", runtime_evidence) + self.assertNotIn("rule_id", runtime_evidence) + self.assertNotIn("priority", runtime_evidence) + self.assertNotIn("quota_snapshot", runtime_evidence) + active_history = store.task_state(task)[ + "route_transition_history" + ][-1] + self.assertIn("decision", active_history) + self.assertIn("quota", active_history) + self.assertNotIn("rule_id", active_history) + self.assertNotIn("priority", active_history) + self.assertNotIn("quota_snapshot", active_history) + + legacy_decision = { + "schema_version": "1.0", + "work_unit_id": dec_rev["work_unit_id"], + "stage": "review", + "rule_id": "official-review-codex", + "priority": 10, + "candidates": [{ + "candidate_rank": 1, + "adapter": "codex", + "target": "gpt-5.6-sol", + "execution_class": "cloud_model", + "eligibility": "eligible", + "reason_codes": ["official_review_fixed_target"], + "selfcheck_required": False, + }], + "selected": { + "adapter": "codex", + "target": "gpt-5.6-sol", + "execution_class": "cloud_model", + "selfcheck_required": False, + "reason_codes": ["official_review_fixed_target"], + }, + "quota_snapshot": { + "id": "fixed", "status": "not_applicable" + }, + "transition": {"trigger": "resume"}, + } + legacy_evidence = dispatch.selector_evidence_lines( + legacy_decision + ) + self.assertIn("rule_id=official-review-codex", legacy_evidence) + self.assertIn("quota_status=not_applicable", legacy_evidence) + + # 5. Legacy finalization recovery restores only the matching + # archived PLAN route/identity, then writes a canonical decision. + assert task.plan is not None and task.review is not None + task.review.write_text( + task.review.read_text(encoding="utf-8") + + "\n## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n- ์ข…ํ•ฉ ํŒ์ •: FAIL\n", + encoding="utf-8", + ) + archived_plan = task.directory / "plan_local_G08_10.log" + archived_review = task.directory / "code_review_cloud_G09_10.log" + task.plan.rename(archived_plan) + task.review.rename(archived_review) + non_verdict_review = ( + task.directory / "code_review_cloud_G09_11.log" + ) + non_verdict_review.write_text( + f"\n", + encoding="utf-8", + ) + malformed_review = ( + task.directory / "code_review_cloud_G09_99_extra.log" + ) + malformed_review.write_text( + f"\n" + "\n## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n- ์ข…ํ•ฉ ํŒ์ •: PASS\n", + encoding="utf-8", + ) + leading_zero_review = ( + task.directory / "code_review_cloud_G09_099.log" + ) + leading_zero_review.write_text( + archived_review.read_text(encoding="utf-8"), + encoding="utf-8", + ) + leading_zero_plan = task.directory / "plan_local_G08_010.log" + leading_zero_plan.write_text( + archived_plan.read_text(encoding="utf-8"), + encoding="utf-8", + ) + non_file_plan = task.directory / "plan_local_G08_12.log" + non_file_plan.mkdir() + historical_review = ( + task.directory / "code_review_cloud_G07_9.log" + ) + os.utime(archived_review, (100, 100)) + os.utime(non_verdict_review, (200, 200)) + os.utime(malformed_review, (300, 300)) + os.utime(historical_review, (400, 400)) + os.utime(leading_zero_review, (500, 500)) + os.utime(leading_zero_plan, (600, 600)) + self.assertIsNone( + dispatch.REVIEW_LOG_RE.fullmatch(leading_zero_review.name) + ) + self.assertIsNone( + dispatch.REVIEW_LOG_RE.fullmatch( + "code_review_cloud_G09_ูก.log" + ) + ) + self.assertIsNone( + dispatch.PLAN_LOG_RE.fullmatch(leading_zero_plan.name) + ) + self.assertIsNone( + dispatch.PLAN_LOG_RE.fullmatch("plan_local_G08_ูก.log") + ) + self.assertIsNotNone( + dispatch.REVIEW_LOG_RE.fullmatch( + "code_review_cloud_G09_0.log" + ) + ) + self.assertIsNotNone( + dispatch.PLAN_LOG_RE.fullmatch("plan_local_G08_10.log") + ) + self.assertEqual( + dispatch.latest_verdict_log(task.directory), + archived_review, + ) + recovery_task = next( + item for item in dispatch.scan_tasks(workspace, None) + if item.name == task.name + ) + self.assertTrue(recovery_task.recovery) + self.assertEqual( + dispatch.official_review_plan_source(recovery_task), + archived_plan, + ) + store.update_task( + recovery_task, + execution_decisions={"review": legacy_decision}, + ) + recovered, recovered_spec = dispatch.persisted_execution_decision( + store, + recovery_task, + stage="review", + evaluated_at=nighttime, + ) + self.assertEqual(recovered_spec, spec_rev) + self.assertEqual(recovered["lane"], "local") + self.assertEqual(recovered["grade"], 8) + self.assertEqual( + recovered["work_unit_id"], dec_rev["work_unit_id"] + ) + self.assertTrue(recovered["decision"]["pinned"]) + self.assertEqual(recovered["transition"]["trigger"], "resume") + self.assertNotIn("rule_id", recovered) + self.assertNotIn("quota_snapshot", recovered) + recovered_history = store.task_state(recovery_task)[ + "route_transition_history" + ][-1] + self.assertIn("decision", recovered_history) + self.assertIn("quota", recovered_history) + self.assertNotIn("rule_id", recovered_history) + self.assertNotIn("quota_snapshot", recovered_history) + + # 6. An identity-matching archive with a non-canonical route + # filename fails closed instead of inventing plan-0/lane/grade. + invalid_plan = task.directory / "plan_legacy_G08_0.log" + archived_plan.rename(invalid_plan) + invalid_recovery_task = next( + item for item in dispatch.scan_tasks(workspace, None) + if item.name == task.name + ) + with self.assertRaises(dispatch.ExecutionDecisionError) as ctx: + dispatch.read_or_preview_stage_decision( + invalid_recovery_task, + {}, + stage="review", + evaluated_at=daytime, + ) + self.assertIn( + "matching archived PLAN identity", + str(ctx.exception), + ) + + self.invoke_deny_guard.assert_not_called() + self.build_cmd_deny_guard.assert_not_called() + finally: + store.close() + + async def test_completing_target_controls_selfcheck_and_reuses_pin(self): + daytime = datetime(2026, 7, 26, 14, 0, 0, tzinfo=timezone(timedelta(hours=9))) + nighttime = datetime(2026, 7, 26, 1, 0, 0, tzinfo=timezone(timedelta(hours=9))) + + # Case 1: Day local G08 completion on Laguna requires selfcheck with pinned Laguna + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + laguna_spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + loc_gemini = self.make_attempt_locator(workspace, task, gemini_spec) + loc_laguna = self.make_attempt_locator(workspace, task, laguna_spec) + + invoked_specs = [] + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "agy": + return (1, "provider-quota", loc_gemini) + return (0, None, loc_laguna) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + await dispatch.run_worker(workspace, store, task) + + self.assertEqual([s.cli for s in invoked_specs], ["agy", "pi"]) + state = store.task_state(task) + self.assertEqual(state["execution_class"], "local_model") + self.assertFalse(state["selfcheck_done"]) + self.assertEqual(dispatch.task_stage(task, state), "selfcheck") + self.assertEqual(state["execution_decisions"]["worker"]["selected"]["adapter"], "pi") + self.assertEqual( + state["completing_decision"]["selected"]["execution_class"], "local_model" + ) + hist1 = list(state["route_transition_history"]) + self.assertEqual([h["transition"] for h in hist1], ["initial", "resume", "provider-quota"]) + + selfcheck_specs = [] + async def mock_invoke_selfcheck(*args, **kwargs): + spec = args[4] + selfcheck_specs.append(spec) + return (0, None, loc_laguna) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke_selfcheck), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + mock.patch.object(dispatch, "implementation_review_errors", return_value=[]), + ): + await dispatch.run_selfcheck(workspace, store, task) + + self.assertEqual([s.cli for s in selfcheck_specs], ["pi"]) + state2 = store.task_state(task) + self.assertTrue(state2["selfcheck_done"]) + self.assertEqual(dispatch.task_stage(task, state2), "review") + hist2 = state2["route_transition_history"] + self.assertEqual([h["transition"] for h in hist2], ["initial", "resume", "provider-quota"]) + finally: + store.close() + + # Case 2: Night local G08 completion on Gemini skips selfcheck + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + gemini_spec = dispatch.AgentSpec("agy", "Gemini 3.6 Flash (Medium)", "agy/Gemini 3.6 Flash (Medium)") + laguna_spec = dispatch.AgentSpec("pi", "laguna-s:2.1", "pi/iop/laguna-s:2.1", local_pi=True) + loc_gemini = self.make_attempt_locator(workspace, task, gemini_spec) + loc_laguna = self.make_attempt_locator(workspace, task, laguna_spec) + + invoked_specs = [] + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + if spec.cli == "pi": + return (1, "provider-stream-disconnect", loc_laguna) + return (0, None, loc_gemini) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=nighttime) + await dispatch.run_worker(workspace, store, task) + + self.assertEqual([s.cli for s in invoked_specs], ["pi", "agy"]) + state = store.task_state(task) + self.assertEqual(state["execution_class"], "cloud_model") + self.assertTrue(state["selfcheck_done"]) + self.assertEqual(dispatch.task_stage(task, state), "review") + self.assertEqual([h["transition"] for h in state["route_transition_history"]], ["initial", "resume", "provider-stream-disconnect"]) + finally: + store.close() + + # Case 3: Cloud G07 completion on Claude skips selfcheck + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = self.make_task(workspace, lane="cloud", grade=7) + store = dispatch.StateStore(workspace) + try: + claude_spec = dispatch.AgentSpec("claude", "claude-opus-4-8", "claude/claude-opus-4-8 xhigh") + loc_claude = self.make_attempt_locator(workspace, task, claude_spec) + + invoked_specs = [] + async def mock_invoke(*args, **kwargs): + spec = args[4] + invoked_specs.append(spec) + return (0, None, loc_claude) + + with ( + mock.patch.object(dispatch, "invoke", new=mock_invoke), + mock.patch.object(dispatch.asyncio, "sleep", new=mock.AsyncMock()), + ): + dispatch.persisted_execution_decision(store, task, stage="worker", evaluated_at=daytime) + await dispatch.run_worker(workspace, store, task) + + self.assertEqual([s.cli for s in invoked_specs], ["claude"]) + state = store.task_state(task) + self.assertEqual(state["execution_class"], "cloud_model") + self.assertTrue(state["selfcheck_done"]) + self.assertEqual(dispatch.task_stage(task, state), "review") + self.assertEqual([h["transition"] for h in state["route_transition_history"]], ["initial", "resume"]) + finally: + store.close() + + +class ThroughputQuotaBatchTest(unittest.TestCase): + def make_task( + self, + workspace: Path, + name: str = "route/01_unit", + *, + lane: str = "cloud", + grade: int = 7, + ): + directory = workspace / "agent-task" / name + directory.mkdir(parents=True) + header = f"\n" + (directory / f"PLAN-{lane}-G{grade:02d}.md").write_text( + header + + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n| ํŒŒ์ผ | ํ•ญ๋ชฉ |\n|---|---|\n" + + f"| `src/{name.replace('/', '_')}.py` | ROUTE-1 |\n", + encoding="utf-8", + ) + (directory / f"CODE_REVIEW-{lane}-G{grade:02d}.md").write_text( + header, + encoding="utf-8", + ) + return next(t for t in dispatch.scan_tasks(workspace, None) if t.name == name) + + def test_same_target_n_tasks_single_probe(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_task1", lane="cloud", grade=7) + t2 = self.make_task(workspace, "route/02_task2", lane="cloud", grade=7) + t3 = self.make_task(workspace, "route/03_task3", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + probe_calls = [] + + def mock_probe(*args, **kwargs): + probe_calls.append(kwargs) + target = kwargs["target"] + adapter = kwargs["adapter"] + checked_at_iso = kwargs["checked_at"].astimezone(dispatch.KST).isoformat() + return { + "schema_version": "1.0", + "snapshot_id": f"child-{adapter}-{target}", + "source": "iop-node quota-probe", + "checked_at": checked_at_iso, + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [{"name": "overall", "status": "available", "remaining_percent": 80.0}], + "reason_codes": ["ok"], + } + + selector = dispatch._selector_module() + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker"), (t2, "worker"), (t3, "worker")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + self.assertIsNotNone(batch_snap) + # Cloud G7 candidate target: claude/claude-opus-4-8 + # Total unique probe keys = 1. Probed EXACTLY 1 time across all 3 tasks! + self.assertEqual(len(probe_calls), 1) + + # Evaluate decisions for all tasks using batch_snap + d1, _ = dispatch.persisted_execution_decision(store, t1, stage="worker", quota_snapshot=batch_snap) + d2, _ = dispatch.persisted_execution_decision(store, t2, stage="worker", quota_snapshot=batch_snap) + d3, _ = dispatch.persisted_execution_decision(store, t3, stage="worker", quota_snapshot=batch_snap) + + # All decisions share the exact same snapshot_id and checked_at + self.assertEqual(d1["quota"]["snapshot_id"], batch_snap["snapshot_id"]) + self.assertEqual(d2["quota"]["snapshot_id"], batch_snap["snapshot_id"]) + self.assertEqual(d3["quota"]["snapshot_id"], batch_snap["snapshot_id"]) + + self.assertEqual(d1["quota"]["checked_at"], batch_snap["checked_at"]) + self.assertEqual(d2["quota"]["checked_at"], batch_snap["checked_at"]) + self.assertEqual(d3["quota"]["checked_at"], batch_snap["checked_at"]) + + # Child evidence preserved in batch_snap targets + for target_entry in batch_snap["targets"]: + self.assertIn("child_snapshot_id", target_entry) + finally: + store.close() + + def test_mixed_targets_unique_key_probing(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_cloud7", lane="cloud", grade=7) + t2 = self.make_task(workspace, "route/02_cloud9", lane="cloud", grade=9) + + store = dispatch.StateStore(workspace) + try: + probed_keys = [] + + def mock_probe(*args, **kwargs): + probed_keys.append((kwargs["adapter"], kwargs["target"])) + target = kwargs["target"] + adapter = kwargs["adapter"] + checked_at_iso = kwargs["checked_at"].astimezone(dispatch.KST).isoformat() + return { + "schema_version": "1.0", + "snapshot_id": f"snap-{adapter}-{target}", + "source": "iop-node quota-probe", + "checked_at": checked_at_iso, + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [{"name": "overall", "status": "available", "remaining_percent": 100.0}], + "reason_codes": ["ok"], + } + + selector = dispatch._selector_module() + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker"), (t2, "worker")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + self.assertIsNotNone(batch_snap) + # Ensure no duplicate probes were called and exactly 2 unique targets were probed + self.assertEqual(len(probed_keys), 2) + self.assertEqual(len(probed_keys), len(set(probed_keys))) + + d1, _ = dispatch.persisted_execution_decision(store, t1, stage="worker", quota_snapshot=batch_snap) + d2, _ = dispatch.persisted_execution_decision(store, t2, stage="worker", quota_snapshot=batch_snap) + + self.assertEqual(d1["quota"]["snapshot_id"], batch_snap["snapshot_id"]) + self.assertEqual(d2["quota"]["snapshot_id"], batch_snap["snapshot_id"]) + self.assertEqual(d1["quota"]["status"], "available") + self.assertEqual(d2["quota"]["status"], "available") + finally: + store.close() + + def test_local_and_resume_zero_probe_count(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_local = self.make_task(workspace, "route/01_local", lane="local", grade=5) + t_resume = self.make_task(workspace, "route/02_resume", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + # Give t_resume a prior decision + store.update_task( + t_resume, + execution_decisions={ + "worker": { + "work_unit_id": dispatch.work_unit_id_from_file(t_resume.plan), + "stage": "worker", + "selected": { + "adapter": "agy", + "target": "gemini-2.5-flash", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + "quota": { + "snapshot_id": "prior-snap", + "mode": "bounded", + "status": "available", + "source": "iop-node quota-probe", + "checked_at": datetime.now(dispatch.KST).isoformat(), + "targets": [], + }, + } + }, + ) + + probe_calls = [] + selector = dispatch._selector_module() + with mock.patch.object(selector, "probe_candidate_quota", side_effect=lambda **kw: probe_calls.append(kw)): + now = datetime.now(dispatch.KST) + ready = [(t_local, "worker"), (t_resume, "worker")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + # Local task candidate is local_model, resume task has prior decision -> 0 probes needed! + self.assertIsNone(batch_snap) + self.assertEqual(len(probe_calls), 0) + finally: + store.close() + + def test_night_local_and_official_review_zero_probe_count(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_night = self.make_task(workspace, "route/01_night", lane="local", grade=8) + t_review = self.make_task(workspace, "route/02_review", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + probe_calls = [] + selector = dispatch._selector_module() + with ( + mock.patch.object( + selector, + "probe_candidate_quota", + side_effect=lambda **kw: probe_calls.append(kw), + ), + mock.patch("subprocess.run", side_effect=AssertionError("subprocess called")), + ): + now = datetime(2026, 7, 26, 23, 30, 0, tzinfo=dispatch.KST) + ready = [(t_night, "worker"), (t_review, "review")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + # Night local-G08 candidate is local_model first, official review stage is not worker -> 0 probes needed! + self.assertIsNone(batch_snap) + self.assertEqual(len(probe_calls), 0) + finally: + store.close() + + def make_attempt_locator( + self, workspace: Path, task: dispatch.Task, spec: dispatch.AgentSpec + ) -> Path: + attempt = workspace / f"attempt-{spec.cli}" + attempt.mkdir(parents=True, exist_ok=True) + raw, normalized = attempt / "stream.log", attempt / "normalized-output.log" + raw.write_text("raw log\n", encoding="utf-8") + normalized.write_text("normalized output\n", encoding="utf-8") + locator = attempt / "locator.json" + record = { + "task": task.name, + "workspace": str(workspace), + "plan_path": str(task.plan), + "stream_log": str(raw), + "normalized_output_log": str(normalized), + "cli": spec.cli, + "model": spec.model, + "spec": {"adapter": spec.cli, "target": spec.model}, + } + locator.write_text(json.dumps(record), encoding="utf-8") + return locator + + def test_same_provider_target_tasks_with_disjoint_write_sets_admit_without_cap(self): + async def run(): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + bin_dir = workspace / "agent-ops" / "bin" + bin_dir.mkdir(parents=True, exist_ok=True) + ai_ignore = bin_dir / "ai-ignore.sh" + ai_ignore.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") + ai_ignore.chmod(0o755) + tasks = [ + self.make_task(workspace, f"route/0{i}_task", lane="local", grade=8) + for i in range(1, 6) + ] + + store = dispatch.StateStore(workspace) + try: + barrier = asyncio.Barrier(5) + completed_archive = workspace / "completed" + completed_archive.mkdir() + (completed_archive / "complete.log").write_text("complete\n", encoding="utf-8") + + async def fake_invoke(*args, **kwargs): + role = args[3] + task = args[2] + spec = args[4] + loc = self.make_attempt_locator(workspace, task, spec) + if role == "worker": + await asyncio.wait_for(barrier.wait(), timeout=2.0) + elif role == "review": + group, leaf = task.name.split("/", 1) + archive_dir = ( + workspace + / "agent-task" + / "archive" + / "2026" + / "07" + / group + / leaf + ) + archive_dir.mkdir(parents=True, exist_ok=True) + (archive_dir / "complete.log").write_text("complete\n", encoding="utf-8") + (archive_dir / "code_review_cloud_G07_0.log").write_text( + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n\n- ์ข…ํ•ฉ ํŒ์ •: PASS\n", encoding="utf-8" + ) + import shutil + shutil.rmtree(task.directory, ignore_errors=True) + return (0, None, loc) + + def mock_probe(*args, **kwargs): + target = kwargs.get("target", "ornith:35b") + adapter = kwargs.get("adapter", "pi") + return { + "schema_version": "1.0", + "snapshot_id": f"snap-{adapter}-{target}", + "source": "iop-node quota-probe", + "checked_at": datetime.now(dispatch.KST).isoformat(), + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [{"name": "overall", "status": "available", "remaining_percent": 100.0}], + "reason_codes": ["ok"], + } + + selector = dispatch._selector_module() + with ( + mock.patch.object(dispatch, "invoke", side_effect=fake_invoke), + mock.patch.object(dispatch, "implementation_review_errors", return_value=[]), + mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe), + ): + args = SimpleNamespace( + task_group="route", + retry_blocked=False, + dry_run=False, + max_parallel=0, + ) + exit_code = await asyncio.wait_for( + dispatch.dispatch_with_store(args, workspace, store), + timeout=5.0, + ) + self.assertEqual(exit_code, 0) + self.assertEqual(barrier.n_waiting, 0) + finally: + store.close() + + asyncio.run(run()) + + def test_batch_key_unknown_isolation(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_cloud7", lane="cloud", grade=7) + t2 = self.make_task(workspace, "route/02_cloud9", lane="cloud", grade=9) + + store = dispatch.StateStore(workspace) + try: + def mock_probe(*args, **kwargs): + adapter = kwargs["adapter"] + target = kwargs["target"] + if adapter == "claude": + raise RuntimeError("Quota probe unexpected failure") + checked_at_iso = kwargs["checked_at"].astimezone(dispatch.KST).isoformat() + return { + "schema_version": "1.0", + "snapshot_id": f"snap-{adapter}-{target}", + "source": "iop-node quota-probe", + "checked_at": checked_at_iso, + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [{"name": "overall", "status": "available", "remaining_percent": 100.0}], + "reason_codes": ["ok"], + } + + selector = dispatch._selector_module() + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker"), (t2, "worker")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + self.assertIsNotNone(batch_snap) + statuses = {t["adapter"]: t["status"] for t in batch_snap["targets"]} + # Failed probe key is isolated as unknown, while other key succeeded as available + self.assertIn("unknown", list(statuses.values())) + self.assertIn("available", list(statuses.values())) + + d2, _ = dispatch.persisted_execution_decision(store, t2, stage="worker", quota_snapshot=batch_snap) + self.assertEqual(d2["quota"]["status"], "available") + finally: + store.close() + + def test_same_work_unit_resume_zero_probe_and_pin_preserved(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_pinned", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + deterministic_snapshot = { + "schema_version": "1.0", + "snapshot_id": "snap-init", + "source": "fake_probe", + "checked_at": datetime.now(dispatch.KST).isoformat(), + "targets": [ + { + "adapter": "codex", + "target": "gpt-5.6-sol", + "status": "available", + "reason_codes": [], + } + ], + "required_caps": [], + "reason_codes": [], + } + with mock.patch("subprocess.run", side_effect=AssertionError) as run: + init_d, _ = dispatch.persisted_execution_decision( + store, t1, stage="worker", quota_snapshot=deterministic_snapshot + ) + run.assert_not_called() + self.assertIsNotNone(init_d) + + probe_calls = [] + with mock.patch.object(selector, "probe_candidate_quota", side_effect=lambda **kw: probe_calls.append(kw)): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + # Persisted work unit for same work_unit_id -> 0 probe calls + self.assertIsNone(batch_snap) + self.assertEqual(len(probe_calls), 0) + + with mock.patch("subprocess.run", side_effect=AssertionError) as run: + d, spec = dispatch.persisted_execution_decision(store, t1, stage="worker") + run.assert_not_called() + self.assertIs(d["decision"]["pinned"], True) + self.assertEqual(d["work_unit_id"], init_d["work_unit_id"]) + finally: + store.close() + + def test_new_generation_next_batch_available_recovery(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_gen", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + # Store prior decision with an OLD work_unit_id + store.update_task( + t1, + execution_decisions={ + "worker": { + "work_unit_id": "old_task::plan-0::tag-OLD", + "stage": "worker", + "selected": { + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + "quota": { + "snapshot_id": "old-snap", + "mode": "bounded", + "status": "exhausted", + "source": "iop-node quota-probe", + "checked_at": datetime.now(dispatch.KST).isoformat(), + "targets": [], + }, + } + }, + ) + + probe_calls = [] + def mock_probe(*args, **kwargs): + probe_calls.append(kwargs) + target, adapter = kwargs["target"], kwargs["adapter"] + checked_at_iso = kwargs["checked_at"].astimezone(dispatch.KST).isoformat() + return { + "schema_version": "1.0", + "snapshot_id": f"fresh-snap-{adapter}-{target}", + "source": "iop-node quota-probe", + "checked_at": checked_at_iso, + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [{"name": "overall", "status": "available", "remaining_percent": 100.0}], + "reason_codes": ["ok"], + } + + selector = dispatch._selector_module() + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker")] + batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + # New generation -> probe runs, fresh snapshot returned + self.assertIsNotNone(batch_snap) + self.assertGreater(len(probe_calls), 0) + + d, _ = dispatch.persisted_execution_decision(store, t1, stage="worker", quota_snapshot=batch_snap) + self.assertEqual(d["quota"]["status"], "available") + finally: + store.close() + + def test_confirmed_provider_quota_task_local_derived_exhausted(self): + current_decision = { + "work_unit_id": "route/01_unit::plan-0::tag-ROUTE", + "stage": "worker", + "selected": {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + "quota": { + "snapshot_id": "shared-batch-123", + "mode": "bounded", + "status": "available", + "source": "iop-node quota-probe", + "checked_at": "2026-07-26T18:00:00+09:00", + "targets": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)", "status": "available"}, + {"adapter": "codex", "target": "gpt-5.6-sol", "status": "available"}, + ], + "required_caps": [{"name": "overall", "status": "available"}], + "reason_codes": ["ok"], + }, + } + + derived = dispatch.derive_work_unit_quota_evidence( + current_decision, status="exhausted", reason="confirmed_runtime_provider_quota" + ) + + # Observation identity preserved + self.assertEqual(derived["snapshot_id"], "shared-batch-123") + self.assertEqual(derived["checked_at"], "2026-07-26T18:00:00+09:00") + self.assertEqual(derived["source"], "iop-node quota-probe") + self.assertIn("confirmed_runtime_provider_quota", derived["reason_codes"]) + + # Selected target status updated to exhausted + selected_entry = next( + t for t in derived["targets"] if t["adapter"] == "agy" and t["target"] == "Gemini 3.6 Flash (Medium)" + ) + self.assertEqual(selected_entry["status"], "exhausted") + + # Original shared decision quota targets NOT mutated + original_entry = next( + t for t in current_decision["quota"]["targets"] if t["adapter"] == "agy" and t["target"] == "Gemini 3.6 Flash (Medium)" + ) + self.assertEqual(original_entry["status"], "available") + + def test_retry_blocked_quota_refresh_lifecycle(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_retry", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + init_snap = { + "schema_version": "1.0", + "snapshot_id": "snap-initial", + "source": "fake_probe", + "checked_at": datetime.now(dispatch.KST).isoformat(), + "targets": [ + { + "adapter": "codex", + "target": "gpt-5.6-sol", + "status": "available", + "reason_codes": [], + } + ], + "required_caps": [], + "reason_codes": [], + } + with mock.patch("subprocess.run", side_effect=AssertionError) as run: + init_d, _ = dispatch.persisted_execution_decision( + store, t1, stage="worker", quota_snapshot=init_snap + ) + run.assert_not_called() + self.assertEqual(init_d["quota"]["snapshot_id"], "snap-initial") + + # Block the task + locator = workspace / "retry-locator.json" + locator.write_text("{}", encoding="utf-8") + store.update_task( + t1, + blocked="worker failure provider-quota", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(locator), + "selected": init_d["selected"], + "work_unit_id": init_d["work_unit_id"], + }, + ) + self.assertIsNotNone(store.task_state(t1).get("blocked")) + + # Mark retry quota refresh (simulating --retry-blocked) + store.mark_retry_quota_refresh("route/01_retry") + self.assertIsNone(store.task_state(t1).get("blocked")) + self.assertTrue(store.task_state(t1).get("retry_quota_refresh_pending")) + + # Admission batch snapshot now triggers a fresh probe because refresh is pending + probe_calls = [] + + def mock_probe(*args, **kwargs): + probe_calls.append(kwargs) + adapter = kwargs["adapter"] + target = kwargs["target"] + checked_at_iso = kwargs["checked_at"].astimezone(dispatch.KST).isoformat() + return { + "schema_version": "1.0", + "snapshot_id": "fresh-retry-snap", + "source": "iop-node quota-probe", + "checked_at": checked_at_iso, + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [], + "reason_codes": ["ok"], + } + + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker")] + retry_batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + self.assertIsNone(retry_batch_snap) + self.assertEqual(len(probe_calls), 0) + + # With no persisted unused alternate, retry consumes no quota snapshot and resumes. + with mock.patch("subprocess.run", side_effect=AssertionError) as run: + d, spec = dispatch.persisted_execution_decision( + store, t1, stage="worker", quota_snapshot=retry_batch_snap + ) + run.assert_not_called() + self.assertEqual(store.task_state(t1).get("quota_snapshot")["snapshot_id"], "snap-initial") + # retry context is preserved through decision commit so that + # invoke() can read handoff_id and atomically consume it. + # In production run_worker() always calls invoke() after this. + self.assertTrue(store.task_state(t1).get("retry_quota_refresh_pending")) + + # Subsequent admission pass (ordinary resume) -> 0 probe calls + probe_calls.clear() + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + now = datetime.now(dispatch.KST) + ready = [(t1, "worker")] + resume_batch_snap = dispatch.build_admission_batch_snapshot(store, ready, now) + + self.assertIsNone(resume_batch_snap) + self.assertEqual(len(probe_calls), 0) + finally: + store.close() + + def test_retry_blocked_scopes_to_blocked_worker_and_refreshes_pinned_alternate(self): + async def _async_run(): + nighttime = datetime(2026, 7, 26, 23, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_blocked = self.make_task(workspace, "route/01_blocked", lane="local", grade=8) + t_normal = self.make_task(workspace, "route/02_normal", lane="cloud", grade=7) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + + normal_snap = { + "schema_version": "1.0", + "snapshot_id": "snap-normal", + "source": "fake_probe", + "checked_at": nighttime.isoformat(), + "targets": [ + { + "adapter": "codex", + "target": "gpt-5.6-sol", + "status": "available", + "reason_codes": [], + } + ], + "required_caps": [], + "reason_codes": [], + } + with mock.patch("subprocess.run", side_effect=AssertionError) as run: + d_normal, spec_normal = dispatch.persisted_execution_decision( + store, t_normal, stage="worker", evaluated_at=nighttime, quota_snapshot=normal_snap + ) + run.assert_not_called() + + with mock.patch("subprocess.run", side_effect=AssertionError) as run: + d_blocked, spec_blocked = dispatch.persisted_execution_decision( + store, t_blocked, stage="worker", evaluated_at=nighttime, quota_snapshot=normal_snap + ) + run.assert_not_called() + self.assertEqual(d_blocked["selected"]["adapter"], "pi") + self.assertEqual(d_blocked["selected"]["target"], "iop/laguna-s:2.1") + + loc_path = workspace / "attempt-loc.json" + loc_path.write_text("{}", encoding="utf-8") + store.update_task( + t_blocked, + blocked=f"worker failure provider-quota locator={loc_path}", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(loc_path), + "selected": d_blocked["selected"], + "work_unit_id": d_blocked["work_unit_id"], + } + ) + self.assertIsNotNone(store.task_state(t_blocked).get("blocked")) + state_normal_before = dict(store.task_state(t_normal)) + + probe_calls = [] + + def mock_probe(*args, **kwargs): + probe_calls.append(kwargs) + adapter = kwargs["adapter"] + target = kwargs["target"] + checked_at_iso = kwargs["checked_at"].astimezone(dispatch.KST).isoformat() + return { + "schema_version": "1.0", + "snapshot_id": "fresh-retry-alternate-snap", + "source": "iop-node quota-probe", + "checked_at": checked_at_iso, + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [], + "reason_codes": ["ok"], + } + + invoke_calls = [] + async def fake_invoke(ws, st, task, role, spec, prompt, resume_locator=None): + locator = ws / f"{task.name.replace('/', '_')}-{role}.json" + locator.write_text("{}", encoding="utf-8") + # Consume pending retry handoff if one exists, matching + # the real invoke() path so the dispatch flow test remains + # consistent with the production handoff commit behavior. + if isinstance(st, dispatch.StateStore): + retry_ctx = st.task_state(task).get("retry_quota_refresh_context") + if isinstance(retry_ctx, dict) and retry_ctx.get("handoff_id"): + st.commit_retry_handoff_locator(task, retry_ctx["handoff_id"], str(locator)) + invoke_calls.append((task.name, role, spec, prompt, resume_locator)) + return 0, None, locator + + async def fake_run_review(ws, st, task, **kwargs): + archive = ws / "agent-task" / "archive" / "2026" / "07" / task.name + archive.parent.mkdir(parents=True, exist_ok=True) + (task.directory / "complete.log").write_text("simulation complete\n", encoding="utf-8") + task.directory.rename(archive) + return str(archive) + + args = dispatch.argparse.Namespace( + workspace=str(workspace), + task_group="route", + retry_blocked=True, + dry_run=False, + ) + + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe), \ + mock.patch.object(dispatch, "run_review", side_effect=fake_run_review), \ + mock.patch.object(dispatch, "ensure_review_shared_state"), \ + mock.patch.object(dispatch, "invoke", side_effect=fake_invoke), \ + mock.patch.object(dispatch, "datetime") as datetime_mock, \ + mock.patch("subprocess.run", side_effect=AssertionError) as run_sub: + datetime_mock.now.return_value = nighttime + res = await dispatch.dispatch_with_store(args, workspace, store) + + run_sub.assert_not_called() + self.assertEqual(len(probe_calls), 1) + self.assertEqual(probe_calls[0]["adapter"], "agy") + self.assertEqual(probe_calls[0]["target"], "Gemini 3.6 Flash (Medium)") + + st_blocked_after = store.task_state(t_blocked) + self.assertIsNone(st_blocked_after.get("blocked")) + self.assertFalse(st_blocked_after.get("retry_quota_refresh_pending")) + dec_after = st_blocked_after["execution_decisions"]["worker"] + self.assertEqual(dec_after["selected"]["adapter"], "agy") + self.assertEqual(dec_after["selected"]["target"], "Gemini 3.6 Flash (Medium)") + self.assertEqual(dec_after["transition"]["trigger"], "provider-quota") + self.assertEqual(dec_after["work_unit_id"], d_blocked["work_unit_id"]) + + used = dec_after.get("used_candidates", []) + used_adapters = [u.get("adapter") for u in used] + self.assertIn("pi", used_adapters) + self.assertIn("agy", used_adapters) + self.assertTrue(len(st_blocked_after.get("route_transition_history", [])) >= 2) + blocked_invocations = [call for call in invoke_calls if call[0] == t_blocked.name] + self.assertEqual(len(blocked_invocations), 1) + self.assertEqual(blocked_invocations[0][1], "worker") + self.assertEqual(blocked_invocations[0][4], loc_path) + + st_normal_after = store.task_state(t_normal) + self.assertFalse(st_normal_after.get("retry_quota_refresh_pending")) + self.assertEqual( + st_normal_after["execution_decisions"]["worker"]["selected"], + state_normal_before["execution_decisions"]["worker"]["selected"], + ) + + probe_calls.clear() + invoke_calls.clear() + args_normal = dispatch.argparse.Namespace( + workspace=str(workspace), + task_group="route", + retry_blocked=False, + dry_run=False, + ) + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe), \ + mock.patch.object(dispatch, "run_review", side_effect=fake_run_review), \ + mock.patch.object(dispatch, "ensure_review_shared_state"), \ + mock.patch.object(dispatch, "invoke", side_effect=fake_invoke), \ + mock.patch.object(dispatch, "datetime") as datetime_mock, \ + mock.patch("subprocess.run", side_effect=AssertionError) as run_sub: + datetime_mock.now.return_value = nighttime + res2 = await dispatch.dispatch_with_store(args_normal, workspace, store) + + run_sub.assert_not_called() + self.assertEqual(len(probe_calls), 0) + finally: + store.close() + + asyncio.run(_async_run()) + + def test_generic_stderr_unknown_preservation(self): + selector = dispatch._selector_module() + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = SimpleNamespace( + returncode=1, stdout="", stderr="Error: connection timeout to quota service\n" + ) + now = datetime.now(dispatch.KST) + snapshot = selector.probe_candidate_quota( + target="Gemini 3.6 Flash (Medium)", + adapter="agy", + required_caps=["overall"], + checked_at=now, + ) + + self.assertIsNotNone(snapshot) + self.assertEqual(snapshot["targets"][0]["status"], "unknown") + self.assertIn("probe_error", snapshot["reason_codes"]) + + def test_retry_evidence_artifact_identity_variants(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t1 = self.make_task(workspace, "route/01_unit", lane="local", grade=8) + store = dispatch.StateStore(workspace) + try: + # Initialize worker decision + d_init, _ = dispatch.persisted_execution_decision(store, t1, stage="worker") + init_selected = d_init["selected"] + init_work_unit = d_init["work_unit_id"] + + loc_dir = workspace / "attempt-t1" + loc_dir.mkdir(parents=True, exist_ok=True) + loc_file = loc_dir / "locator.json" + stream_log = loc_dir / "stream.log" + stream_log.write_text("sample stream log", encoding="utf-8") + norm_log = loc_dir / "normalized-output.log" + norm_log.write_text("sample normalized output", encoding="utf-8") + loc_file.write_text( + json.dumps({ + "workspace": str(workspace.resolve()), + "task": t1.name, + "plan_path": str(t1.plan.resolve()), + "stream_log": str(stream_log.resolve()), + "normalized_output_log": str(norm_log.resolve()), + }), + encoding="utf-8", + ) + + # Variant 1: Selected mismatch but work_unit_id matches -> qualified True + # (qualified check only validates work_unit_id, not selected identity) + store.update_task( + t1, + blocked="worker failure provider-quota", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(loc_file), + "selected": {"adapter": "other", "target": "other-model"}, + "work_unit_id": init_work_unit, + }, + ) + store.mark_retry_quota_refresh("route/01_unit", workspace) + st = store.task_state(t1) + self.assertIsNone(st.get("blocked")) + self.assertTrue(st.get("retry_quota_refresh_pending"), + "work_unit_id matches so evidence is qualified") + + # Variant 2: Work unit mismatch -> qualified False + store.update_task( + t1, + blocked="worker failure provider-quota", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(loc_file), + "selected": init_selected, + "work_unit_id": "different_work_unit", + }, + ) + store.mark_retry_quota_refresh("route/01_unit", workspace) + st = store.task_state(t1) + self.assertFalse(st.get("retry_quota_refresh_pending")) + self.assertIsNone(st.get("retry_quota_refresh_context")) + + # Variant 3: Generic failure class -> qualified False + store.update_task( + t1, + blocked="worker failure generic-error", + blocker_evidence={ + "role": "worker", + "failure_class": "generic-error", + "locator": str(loc_file), + "selected": init_selected, + "work_unit_id": init_work_unit, + }, + ) + store.mark_retry_quota_refresh("route/01_unit", workspace) + st = store.task_state(t1) + self.assertFalse(st.get("retry_quota_refresh_pending")) + + # Variant 4: Empty/whitespace locator -> qualified False (locator.strip() check) + store.update_task( + t1, + blocked="worker failure provider-quota", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": " ", + "selected": init_selected, + "work_unit_id": init_work_unit, + }, + ) + store.mark_retry_quota_refresh("route/01_unit", workspace) + st = store.task_state(t1) + self.assertFalse(st.get("retry_quota_refresh_pending")) + + # Variant 5: Qualified evidence -> qualified True, retry_quota_refresh_pending True + store.update_task( + t1, + worker_done=False, + worker_decision={ + "work_unit_id": init_work_unit, + "selected": init_selected, + }, + blocked="worker failure provider-quota", + blocker_evidence={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(loc_file), + "selected": init_selected, + "work_unit_id": init_work_unit, + }, + ) + store.mark_retry_quota_refresh("route/01_unit", workspace) + st = store.task_state(t1) + self.assertIsNone(st.get("blocked")) + self.assertTrue(st.get("retry_quota_refresh_pending")) + self.assertIsNotNone(st.get("retry_quota_refresh_context")) + self.assertEqual(st.get("retry_quota_refresh_context")["locator"], str(loc_file)) + + # Variant 6: StageFailureBudget records failure count and last transition + stage_budget = dispatch.StageFailureBudget.from_decision(store, t1, d_init) + count = stage_budget.record_failure( + target=init_selected, transition="failover", + ) + self.assertEqual(count, 1) + budgets = stage_budget._budgets() + budget_entry = budgets.get(stage_budget.key, {}) + self.assertEqual(budget_entry.get("last_transition"), "failover") + self.assertEqual( + budget_entry.get("last_target"), + {"adapter": init_selected.get("adapter"), "target": init_selected.get("target")}, + ) + finally: + store.close() + + def test_retry_handoff_locator_consume_restart_windows(self): + """Verify crash/restart exactly-once: locator-first consume prevents duplicate invoke. + + Simulates the crash window directly: state has active_locator set and + retry_quota_refresh_pending=True (simulating a crash after locator write + but before consume). The run_worker pre-check must find the active + locator and consume the pending handoff before generating a new + handoff_id, preventing a duplicate invocation. + + Asserts: consume returns True, pending cleared, no new handoff_id + generated when active_locator already matches, subprocess never called. + """ + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_task = self.make_task(workspace, "route/01_crash_test", lane="local", grade=8) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + + with mock.patch.object(selector, "probe_candidate_quota", return_value={ + "schema_version": "1.0", + "snapshot_id": "test-snap", + "source": "test_probe", + "checked_at": "2026-07-26T23:00:00+09:00", + "targets": [{"adapter": "pi", "target": "iop/laguna-s:2.1", "status": "available"}], + "required_caps": [], + "reason_codes": [], + }): + d_task, _ = dispatch.persisted_execution_decision( + store, t_task, stage="worker", evaluated_at=dispatch.datetime(2026, 7, 26, 23, 0, 0, tzinfo=dispatch.timezone(dispatch.timedelta(hours=9))) + ) + + # Set up the crash window state: active_locator written but + # retry_quota_refresh_pending still True (crash before consume). + crash_locator = str(workspace / "attempt-crash-worker" / "locator.json") + (workspace / "attempt-crash-worker").mkdir(parents=True, exist_ok=True) + (Path(crash_locator)).write_text( + json.dumps({ + "status": "succeeded", + "task": t_task.name, + "role": "worker", + "handoff_id": "crash-handoff-id-123", + "source_locator": crash_locator, + "source_context": { + "role": "worker", + "failure_class": "provider-quota", + "selected": d_task["selected"], + "work_unit_id": d_task["work_unit_id"], + }, + }), + encoding="utf-8", + ) + + store.update_task( + t_task, + active_locator=crash_locator, + retry_quota_refresh_pending=True, + retry_quota_refresh_context={ + "role": "worker", + "failure_class": "provider-quota", + "locator": crash_locator, + "selected": d_task["selected"], + "work_unit_id": d_task["work_unit_id"], + }, + ) + + st_before = store.task_state(t_task) + self.assertEqual(st_before.get("active_locator"), crash_locator) + self.assertTrue(st_before.get("retry_quota_refresh_pending")) + self.assertIsNotNone(st_before.get("retry_quota_refresh_context")) + + # Simulate the run_worker pre-check: when active_locator exists + # and retry_quota_refresh_pending is True, consume the pending + # handoff to prevent duplicate invocation. + prior_state = store.task_state(t_task) + prior_active = prior_state.get("active_locator") + prior_pending = prior_state.get("retry_quota_refresh_pending") + + consumed = False + if prior_active and prior_pending: + consumed = store.consume_matching_retry_handoff(t_task, prior_active) + + self.assertTrue(consumed, "consume_matching_retry_handoff should return True when active_locator matches pending context locator") + + # Verify pending handoff is consumed + st_after = store.task_state(t_task) + self.assertFalse(st_after.get("retry_quota_refresh_pending"), + "retry_quota_refresh_pending should be False after consume") + self.assertIsNone(st_after.get("retry_quota_refresh_context"), + "retry_quota_refresh_context should be None after consume") + # active_locator should still be set (consume only clears retry fields) + self.assertEqual(st_after.get("active_locator"), crash_locator) + + # Verify consume returns False when no pending handoff (second call) + result_no_pending = store.consume_matching_retry_handoff(t_task, crash_locator) + self.assertFalse(result_no_pending, + "consume should return False when no pending handoff remains") + + # Verify consume returns False when locator doesn't match + result_mismatch = store.consume_matching_retry_handoff(t_task, "/nonexistent/locator.json") + self.assertFalse(result_mismatch, + "consume should return False when locator mismatches active_locator") + + # Verify consume returns False when active_locator is None + store.update_task(t_task, active_locator=None) + result_no_active = store.consume_matching_retry_handoff(t_task, crash_locator) + self.assertFalse(result_no_active, + "consume should return False when active_locator is None") + + # Verify consume returns False when context locator doesn't match active_locator + store.update_task( + t_task, + active_locator=crash_locator, + retry_quota_refresh_pending=True, + retry_quota_refresh_context={ + "role": "worker", + "failure_class": "model-error", + "locator": "/different/locator.json", + "selected": {"adapter": "pi", "target": "other"}, + "work_unit_id": "different-work-unit", + }, + ) + result_ctx_mismatch = store.consume_matching_retry_handoff(t_task, crash_locator) + self.assertFalse(result_ctx_mismatch, + "consume should return False when context locator doesn't match active_locator") + + # subprocess.run must never be called in this test + with mock.patch("subprocess.run", side_effect=AssertionError) as run_sub: + store.consume_matching_retry_handoff(t_task, crash_locator) + run_sub.assert_not_called() + finally: + store.close() + + def test_retry_handoff_first_locator_record_and_commit_guard(self): + """Verify the first durable locator write already embeds the stable + handoff_id, and that both a commit mismatch and a commit save-fault + stop before the provider process seam is reached. + + Calls the real dispatch.invoke() production function (not a helper + copy) and replaces StateStore.commit_retry_handoff_locator with a + deterministic mismatch/fault so the ordering guarantee โ€” first + durable write already carries the ID, then a gated commit โ€” can be + observed directly. + """ + async def _async_run(): + nighttime = datetime(2026, 7, 26, 23, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_task = self.make_task(workspace, "route/01_first_record", lane="local", grade=8) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + quota_result = { + "schema_version": "1.0", + "snapshot_id": "test-snap", + "source": "test_probe", + "checked_at": nighttime.isoformat(), + "targets": [{"adapter": "pi", "target": "iop/laguna-s:2.1", "status": "available"}], + "required_caps": [], + "reason_codes": [], + } + with mock.patch.object(selector, "probe_candidate_quota", return_value=quota_result): + d_initial, _ = dispatch.persisted_execution_decision( + store, t_task, stage="worker", evaluated_at=nighttime + ) + + prior_locator = workspace / "prior-attempt" / "locator.json" + prior_locator.parent.mkdir(parents=True, exist_ok=True) + prior_locator.write_text( + json.dumps({"status": "failed", "task": t_task.name, "role": "worker"}), + encoding="utf-8", + ) + + def set_pending_context(): + store.update_task( + t_task, + worker_done=False, + blocked=None, + blocker_evidence=None, + retry_quota_refresh_pending=True, + retry_quota_refresh_context={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(prior_locator), + "selected": d_initial["selected"], + "work_unit_id": d_initial["work_unit_id"], + "handoff_id": "stable-handoff-id-guard-001", + }, + ) + + set_pending_context() + + with mock.patch.object(selector, "probe_candidate_quota", return_value=quota_result): + d_decision, spec = dispatch.persisted_execution_decision( + store, t_task, stage="worker", evaluated_at=nighttime + ) + + write_calls: list[dict[str, Any]] = [] + original_write_json = dispatch.write_json + + def counting_write_json(path, payload): + # StateStore.save() also goes through write_json (for + # state.json); only locator.json writes are the ones + # this test's first-record ordering guarantee is about. + if Path(path).name == "locator.json": + write_calls.append(json.loads(json.dumps(payload))) + return original_write_json(path, payload) + + subprocess_count = [0] + + async def deny_subprocess(*args, **kwargs): + subprocess_count[0] += 1 + raise RuntimeError("provider process seam must not be reached") + + original_commit = store.commit_retry_handoff_locator + + # Variant A: commit mismatch. Something else consumes the + # pending handoff out from under invoke() right before its + # own commit call, so the real commit legitimately returns + # False (pending already cleared). + def mismatching_commit(task, handoff_id, locator_path): + store.update_task( + task, + retry_quota_refresh_pending=False, + retry_quota_refresh_context=None, + ) + return original_commit(task, handoff_id, locator_path) + + with ( + mock.patch.object(dispatch, "write_json", side_effect=counting_write_json), + mock.patch.object(store, "commit_retry_handoff_locator", side_effect=mismatching_commit), + mock.patch("asyncio.create_subprocess_exec", side_effect=deny_subprocess), + ): + with self.assertRaises(dispatch.ExecutionDecisionError): + await dispatch.invoke( + workspace, store, t_task, "worker", spec, + f"test prompt A for {t_task.name}", None, + ) + + self.assertEqual(subprocess_count[0], 0, + "commit mismatch must stop before the provider process seam") + self.assertEqual(len(write_calls), 1, + "the locator must be durably written exactly once") + self.assertEqual( + write_calls[0].get("retry_handoff_id"), "stable-handoff-id-guard-001", + "the single durable write must already embed the stable handoff_id", + ) + + # Restore the pending handoff for variant B. + set_pending_context() + write_calls.clear() + subprocess_count[0] = 0 + + # Variant B: commit save-fault. + def faulting_commit(task, handoff_id, locator_path): + raise OSError("simulated disk fault during commit") + + with ( + mock.patch.object(dispatch, "write_json", side_effect=counting_write_json), + mock.patch.object(store, "commit_retry_handoff_locator", side_effect=faulting_commit), + mock.patch("asyncio.create_subprocess_exec", side_effect=deny_subprocess), + ): + with self.assertRaises(OSError): + await dispatch.invoke( + workspace, store, t_task, "worker", spec, + f"test prompt B for {t_task.name}", None, + ) + + self.assertEqual(subprocess_count[0], 0, + "commit save-fault must stop before the provider process seam") + self.assertEqual(len(write_calls), 1, + "the locator must be durably written exactly once even under save-fault") + self.assertEqual( + write_calls[0].get("retry_handoff_id"), "stable-handoff-id-guard-001", + ) + finally: + store.close() + + asyncio.run(_async_run()) + + def test_retry_handoff_production_save_fault_preserves_pending(self): + """Verify production commit save-fault preserves the pending handoff exactly. + + Drives the real dispatch.run_worker() production path (persisted + decision -> run_escalating -> invoke()) up to the point where + StateStore.commit_retry_handoff_locator() performs its durable save. + A fault injected precisely inside that call must leave the pending + handoff state โ€” keys, values, and on-disk serialization โ€” exactly + preserved, and the provider process seam must never be reached, even + though the crashed attempt's locator was already durably written + with the stable handoff_id before the faulting commit was attempted. + """ + async def _async_run(): + nighttime = datetime(2026, 7, 26, 23, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_task = self.make_task(workspace, "route/01_savefault", lane="local", grade=8) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + quota_result = { + "schema_version": "1.0", + "snapshot_id": "test-snap", + "source": "test_probe", + "checked_at": nighttime.isoformat(), + "targets": [{"adapter": "pi", "target": "iop/laguna-s:2.1", "status": "available"}], + "required_caps": [], + "reason_codes": [], + } + + with mock.patch.object(selector, "probe_candidate_quota", return_value=quota_result): + d_initial, _ = dispatch.persisted_execution_decision( + store, t_task, stage="worker", evaluated_at=nighttime + ) + + prior_locator = workspace / "prior-attempt" / "locator.json" + prior_locator.parent.mkdir(parents=True, exist_ok=True) + prior_locator.write_text( + json.dumps({ + "status": "failed", + "task": t_task.name, + "role": "worker", + "failure_class": "provider-quota", + }), + encoding="utf-8", + ) + + store.update_task( + t_task, + worker_done=False, + blocked=None, + blocker_evidence=None, + retry_quota_refresh_pending=True, + retry_quota_refresh_context={ + "role": "worker", + "failure_class": "provider-quota", + "locator": str(prior_locator), + "selected": d_initial["selected"], + "work_unit_id": d_initial["work_unit_id"], + "handoff_id": "stable-handoff-id-savefault-001", + }, + ) + + # persisted_execution_decision (called inside run_worker before + # invoke()) legitimately commits a fresh failover decision, so + # the rollback guarantee is scoped to the state immediately + # before the faulting commit attempt, not to the state before + # run_worker() started. Snapshot it right there. + pre_commit_snapshot: list[dict[str, Any] | None] = [None] + original_commit = store.commit_retry_handoff_locator + + def snapshotting_commit(task, handoff_id, locator_path): + pre_commit_snapshot[0] = json.loads(json.dumps(store.task_state(task))) + return original_commit(task, handoff_id, locator_path) + + original_save = store.save + fault_triggered = [False] + + def faulting_save(): + if not fault_triggered[0] and any( + frame.function == "commit_retry_handoff_locator" + for frame in inspect.stack() + ): + fault_triggered[0] = True + raise OSError("simulated disk fault during commit_retry_handoff_locator") + original_save() + + store.save = faulting_save + + subprocess_count = [0] + + async def deny_subprocess(*args, **kwargs): + subprocess_count[0] += 1 + raise RuntimeError( + "provider process seam must not be reached when commit save faults" + ) + + with ( + mock.patch.object(store, "commit_retry_handoff_locator", side_effect=snapshotting_commit), + mock.patch.object(selector, "probe_candidate_quota", return_value=quota_result), + mock.patch("asyncio.create_subprocess_exec", side_effect=deny_subprocess), + ): + with self.assertRaises(OSError): + await dispatch.run_worker(workspace, store, t_task) + + store.save = original_save + self.assertTrue( + fault_triggered[0], + "fault must have been injected inside commit_retry_handoff_locator", + ) + self.assertIsNotNone( + pre_commit_snapshot[0], + "commit_retry_handoff_locator must have been reached before faulting", + ) + self.assertEqual( + subprocess_count[0], 0, + "provider seam must not be reached when the handoff commit faults", + ) + + st_after = json.loads(json.dumps(store.task_state(t_task))) + self.assertEqual( + st_after, pre_commit_snapshot[0], + "task state must be restored to exactly the pre-commit-attempt snapshot", + ) + + written = [ + json.loads(p.read_text(encoding="utf-8")) + for p in store.runs.glob("*/locator.json") + ] + matching = [ + r for r in written + if r.get("retry_handoff_id") == "stable-handoff-id-savefault-001" + ] + self.assertTrue( + matching, + "the crashed attempt's locator record must embed the stable handoff_id", + ) + finally: + store.close() + + asyncio.run(_async_run()) + + def test_retry_restart_does_not_duplicate_provider_or_mutate_sibling(self): + """Verify scheduler live-locator gate blocks re-launch after StateStore restart. + + Pre-populates a task state with a committed retry handoff and an active + locator recording a simulated live agent PID. After StateStore close/ + reopen (restart), dispatch.dispatch_with_store() must classify the task + as externally active via external_active_is_live() and skip it without + calling dispatch.invoke(). An independent normal sibling task's + decision/quota/transition state must stay exactly unchanged throughout. + """ + async def _async_run(): + nighttime = datetime(2026, 7, 26, 23, 0, 0, tzinfo=timezone(timedelta(hours=9))) + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + t_task = self.make_task(workspace, "route/01_restart", lane="local", grade=8) + t_sibling = self.make_task(workspace, "route/02_sibling_normal", lane="local", grade=8) + + store = dispatch.StateStore(workspace) + try: + selector = dispatch._selector_module() + quota_result = { + "schema_version": "1.0", + "snapshot_id": "test-snap", + "source": "test_probe", + "checked_at": nighttime.isoformat(), + "targets": [{"adapter": "pi", "target": "iop/laguna-s:2.1", "status": "available"}], + "required_caps": [], + "reason_codes": [], + } + + with mock.patch.object(selector, "probe_candidate_quota", return_value=quota_result): + d_initial, _ = dispatch.persisted_execution_decision( + store, t_task, stage="worker", evaluated_at=nighttime + ) + + # Mark the task as actively in the worker stage so the + # scheduler live-locator gate has a stage to evaluate. + store.update_task(t_task, active_stage="worker") + + # Pre-populate sibling with deterministic quota/decision/transition + # so we can prove it stays unchanged across the restart+dispatch cycle. + sibling_quota = { + "schema_version": "1.0", + "snapshot_id": "sibling-snap-001", + "source": "sibling-probe", + "checked_at": nighttime.isoformat(), + "targets": [ + {"adapter": "pi", "target": "iop/laguna-s:2.1", "status": "available"}, + {"adapter": "pi", "target": "pi/north-7:1.0", "status": "available"}, + ], + "required_caps": [ + {"name": "overall", "status": "available", "remaining_percent": 75.0} + ], + "reason_codes": ["ok"], + } + sibling_decision = { + "schema_version": "1.0", + "work_unit_id": "route/02_sibling_normal::plan-0::tag-ROUTE", + "stage": "worker", + "lane": "local", + "grade": 8, + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": False, + }, + "candidates": [ + { + "candidate_rank": 1, + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": False, + "quota_mode": "bounded", + "quota_status": "available", + "eligibility": "eligible", + "rejection_reason": None, + } + ], + "decision": { + "rule_id": "sibling-test-rule", + "policy_priority": 1, + "reason_codes": ["ok"], + "evaluated_at": nighttime.isoformat(), + "timezone": "KST", + "time_window": "2026-07-26T23:00:00+09:00~2026-07-27T23:00:00+09:00", + "pinned": False, + "resume": False, + }, + "quota": { + "snapshot_id": "sibling-snap-001", + "checked_at": nighttime.isoformat(), + "source": "sibling-probe", + "targets": sibling_quota["targets"], + }, + "transition": {"trigger": "initial", "from": None, "to": "worker"}, + } + sibling_transition_history = [ + { + "stage": "worker", + "transition": "initial", + "work_unit_id": "route/02_sibling_normal::plan-0::tag-ROUTE", + "candidates": sibling_decision["candidates"], + "selected": sibling_decision["selected"], + "decision": sibling_decision["decision"], + "reason_codes": ["ok"], + "quota": sibling_decision["quota"], + "stage_budget": 0, + } + ] + + # Build sibling decision/quota through the canonical selector + # path so the snapshot carries real persisted decision + quota + # evidence instead of handcrafted dict values. + with mock.patch.object(selector, "probe_candidate_quota", return_value=sibling_quota): + d_sibling, _ = dispatch.persisted_execution_decision( + store, t_sibling, stage="worker", evaluated_at=nighttime, + ) + # Re-apply only the isolation fields commit_execution_decision + # cleared, so the scheduler must not touch them either. + sibling_state = store.task_state(t_sibling) + sibling_state["blocked"] = "sibling-pinned-for-isolation" + sibling_state["blocker_evidence"] = "sibling-isolation-invariant-test" + store.save() + + # Capture pre-restart sibling snapshot for deep-equality check. + sibling_snapshot_before = json.loads( + json.dumps(store.data["tasks"][t_sibling.name]) + ) + + # Pre-populate: simulate that the first attempt already committed + # a handoff and created an active locator with a live agent PID. + first_attempt_dir = ( + store.runs / "20260726T230000SZ__retry-worker-01" + ) + first_attempt_dir.mkdir(parents=True) + first_locator_path = first_attempt_dir / "locator.json" + fake_agent_pid = 99999 + first_locator_data = { + "status": "running", + "workspace": str(workspace.resolve()), + "workspace_id": store.workspace_id, + "task": t_task.name, + "role": "worker", + "attempt": 0, + "agent_pid": fake_agent_pid, + "agent_process_start_token": "fake-token-abc123", + "dispatcher_pid": os.getpid(), + "dispatcher_process_start_token": "fake-dispatcher-token", + "agent_process_marker": ( + f"w{store.workspace_id}__retry-worker-01__{uuid.uuid4()}" + ), + "retry_handoff_id": "stable-handoff-id-restart-001", + "cli": "agy", + "model": "Gemini 3.6 Flash (Medium)", + "reasoning_effort": "high", + } + first_locator_path.write_text( + json.dumps(first_locator_data), encoding="utf-8" + ) + + # Set up pending retry handoff so commit_retry_handoff_locator + # has a matching pending context to consume. + handoff_id = "stable-handoff-id-restart-001" + store.update_task( + t_task, + worker_done=False, + blocked=None, + blocker_evidence=None, + retry_quota_refresh_pending=True, + retry_quota_refresh_context={ + "handoff_id": handoff_id, + "role": "worker", + "failure_class": "provider-quota", + "locator": str(first_locator_path), + }, + ) + + # Actually consume the pending handoff through the production + # commit path instead of patching the state directly. + consumed = store.commit_retry_handoff_locator( + t_task, handoff_id, str(first_locator_path), + ) + self.assertTrue(consumed, "commit_retry_handoff_locator must consume the pending handoff") + self.assertFalse( + store.task_state(t_task)["retry_quota_refresh_pending"], + "retry_quota_refresh_pending must be cleared after consume", + ) + self.assertIsNone( + store.task_state(t_task)["retry_quota_refresh_context"], + "retry_quota_refresh_context must be None after consume", + ) + + # Verify pre-restart state + st_before = store.task_state(t_task) + self.assertFalse(st_before.get("retry_quota_refresh_pending")) + self.assertIsNone(st_before.get("retry_quota_refresh_context")) + self.assertEqual(st_before.get("active_stage"), "worker") + self.assertEqual(st_before.get("active_locator"), str(first_locator_path)) + + # Simulate restart: close and reopen the StateStore. + store.close() + store2 = dispatch.StateStore(workspace) + try: + st_restart = store2.task_state(t_task) + self.assertFalse(st_restart.get("retry_quota_refresh_pending")) + self.assertIsNone(st_restart.get("retry_quota_refresh_context")) + self.assertEqual(st_restart.get("active_stage"), "worker") + + # Verify sibling state survived the restart byte-for-byte. + sibling_after_restart = json.loads( + json.dumps(store2.data["tasks"][t_sibling.name]) + ) + self.assertEqual( + sibling_after_restart, + sibling_snapshot_before, + "sibling state must survive StateStore close/reopen unchanged", + ) + + invoke_calls = [] + + async def spy_invoke(workspace, store, task, role, spec, prompt, resume_locator=None): + invoke_calls.append(task.name) + raise RuntimeError("provider process seam denied for test") + + subprocess_count = [0] + + async def deny_subprocess(*args, **kwargs): + subprocess_count[0] += 1 + raise RuntimeError("provider process seam denied for test") + + # Mock process_is_alive to return True for the recorded agent PID, + # simulating that the original agent process is still alive after restart. + original_process_is_alive = dispatch.process_is_alive + + def mock_process_is_alive(value, expected_start_token=None): + try: + pid = int(value) + if pid == fake_agent_pid: + return True + except (TypeError, ValueError): + pass + return original_process_is_alive(value, expected_start_token) + + # Second dispatch_with_store() run: the scheduler live-locator gate + # must classify the task as externally active and skip it. + args_restart = dispatch.argparse.Namespace( + workspace=str(workspace), + task_group=None, + retry_blocked=False, + dry_run=False, + ) + with mock.patch.object(dispatch, "process_is_alive", side_effect=mock_process_is_alive), \ + mock.patch.object(selector, "probe_candidate_quota", return_value=quota_result), \ + mock.patch.object(dispatch, "invoke", side_effect=spy_invoke), \ + mock.patch("asyncio.create_subprocess_exec", side_effect=deny_subprocess), \ + mock.patch.object(dispatch, "datetime") as datetime_mock: + datetime_mock.now.return_value = nighttime + result = await dispatch.dispatch_with_store(args_restart, workspace, store2) + + # The scheduler must NOT call invoke() for the already-active task. + retry_invoke_calls = [c for c in invoke_calls if c == t_task.name] + self.assertEqual( + len(retry_invoke_calls), 0, + "scheduler live-locator gate must prevent re-invoking the active task after restart", + ) + # The provider seam must NOT be reached for the already-active task. + self.assertEqual( + subprocess_count[0], 0, + "scheduler live-locator gate must prevent reaching the provider seam after restart", + ) + # dispatch_with_store should return 3 (blocked/waiting) since + # the task is externally active and cannot make progress. + self.assertEqual(result, 3, "dispatch should return blocked (3) when task is externally active") + + # === Sibling invariance assertions === + # After the full dispatch cycle, the independent normal sibling's + # quota_snapshot, execution_decisions, and route_transition_history + # must be JSON-deep-equal to the pre-restart snapshot. + sibling_snapshot_after = json.loads( + json.dumps(store2.data["tasks"][t_sibling.name]) + ) + self.assertEqual( + sibling_snapshot_after, + sibling_snapshot_before, + ( + "sibling quota_snapshot, execution_decisions, and " + "route_transition_history must remain JSON-deep-equal " + "after dispatch_with_store() cycle" + ), + ) + # Verify each sub-field individually for clearer failure messages. + self.assertEqual( + sibling_snapshot_after.get("quota_snapshot"), + sibling_snapshot_before.get("quota_snapshot"), + "sibling quota_snapshot must be unchanged", + ) + self.assertEqual( + sibling_snapshot_after.get("execution_decisions"), + sibling_snapshot_before.get("execution_decisions"), + "sibling execution_decisions must be unchanged", + ) + self.assertEqual( + sibling_snapshot_after.get("route_transition_history"), + sibling_snapshot_before.get("route_transition_history"), + "sibling route_transition_history must be unchanged", + ) + # The sibling must NOT have acquired an active_locator or + # active_stage from the restart dispatch cycle. + self.assertIsNone( + sibling_snapshot_after.get("active_locator"), + "sibling must not have active_locator set by restart dispatch", + ) + self.assertIsNone( + sibling_snapshot_after.get("active_stage"), + "sibling must not have active_stage set by restart dispatch", + ) + finally: + store2.close() + finally: + store.close() + + asyncio.run(_async_run()) + +class ArtifactLanguageContractTest(unittest.TestCase): + def test_canonical_english_sections_drive_runtime_contract(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + plan = root / "PLAN-local-G05.md" + plan.write_text( + "## Modified Files Summary\n\n" + "| File | Note |\n" + "|---|---|\n" + "| `apps/node/main.go:12` | main |\n", + encoding="utf-8", + ) + write_set, known = dispatch.extract_write_set(plan, root) + self.assertTrue(known) + self.assertIn(str((root / "apps/node/main.go").resolve()), write_set) + + task = TaskStageTest().make_task(root, "## Implementation Checklist\n\n- [ ] item 1\n") + errors = dispatch.implementation_review_errors(task) + self.assertEqual(errors, ["๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ ๋ฏธ์™„๋ฃŒ"]) + + task.review.write_text("## Implementation Checklist\n\n- [x] item 1\n", encoding="utf-8") + errors = dispatch.implementation_review_errors(task) + self.assertEqual(errors, []) + + verdict_text = ( + "## Code Review Result\n\n" + "- **Overall Verdict**: PASS\n" + ) + self.assertEqual(dispatch.verdict_from_text(verdict_text), "PASS") + + def test_legacy_korean_sections_remain_readable(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + plan = root / "PLAN-local-G05.md" + plan.write_text( + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n" + "| ํŒŒ์ผ | ๋น„๊ณ  |\n" + "|---|---|\n" + "| `apps/node/main.go:12` | main |\n", + encoding="utf-8", + ) + write_set, known = dispatch.extract_write_set(plan, root) + self.assertTrue(known) + self.assertIn(str((root / "apps/node/main.go").resolve()), write_set) + + task = TaskStageTest().make_task(root, "## ๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ\n\n- [x] item 1\n") + errors = dispatch.implementation_review_errors(task) + self.assertEqual(errors, []) + + verdict_text = ( + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n\n" + "- **์ข…ํ•ฉ ํŒ์ •**: WARN\n" + ) + self.assertEqual(dispatch.verdict_from_text(verdict_text), "WARN") + + def test_duplicate_language_aliases_fail_closed(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + plan = root / "PLAN-local-G05.md" + plan.write_text( + "## Modified Files Summary\n\n" + "| File |\n|---| \n| `apps/node/main.go` |\n\n" + "## ์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ\n\n" + "| ํŒŒ์ผ |\n|---| \n| `apps/node/main.go` |\n", + encoding="utf-8", + ) + write_set, known = dispatch.extract_write_set(plan, root) + self.assertFalse(known) + self.assertEqual(write_set, set()) + + review_text = ( + "## Implementation Checklist\n\n- [x] item 1\n\n" + "## ๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ\n\n- [x] item 1\n" + ) + task = TaskStageTest().make_task(root, review_text) + errors = dispatch.implementation_review_errors(task) + self.assertEqual(errors, ["๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ ๋ฏธ์™„๋ฃŒ"]) + + dup_verdict = ( + "## Code Review Result\n\n- **Overall Verdict**: PASS\n\n" + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n\n- **์ข…ํ•ฉ ํŒ์ •**: PASS\n" + ) + self.assertIsNone(dispatch.verdict_from_text(dup_verdict)) + + def test_recovery_accepts_canonical_and_legacy_logs(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + outside_verdict = ( + "## Overview\n\n- **Overall Verdict**: PASS\n\n" + "## Code Review Result\n\n- **Overall Verdict**: WARN\n" + ) + self.assertEqual(dispatch.verdict_from_text(outside_verdict), "WARN") + + canon_plan = root / "plan_local_G05_0.log" + canon_plan.write_text( + "\n\n# Plan\n", + encoding="utf-8", + ) + canon_review = root / "code_review_local_G05_0.log" + canon_review.write_text( + "\n\n" + "## Code Review Result\n\n- **Overall Verdict**: PASS\n", + encoding="utf-8", + ) + self.assertEqual(dispatch.read_verdict(canon_review), "PASS") + self.assertEqual(dispatch.latest_verdict_log(root), canon_review) + self.assertEqual(dispatch.matching_plan_log(root, canon_review), canon_plan) + + legacy_plan = root / "plan_local_G05_1.log" + legacy_plan.write_text( + "\n\n# Plan\n", + encoding="utf-8", + ) + legacy_review = root / "code_review_local_G05_1.log" + legacy_review.write_text( + "\n\n" + "## ์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ\n\n- **์ข…ํ•ฉ ํŒ์ •**: FAIL\n", + encoding="utf-8", + ) + self.assertEqual(dispatch.read_verdict(legacy_review), "FAIL") + self.assertEqual(dispatch.latest_verdict_log(root), legacy_review) + self.assertEqual(dispatch.matching_plan_log(root, legacy_review), legacy_plan) + + mismatch_review = root / "code_review_local_G05_2.log" + mismatch_review.write_text( + "\n\n" + "## Code Review Result\n\n- **Overall Verdict**: WARN\n", + encoding="utf-8", + ) + mismatch_plan = root / "plan_local_G05_2.log" + mismatch_plan.write_text( + "\n\n# Plan\n", + encoding="utf-8", + ) + self.assertEqual(dispatch.latest_verdict_log(root), mismatch_review) + self.assertIsNone(dispatch.matching_plan_log(root, mismatch_review)) + + def test_verdict_schema_pairs_reject_mixed_heading_labels(self): + self.assertEqual( + dispatch.CODE_REVIEW_RESULT_SCHEMAS, + ( + ("Code Review Result", "Overall Verdict"), + ("์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ", "์ข…ํ•ฉ ํŒ์ •"), + ), + ) + forms = { + "inline": "- **{label}**: {verdict}\n", + "block": "### {label}\n\n**{verdict}**\n", + } + for heading, paired_label in dispatch.CODE_REVIEW_RESULT_SCHEMAS: + for _, label in dispatch.CODE_REVIEW_RESULT_SCHEMAS: + for form_name, form in forms.items(): + text = f"## {heading}\n\n" + form.format( + label=label, verdict="PASS" + ) + with self.subTest(heading=heading, label=label, form=form_name): + if label == paired_label: + self.assertEqual(dispatch.verdict_from_text(text), "PASS") + else: + self.assertIsNone(dispatch.verdict_from_text(text)) + + @staticmethod + def contract_documents() -> dict[str, str]: + skills_root = Path(__file__).resolve().parents[3] + paths = { + "plan_skill": skills_root / "common" / "plan" / "SKILL.md", + "review_skill": skills_root / "common" / "code-review" / "SKILL.md", + "review_template": ( + skills_root / "common" / "plan" / "templates" / "review-stub-template.md" + ), + "orchestrator_skill": ( + skills_root + / "common" + / "orchestrate-agent-task-loop" + / "SKILL.md" + ), + } + return {name: path.read_text(encoding="utf-8") for name, path in paths.items()} + + def test_external_execution_user_review_contract_is_shared(self): + documents = self.contract_documents() + skills_root = Path(__file__).resolve().parents[3] + user_review_template = ( + skills_root + / "common" + / "code-review" + / "templates" + / "user-review-template.md" + ).read_text(encoding="utf-8") + + self.assertIn("`external-execution`", documents["plan_skill"]) + self.assertIn("`external-execution`", documents["review_skill"]) + self.assertIn("For `external-execution`", documents["orchestrator_skill"]) + self.assertIn( + "Do not create another follow-up PLAN that repeats the same inaccessible preflight.", + documents["review_skill"], + ) + self.assertIn( + "{milestone-lock | external-execution}", + user_review_template, + ) + self.assertIn("## Required User Action", user_review_template) + + def test_templates_and_prompts_separate_artifact_and_final_languages(self): + documents = self.contract_documents() + template = documents["review_template"] + plan_skill = documents["plan_skill"] + review_skill = documents["review_skill"] + orchestrator_skill = documents["orchestrator_skill"] + + for heading in ( + "## Overview", + "## For the Review Agent", + "## Implementation Checklist", + "## Review-Only Checklist", + "## Deviations from Plan", + "## Verification Results", + "## Key Design Decisions", + "## Reviewer Checkpoints", + ): + with self.subTest(template_heading=heading): + self.assertIn(heading, template) + + for label in ( + "Verification Results", + "Deviations from Plan", + "Background", + "Analysis", + "Split Judgment", + "Dependencies and Execution Order", + "Implementation Checklist", + "Review-Only Checklist", + "Code Review Result", + ): + with self.subTest(canonical_label=label): + self.assertIn(label, plan_skill) + + legacy_alias_pairs = { + "plan_skill": ( + "`Verification Results` or `Deviations from Plan` " + "(legacy: `๊ฒ€์ฆ ๊ฒฐ๊ณผ` or `๊ณ„ํš ๋Œ€๋น„ ๋ณ€๊ฒฝ ์‚ฌํ•ญ`)", + "`Code Review Result` [legacy: `์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ`]", + "`Verification Results` (legacy: `๊ฒ€์ฆ ๊ฒฐ๊ณผ`)", + "`Deviations from Plan` (legacy: `๊ณ„ํš ๋Œ€๋น„ ๋ณ€๊ฒฝ ์‚ฌํ•ญ`)", + "`Implementation Checklist` (legacy: `๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ`)", + "`Review-Only Checklist` (legacy: `์ฝ”๋“œ๋ฆฌ๋ทฐ ์ „์šฉ ์ฒดํฌ๋ฆฌ์ŠคํŠธ`)", + ), + "review_skill": ( + "`Implementation Checklist` (legacy: `๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ`)", + "`Review-Only Checklist` (legacy: `์ฝ”๋“œ๋ฆฌ๋ทฐ ์ „์šฉ ์ฒดํฌ๋ฆฌ์ŠคํŠธ`)", + ), + "orchestrator_skill": ( + "`Modified Files Summary` (and legacy `์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ`)", + "`## Implementation Checklist` (or legacy `## ๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ`)", + ), + } + for name, pairs in legacy_alias_pairs.items(): + for pair in pairs: + with self.subTest(document=name, alias_pair=pair): + self.assertIn(pair, documents[name]) + + def test_plan_skill_requires_backticks_for_claimed_paths(self): + plan_skill = self.contract_documents()["plan_skill"] + + self.assertIn("Wrap every claimed file path in backticks", plan_skill) + + def test_plan_and_review_share_dispatch_write_set_contract(self): + documents = self.contract_documents() + plan_skill = documents["plan_skill"] + review_skill = documents["review_skill"] + orchestrator_skill = documents["orchestrator_skill"] + + for document in (plan_skill, review_skill): + self.assertIn("dispatch.py --workspace --validate-plan", document) + self.assertIn("exact workspace", document) + self.assertIn("Never use a glob (`*`, `?`, `[]`)", plan_skill) + self.assertIn( + "globs, directories, workspace root", + review_skill.casefold(), + ) + self.assertIn( + "Fail the task closed when any path is broad", + orchestrator_skill, + ) + + # Legacy Korean artifact labels are allowed only as explicit aliases. + # Korean roadmap, USER_REVIEW.md, runtime banner, and user-facing + # response literals are deliberately outside this assertion. + legacy_terms = ( + "๊ฒ€์ฆ ๊ฒฐ๊ณผ", + "๊ณ„ํš ๋Œ€๋น„ ๋ณ€๊ฒฝ ์‚ฌํ•ญ", + "์ฝ”๋“œ๋ฆฌ๋ทฐ ๊ฒฐ๊ณผ", + "์ฝ”๋“œ๋ฆฌ๋ทฐ ์ „์šฉ ์ฒดํฌ๋ฆฌ์ŠคํŠธ", + "๊ตฌํ˜„ ์ฒดํฌ๋ฆฌ์ŠคํŠธ", + "์ˆ˜์ • ํŒŒ์ผ ์š”์•ฝ", + "์ข…ํ•ฉ ํŒ์ •", + ) + for name, text in documents.items(): + for number, line in enumerate(text.splitlines(), 1): + for term in legacy_terms: + if term not in line: + continue + with self.subTest(document=name, line=number, term=term): + self.assertIn("legacy", line.lower()) + + self.assertIn("append `## Code Review Result`", review_skill) + self.assertIn( + "- `Overall Verdict`: exactly `PASS`, `WARN`, or `FAIL`.", review_skill + ) + canonical_schema, legacy_schema = dispatch.CODE_REVIEW_RESULT_SCHEMAS + self.assertIn( + f"`## {canonical_schema[0]}` (with `{canonical_schema[1]}: PASS|WARN|FAIL`)", + orchestrator_skill, + ) + self.assertIn( + f"legacy `## {legacy_schema[0]}` (with `{legacy_schema[1]}: PASS|WARN|FAIL`)", + orchestrator_skill, + ) + + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + task = TaskStageTest().make_task(root) + review_missing = dispatch.Task( + name=task.name, + directory=task.directory, + plan=task.plan, + review=None, + user_review=None, + recovery=False, + lane="local", + grade=5, + ) + pi = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + codex = dispatch.AgentSpec("codex", "gpt-5.6-sol", "codex/gpt-5.6-sol xhigh") + locator = root / "locator.json" + context = { + "plan": str(task.plan.resolve()), + "locator": str(locator), + "workspace": str(root), + "raw_log": str(root / "stream.log"), + "normalized_output": str(root / "normalized-output.log"), + } + prompts = { + "worker": dispatch.base_prompt(task, "worker", codex), + "pi_worker": dispatch.base_prompt(task, "worker", pi), + "selfcheck": dispatch.base_prompt(task, "selfcheck", pi), + "selfcheck_unchecked": dispatch.base_prompt( + task, "selfcheck", pi, unchecked_items=True + ), + "official_review": dispatch.base_prompt(task, "review", codex), + "review_without_stub": dispatch.base_prompt( + review_missing, "review", codex + ), + "review_recovery": dispatch.continuation_prompt(task, "review"), + "logical_context": dispatch.logical_context_prompt(context), + "native_continuation": dispatch.continuation_prompt( + task, "worker", local_pi=True, resume_same_pi_session=True + ), + "pi_worker_continuation": dispatch.continuation_prompt( + task, "worker", local_pi=True + ), + "pi_selfcheck_continuation": dispatch.continuation_prompt( + task, "selfcheck", local_pi=True + ), + "pi_selfcheck_unchecked_continuation": ( + dispatch.continuation_prompt( + task, + "selfcheck", + local_pi=True, + unchecked_items=True, + ) + ), + "pi_selfcheck_native_continuation": ( + dispatch.continuation_prompt( + task, + "selfcheck", + local_pi=True, + resume_same_pi_session=True, + ) + ), + "worker_continuation": dispatch.continuation_prompt( + task, "worker", locator + ), + "package_continuation": dispatch.continuation_prompt_from_package( + context + ), + "package_native_continuation": ( + dispatch.continuation_prompt_from_package( + context, native_resume=True + ) + ), + } + concise_selfcheck_prompts = { + "selfcheck", + "selfcheck_unchecked", + "pi_selfcheck_continuation", + "pi_selfcheck_unchecked_continuation", + "pi_selfcheck_native_continuation", + } + for name, prompt in prompts.items(): + with self.subTest(prompt=name): + self.assertTrue( + prompt.startswith( + dispatch.SELF_CHECK_PROMPT_PREFIX + if name in concise_selfcheck_prompts + else dispatch.DISPATCHER_CHILD_BOUNDARY_PROMPT + ) + ) + if name in concise_selfcheck_prompts: + self.assertIn("Keep files in English.", prompt) + self.assertTrue( + prompt.startswith( + "Think in English. Final in Korean." + ) + ) + else: + self.assertIn( + "Keep artifact content in English.", prompt + ) + self.assertIn("Final in Korean.", prompt) + + self.assertIn( + "`AGENT_TASK_EXECUTION_ID` is present", + orchestrator_skill, + ) + self.assertIn( + dispatch.DISPATCHER_CHILD_BOUNDARY_PROMPT, + orchestrator_skill, + ) + self.assertIn( + dispatch.SELF_CHECK_PROMPT_PREFIX, + orchestrator_skill, + ) + self.assertIn( + "You may run dispatch.py --validate-plan only when required by " + "plan or code-review finalization", + dispatch.DISPATCHER_CHILD_BOUNDARY_PROMPT, + ) + self.assertNotIn( + "Do not invoke, monitor, or wait for dispatch.py", + dispatch.DISPATCHER_CHILD_BOUNDARY_PROMPT, + ) + +class ParallelLimitSchedulingTest(unittest.IsolatedAsyncioTestCase): + """Deterministic regressions for the workspace-global --max-parallel cap.""" + + def setUp(self) -> None: + super().setUp() + self._provider_deny = mock.patch.object( + subprocess, + "Popen", + side_effect=AssertionError( + "real subprocess execution forbidden in parallel limit tests" + ), + ) + self._build_command_deny = mock.patch.object( + dispatch, + "build_command", + side_effect=AssertionError( + "build_command must not be called in parallel limit tests" + ), + ) + self._provider_deny.start() + self._build_command_deny.start() + + def tearDown(self) -> None: + self._provider_deny.stop() + self._build_command_deny.stop() + super().tearDown() + + def _make_workspace( + self, group_name: str = "sim", count: int = 4, workspace: Path | None = None + ) -> tuple[Path, list[dispatch.Task]]: + if workspace is None: + workspace = Path(tempfile.mkdtemp()) + (workspace / ".git").mkdir() + task_dir = workspace / "agent-task" / group_name + task_dir.mkdir(parents=True, exist_ok=True) + tasks: list[dispatch.Task] = [] + for index in range(count): + sub_name = f"{index+1:02d}_task_{index}" + directory = task_dir / sub_name + directory.mkdir() + target = (workspace / "src" / f"{group_name}_{sub_name}.py").resolve() + target.parent.mkdir(exist_ok=True) + target.write_text("", encoding="utf-8") + plan = directory / "PLAN-local-G05.md" + review = directory / "CODE_REVIEW-local-G05.md" + plan.write_text( + f"\n" + "## Modified Files Summary\n\n" + "| File | Item |\n|---|---|\n" + f"| `{target}` | PLIM-{index} |\n", + encoding="utf-8", + ) + review.write_text( + f"\n", + encoding="utf-8", + ) + task = dispatch.Task( + name=f"{group_name}/{sub_name}", + directory=directory, + plan=plan, + review=review, + user_review=None, + recovery=False, + index=index + 1, + write_set={str(target)}, + write_set_known=True, + plan_hash=f"hash-{index}", + ) + tasks.append(task) + return workspace, tasks + + def test_omitted_cli_value_defaults_to_three(self): + """Omitting --max-parallel applies the workspace-global default of three.""" + with mock.patch("sys.argv", ["dispatch.py"]): + args = dispatch.parse_args() + self.assertEqual(dispatch.DEFAULT_MAX_PARALLEL, 3) + self.assertEqual(args.max_parallel, dispatch.DEFAULT_MAX_PARALLEL) + + def test_explicit_zero_selects_all_disjoint_ready(self): + """Explicit max_parallel=0 preserves the unlimited override.""" + workspace, tasks = self._make_workspace() + ready = [ + (tasks[0], "review"), + (tasks[1], "review"), + (tasks[2], "worker"), + ] + store = dispatch.StateStore(workspace) + try: + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, ready, persist=False, available_slots=None, + ) + finally: + store.close() + self.assertEqual(selected, ready) + self.assertEqual(deferred, []) + + def test_limit_two_selects_reviews_before_worker_and_caps_total(self): + """limit=2 selects reviews first; concurrent attempts never exceed cap.""" + workspace, tasks = self._make_workspace("sim", 4) + tasks[0].review.write_text( + f"\n" + "## Code Review Result\n\n" + "Overall Verdict: FAIL\n", + encoding="utf-8", + ) + tasks[1].review.write_text( + f"\n" + "## Code Review Result\n\n" + "Overall Verdict: FAIL\n", + encoding="utf-8", + ) + store = dispatch.StateStore(workspace) + task3_snapshot = dispatch.read_task_directory(workspace, tasks[3].directory) + store.update_task( + task3_snapshot, + worker_done=True, + worker_cli="pi", + worker_model="laguna-s:2.1", + execution_class="local_model", + completing_decision={ + "work_unit_id": dispatch.work_unit_id_from_file(task3_snapshot.plan), + "stage": "worker", + "selected": { + "adapter": "pi", + "target": "iop/laguna-s:2.1", + "execution_class": "local_model", + "selfcheck_required": True, + }, + }, + ) + + args = SimpleNamespace( + workspace=str(workspace), + task_group="sim", + dry_run=False, + retry_blocked=False, + max_parallel=2, + ) + active: set[str] = set() + peak: int = 0 + role_starts: list[tuple[str, str]] = [] + release = asyncio.Event() + + async def fake_role(role_name: str, workspace_path, store_arg, task_arg, *a, **kw): + nonlocal peak + role_starts.append((task_arg.name, role_name)) + active.add(task_arg.name) + peak = max(peak, len(active)) + if len(active) == 2: + release.set() + await release.wait() + self.assertIn(task_arg.name, store_arg.write_claim_snapshot()) + active.remove(task_arg.name) + store_arg.update_task(task_arg, blocked=f"{role_name} done") + return None + + try: + with ( + mock.patch.object( + dispatch, + "run_review", + new=lambda w, s, t, **kw: fake_role("review", w, s, t, **kw), + ), + mock.patch.object( + dispatch, + "run_worker", + new=lambda w, s, t, **kw: fake_role("worker", w, s, t, **kw), + ), + mock.patch.object( + dispatch, + "run_selfcheck", + new=lambda w, s, t, **kw: fake_role("selfcheck", w, s, t, **kw), + ), + mock.patch.object(dispatch, "ensure_review_shared_state"), + ): + result = asyncio.run( + dispatch.dispatch_with_store(args, workspace, store) + ) + self.assertLessEqual(peak, 2) + self.assertEqual(len(role_starts), 4) + self.assertEqual([role for _, role in role_starts[:2]], ["review", "review"]) + self.assertEqual(set(role for _, role in role_starts[2:]), {"worker", "selfcheck"}) + finally: + store.close() + + def test_limit_one_serializes_and_re_admits_capacity_waiter(self): + """limit=1 admits one task; stage transition without complete.log re-admits waiter from cache.""" + workspace, tasks = self._make_workspace("sim", 2) + store = dispatch.StateStore(workspace) + args = SimpleNamespace( + workspace=str(workspace), + task_group="sim", + dry_run=False, + retry_blocked=False, + max_parallel=1, + ) + worker_calls: list[str] = [] + + async def fake_worker(workspace_path, store_arg, task_arg, *a, **kw): + worker_calls.append(task_arg.name) + self.assertIn(task_arg.name, store_arg.write_claim_snapshot()) + store_arg.update_task(task_arg, blocked="stage done") + return None + + try: + with ( + mock.patch.object( + dispatch, "scan_tasks", wraps=dispatch.scan_tasks + ) as mock_scan, + mock.patch.object(dispatch, "run_worker", new=fake_worker), + mock.patch.object(dispatch, "ensure_review_shared_state"), + ): + result = asyncio.run( + dispatch.dispatch_with_store(args, workspace, store) + ) + self.assertEqual(worker_calls, ["sim/01_task_0", "sim/02_task_1"]) + self.assertEqual(mock_scan.call_count, 1) + finally: + store.close() + + def test_capacity_deferred_does_not_acquire_claim(self): + """A newly capacity-deferred task gets no claim.""" + workspace, tasks = self._make_workspace() + ready = [ + (tasks[0], "review"), + (tasks[1], "review"), + (tasks[2], "worker"), + ] + store = dispatch.StateStore(workspace) + try: + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, ready, persist=True, available_slots=1, + ) + finally: + store.close() + self.assertEqual(len(selected), 1) + selected_name = selected[0][0].name + self.assertIn( + selected_name, + store.data.get("write_claims", {}), + ) + for task, stage, reason in deferred: + self.assertTrue( + reason.startswith("capacity waiting:"), + f"expected capacity waiting, got: {reason}", + ) + self.assertNotIn( + task.name, + store.data.get("write_claims", {}), + f"capacity-deferred task {task.name} must not acquire a claim", + ) + + def test_existing_lifecycle_owner_retains_claim_while_waiting(self): + """A task that already owns its lifecycle claim keeps it while capacity-deferred.""" + workspace, tasks = self._make_workspace() + store = dispatch.StateStore(workspace) + try: + selected_preseed, _, _ = dispatch.select_dispatch_candidates( + store, [(tasks[1], "review")], persist=True, available_slots=1, + ) + self.assertEqual(len(selected_preseed), 1) + self.assertEqual(selected_preseed[0][0].name, "sim/02_task_1") + prior_claim = copy.deepcopy(store.write_claim_snapshot()["sim/02_task_1"]) + + selected, deferred, _ = dispatch.select_dispatch_candidates( + store, [(tasks[0], "review"), (tasks[1], "review")], persist=True, available_slots=1, + ) + self.assertEqual(len(selected), 1) + self.assertEqual(selected[0][0].name, "sim/01_task_0") + self.assertEqual(len(deferred), 1) + self.assertEqual(deferred[0][0].name, "sim/02_task_1") + self.assertTrue(deferred[0][2].startswith("capacity waiting:")) + + current_claim = store.write_claim_snapshot()["sim/02_task_1"] + self.assertEqual(current_claim, prior_claim) + finally: + store.close() + + def test_dry_run_applies_cap_and_leaves_state_unchanged(self): + """Dry-run applies the cap using global occupancy without persisting dispatcher state.""" + workspace, tasks_g1 = self._make_workspace("g1", 1) + _, tasks_g2 = self._make_workspace("g2", 1, workspace=workspace) + + store = dispatch.StateStore(workspace) + runs_dir = store.runs / "g1" / "01_task_0" + runs_dir.mkdir(parents=True, exist_ok=True) + locator_path = runs_dir / "locator.json" + locator_path.write_text( + json.dumps({ + "agent_pid": os.getpid(), + "workspace": str(workspace.resolve()), + "workspace_id": store.workspace_id, + "task": "g1/01_task_0", + }), + encoding="utf-8", + ) + store.data.setdefault("tasks", {})["g1/01_task_0"] = { + "active_locator": str(locator_path), + "active_stage": "worker", + } + store.save() + + args = SimpleNamespace( + workspace=str(workspace), + task_group="g2", + dry_run=True, + retry_blocked=False, + max_parallel=1, + ) + store_data_before = copy.deepcopy(store.data) + runner_calls: list[int] = [] + + async def fake_runner(*a, **kw): + runner_calls.append(1) + return None + + try: + with ( + mock.patch.object(dispatch, "ensure_review_shared_state"), + mock.patch.object(dispatch, "run_worker", new=fake_runner), + mock.patch.object(dispatch, "run_review", new=fake_runner), + mock.patch.object(dispatch, "run_selfcheck", new=fake_runner), + ): + result = asyncio.run( + dispatch.dispatch_with_store(args, workspace, store) + ) + self.assertEqual(result, 2) + self.assertEqual(runner_calls, []) + self.assertEqual(store.data, store_data_before) + finally: + store.close() + + def test_cross_group_occupancy_evaluates_workspace_state(self): + """Verified external-active task from another task group consumes capacity.""" + workspace, tasks_g1 = self._make_workspace("g1", 1) + _, tasks_g2 = self._make_workspace("g2", 1, workspace=workspace) + + store = dispatch.StateStore(workspace) + runs_dir = store.runs / "g1" / "01_task_0" + runs_dir.mkdir(parents=True, exist_ok=True) + locator_path = runs_dir / "locator.json" + locator_path.write_text( + json.dumps({ + "agent_pid": os.getpid(), + "workspace": str(workspace.resolve()), + "workspace_id": store.workspace_id, + "task": "g1/01_task_0", + }), + encoding="utf-8", + ) + store.data.setdefault("tasks", {})["g1/01_task_0"] = { + "active_locator": str(locator_path), + "active_stage": "worker", + } + store.save() + + args = SimpleNamespace( + workspace=str(workspace), + task_group="g2", + dry_run=False, + retry_blocked=False, + max_parallel=1, + ) + worker_called = [] + try: + with ( + mock.patch.object(dispatch, "ensure_review_shared_state"), + mock.patch.object(dispatch, "run_worker", side_effect=lambda *a, **kw: worker_called.append(1)), + ): + result = asyncio.run( + dispatch.dispatch_with_store(args, workspace, store) + ) + self.assertEqual(result, 3) + self.assertEqual(worker_called, []) + finally: + store.close() + + def test_capped_review_preflight_blocks_reviews_fills_with_worker(self): + """Failed review preflight blocks reviews but refills with disjoint worker.""" + workspace, tasks = self._make_workspace("sim", 4) + store = dispatch.StateStore(workspace) + # Put tasks 0, 1, 2 into review stage via recovery state with code_review_local_G05_0.log + for task in tasks[:3]: + task.review.write_text( + f"\n" + "# Code Review Result\n\n" + "## Code Review Result\n\n" + "- Overall Verdict: PASS\n", + encoding="utf-8", + ) + + args = SimpleNamespace( + workspace=str(workspace), + task_group="sim", + dry_run=False, + retry_blocked=False, + max_parallel=2, + ) + worker_called: list[str] = [] + review_called: list[str] = [] + worker_had_claim: list[bool] = [] + + async def fake_worker(workspace_path, store_arg, task_arg, *a, **kw): + worker_called.append(task_arg.name) + has_claim = task_arg.name in store_arg.write_claim_snapshot() + worker_had_claim.append(has_claim) + store_arg.update_task(task_arg, worker_done="completed") + completed_archive = workspace_path / "completed-task-refill" + completed_archive.mkdir(exist_ok=True) + (completed_archive / "complete.log").write_text("completed\n", encoding="utf-8") + task_arg.directory.joinpath("complete.log").write_text("completed\n", encoding="utf-8") + return str(completed_archive) + + async def fake_review(workspace_path, store_arg, task_arg, *a, **kw): + review_called.append(task_arg.name) + return None + + try: + with ( + mock.patch.object(dispatch, "run_worker", new=fake_worker), + mock.patch.object(dispatch, "run_review", new=fake_review), + mock.patch.object( + dispatch, "ensure_review_shared_state", + side_effect=RuntimeError("gitignore helper missing"), + ), + ): + result = asyncio.run( + dispatch.dispatch_with_store(args, workspace, store) + ) + self.assertEqual(review_called, []) + self.assertEqual(worker_called, ["sim/04_task_3"]) + self.assertEqual(worker_had_claim, [True]) + + claims = store.write_claim_snapshot() + self.assertIn("sim/01_task_0", claims) + self.assertIn("sim/02_task_1", claims) + self.assertNotIn("sim/03_task_2", claims) + finally: + store.close() + + def test_negative_value_rejected_at_cli_boundary(self): + """Negative --max-parallel is rejected with exit code 2.""" + with mock.patch("sys.argv", ["dispatch.py", "--max-parallel", "-1"]): + self.assertEqual(dispatch.main(), 2) + + def test_non_integer_value_rejected_at_cli_boundary(self): + """Non-integer --max-parallel is rejected at CLI boundary.""" + with mock.patch("sys.argv", ["dispatch.py", "--max-parallel", "abc"]): + with self.assertRaises(SystemExit) as cm: + dispatch.main() + self.assertEqual(cm.exception.code, 2) + + def test_valid_values_returned(self): + """Valid non-negative integers pass through.""" + self.assertEqual(dispatch.validated_max_parallel(0), 0) + self.assertEqual(dispatch.validated_max_parallel(1), 1) + self.assertEqual(dispatch.validated_max_parallel(100), 100) + + def test_provider_subprocess_not_invoked(self): + """No real provider subprocess should be invoked during tests.""" + invoked = {"called": False} + + def deny_subprocess(*args, **kwargs): + invoked["called"] = True + raise RuntimeError("real subprocess must not be invoked in tests") + + workspace, tasks = self._make_workspace() + args = SimpleNamespace( + workspace=str(workspace), + task_group="sim", + dry_run=False, + retry_blocked=False, + max_parallel=2, + ) + store = dispatch.StateStore(workspace) + + async def fake_worker(workspace_path, store_arg, task_arg, *args, **kwargs): + completed_archive = workspace_path / "completed-task" + completed_archive.mkdir(exist_ok=True) + (completed_archive / "complete.log").write_text("completed\n", encoding="utf-8") + task_arg.directory.joinpath("complete.log").write_text("completed\n", encoding="utf-8") + return str(completed_archive) + + try: + with ( + mock.patch.object( + dispatch, "scan_tasks", + side_effect=[[tasks[0]], []], + ), + mock.patch.object(dispatch, "run_worker", new=fake_worker), + mock.patch.object(dispatch, "ensure_review_shared_state"), + mock.patch.object(subprocess, "run", new=deny_subprocess), + ): + result = asyncio.run( + dispatch.dispatch_with_store(args, workspace, store) + ) + self.assertFalse( + invoked["called"], + "real subprocess.run must not be invoked", + ) + finally: + store.close() + +if __name__ == "__main__": + unittest.main() diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_dispatcher_observation.py b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_dispatcher_observation.py new file mode 100644 index 0000000..066c908 --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_dispatcher_observation.py @@ -0,0 +1,294 @@ +import ast +import asyncio +import importlib.util +import io +import json +import os +import re +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +SCRIPT = Path(__file__).parents[1] / "scripts" / "dispatch.py" +loaded = sys.modules.get("agent_task_dispatch") +if loaded is not None: + dispatch = loaded +else: + SPEC = importlib.util.spec_from_file_location("agent_task_dispatch", SCRIPT) + assert SPEC and SPEC.loader + dispatch = importlib.util.module_from_spec(SPEC) + sys.modules[SPEC.name] = dispatch + SPEC.loader.exec_module(dispatch) + + +def make_test_task(root: Path) -> dispatch.Task: + plan = root / "PLAN-local-G05.md" + review = root / "CODE_REVIEW-local-G05.md" + plan.write_text("\n", encoding="utf-8") + review.write_text("\n", encoding="utf-8") + return dispatch.Task( + name="test", + directory=root, + plan=plan, + review=review, + user_review=None, + recovery=False, + lane="local", + grade=5, + ) + + +class ObservationOutputTest(unittest.TestCase): + def test_banner_preserves_existing_format_and_nested_task_identity(self): + buffer = io.StringIO() + with mock.patch("sys.stdout", buffer): + dispatch.banner("START", "group/subtask/task_name", ["line 1", "line 2"]) + output = buffer.getvalue() + expected = ( + "------------------------------------------\n" + "START: task_name\n" + "------------------------------------------\n" + "task=group/subtask/task_name\n" + "line 1\n" + "line 2\n" + ) + self.assertEqual(output, expected) + + buffer_flat = io.StringIO() + with mock.patch("sys.stdout", buffer_flat): + dispatch.banner("START", "task_name") + output_flat = buffer_flat.getvalue() + expected_flat = ( + "------------------------------------------\n" + "START: task_name\n" + "------------------------------------------\n" + ) + self.assertEqual(output_flat, expected_flat) + + def test_attempt_event_is_one_flushed_stdout_line(self): + buffer = io.StringIO() + with mock.patch("sys.stdout", buffer): + dispatch.attempt_event("[test-prefix]", "event message detail") + output = buffer.getvalue() + self.assertEqual(output, "[test-prefix] event message detail\n") + + def test_dispatch_compatibility_aliases_point_to_observation_module(self): + self.assertEqual(dispatch.SEP, dispatch.observation.SEP) + self.assertIs(dispatch.banner, dispatch.observation.banner) + self.assertIs(dispatch.attempt_event, dispatch.observation.attempt_event) + + def test_observation_module_identity_is_reused(self): + module1 = dispatch.load_sibling_observation_module() + module2 = dispatch.load_sibling_observation_module() + self.assertIs(module1, module2) + self.assertIs(module1, sys.modules["agent_task_dispatcher_observation"]) + + def test_dispatch_has_no_direct_stdout_print_calls(self): + source = SCRIPT.read_text(encoding="utf-8") + tree = ast.parse(source, filename=str(SCRIPT)) + stdout_prints = [] + for node in ast.walk(tree): + if isinstance(node, ast.Call): + func = node.func + if isinstance(func, ast.Name) and func.id == "print": + is_stderr = False + for kw in node.keywords: + if kw.arg == "file": + val = kw.value + if ( + isinstance(val, ast.Attribute) + and isinstance(val.value, ast.Name) + and val.value.id == "sys" + and val.attr == "stderr" + ): + is_stderr = True + break + if not is_stderr: + stdout_prints.append(node.lineno) + self.assertEqual( + stdout_prints, + [], + f"found direct stdout print() calls on lines: {stdout_prints}", + ) + + +class ObservationInvokeIntegrationTest(unittest.IsolatedAsyncioTestCase): + async def test_heartbeat_and_child_output_stay_in_logs_not_user_event_stream(self): + with tempfile.TemporaryDirectory() as temporary: + workspace = Path(temporary) + (workspace / ".git").mkdir() + task = make_test_task(workspace) + store = dispatch.StateStore(workspace) + session_id = "11111111-1111-1111-1111-111111111111" + + def command_for( + spec, + prompt, + cwd, + actual_session_id, + attempt_dir, + pi_resume_session=None, + ): + self.assertEqual(actual_session_id, session_id) + native = attempt_dir / "pi-sessions" / f"session_{session_id}.jsonl" + child = ( + "from pathlib import Path\n" + "import sys,time\n" + "path = Path(sys.argv[1])\n" + "path.parent.mkdir(parents=True, exist_ok=True)\n" + "path.write_text(" + "'{\"type\":\"session\",\"version\":3,\"id\":\"test\"," + "\"timestamp\":\"2026-07-25T00:00:00.000Z\"," + "\"cwd\":\"/tmp/test\"}\\n', encoding='utf-8')\n" + "time.sleep(0.05)\n" + "print('done', flush=True)\n" + ) + return [sys.executable, "-c", child, str(native)] + + spec = dispatch.AgentSpec("pi", "ornith:35b", "pi", local_pi=True) + try: + with ( + mock.patch.object(dispatch, "build_command", side_effect=command_for), + mock.patch.object(dispatch.uuid, "uuid4", return_value=session_id), + mock.patch.object(dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01), + mock.patch("builtins.print") as print_mock, + ): + rc, failure, locator = await dispatch.invoke( + workspace, store, task, "review", spec, "Reply briefly." + ) + finally: + store.close() + + self.assertEqual(rc, 0) + self.assertIsNone(failure) + record = json.loads(locator.read_text(encoding="utf-8")) + self.assertTrue(record["native_session_path"].endswith(f"{session_id}.jsonl")) + self.assertIsInstance(record["native_session_mtime_ns"], int) + heartbeat = Path(record["heartbeat_log"]).read_text(encoding="utf-8") + self.assertIn("[heartbeat] ์ž‘์—…์ค‘...", heartbeat) + self.assertIn("native_session=", heartbeat) + self.assertIn("native_mtime_ns=", heartbeat) + stream = Path(record["stream_log"]).read_text(encoding="utf-8") + self.assertIn("[stdout] done", stream) + self.assertNotIn("[heartbeat]", stream) + normalized = Path(record["normalized_output_log"]).read_text( + encoding="utf-8" + ) + self.assertIn("done", normalized) + visible_output = "\n".join( + " ".join(str(value) for value in call.args) + for call in print_mock.call_args_list + ) + self.assertIn("locator=", visible_output) + self.assertNotIn("์ž‘์—…์ค‘...", visible_output) + self.assertNotIn("done", visible_output) + + +class SkillObservationContractTest(unittest.TestCase): + def test_dispatcher_owns_observation_and_caller_wakes_only_for_attention(self): + skill = ( + Path(__file__).parents[1] / "SKILL.md" + ).read_text(encoding="utf-8") + self.assertIn( + "dispatcher as the execution lifecycle and observation owner", + skill, + ) + self.assertIn( + "without caller-LLM supervision", + skill, + ) + self.assertIn( + "The caller never monitors", + skill, + ) + self.assertIn( + "Wake the caller LLM only for an attention event that the dispatcher cannot resolve autonomously", + skill, + ) + self.assertIn( + "Exit code `3` is a non-terminal tracking state, including another dispatcher workspace lock, " + "a live external agent, or an unexpected dispatcher interruption", + skill, + ) + self.assertIn( + "every CLI's health/progress primarily from actual stdout/stderr in `stream.log`, " + "plus native session events when available", + skill, + ) + self.assertIn( + "dispatcher PID, agent PID, each process start token, and the per-attempt " + "process environment marker", + skill, + ) + self.assertIn( + "use only an actual terminal error or confirmed process exit as recovery " + "evidence for every model", + skill, + ) + self.assertIn( + "every `toolCall.id` in the preceding assistant event matches a later `toolResult.toolCallId`", + skill, + ) + self.assertIn( + "stream stops for three minutes outside tool execution", + skill, + ) + self.assertIn( + "locator lacks an agent PID during this interval, never classify it as stale or " + "duplicate recovery based on log age", + skill, + ) + self.assertIn( + "original exception is a persistent-state error, do not convert it to exit `2` if any agent was running", + skill, + ) + self.assertIn( + "do not return successful exit `0` while any attempt directory remains", + skill, + ) + self.assertIn( + "share a budget of 10 consecutive automatic recovery failures for the same task stage", + skill, + ) + self.assertIn( + "On the 10th failure, block that task and do not auto-resume after cooldown", + skill, + ) + self.assertIn( + "legacy locator `session-stall` as a record of an earlier dispatcher timeout policy, not as provider failure", + skill, + ) + self.assertIn( + "Never classify exit code `143` as provider failure without actual provider terminal evidence", + skill, + ) + self.assertIn( + "Do not generalize one `pi -p` fresh/isolated session attempt to a Pi TUI or system-wide provider outage", + skill, + ) + self.assertIn("provider_transport_failure_confirmed", skill) + self.assertIn( + "Do not infer provider failure from `connection refused`, `dial tcp`, or `curl` peer failure in ordinary tool/test stderr", + skill, + ) + self.assertIn( + "A running Python dispatcher does not hot-reload source edits", + skill, + ) + self.assertIn("dispatcher_source_sha256", skill) + self.assertIn("`dispatcher_source_matches_loaded=false`", skill) + self.assertIn( + "KST-night `local-G07`โ€“`local-G08` Laguna locator `context-limit`/`session-stall`", + skill, + ) + self.assertIn( + "fresh session and `์„ธ์…˜์‘๋‹ต๋ณต๊ตฌ์žฌ์‹œ๋„` only for other legacy Pi `session-stall` recovery", + skill, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_execution_target_policy.py b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_execution_target_policy.py new file mode 100644 index 0000000..cefa9ea --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_execution_target_policy.py @@ -0,0 +1,249 @@ +import importlib.util +import sys +import unittest +from unittest import mock +from datetime import datetime, timezone +from pathlib import Path + + +SCRIPT = ( + Path(__file__).resolve().parents[1] + / "scripts" + / "execution_target_policy.py" +) +SPEC = importlib.util.spec_from_file_location("execution_target_policy", SCRIPT) +policy = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +sys.modules[SPEC.name] = policy +SPEC.loader.exec_module(policy) + + +def at_utc(hour: int, minute: int = 0, second: int = 0) -> datetime: + return datetime(2026, 7, 24, hour, minute, second, tzinfo=timezone.utc) + + +class ExecutionTargetPolicyTests(unittest.TestCase): + def test_local_g07_route_uses_kst_boundaries(self): + cases = [ + (at_utc(21, 59, 59), "pi", "iop/laguna-s:2.1", "kst-night-[23:00,07:00)"), + (at_utc(22, 0, 0), "agy", "Gemini 3.6 Flash (Medium)", "kst-day-[07:00,23:00)"), + (at_utc(13, 59, 59), "agy", "Gemini 3.6 Flash (Medium)", "kst-day-[07:00,23:00)"), + (at_utc(14, 0, 0), "pi", "iop/laguna-s:2.1", "kst-night-[23:00,07:00)"), + ] + for evaluated_at, adapter, target, time_window in cases: + with self.subTest(evaluated_at=evaluated_at): + decision = policy.select_policy( + stage="worker", + lane="local", + grade=7, + evaluated_at=evaluated_at, + ) + self.assertEqual(decision.candidates[0].adapter, adapter) + self.assertEqual(decision.candidates[0].target, target) + self.assertEqual(decision.time_window, time_window) + + def test_policy_is_unaffected_by_process_environment_variables(self): + night_time = datetime(2026, 7, 25, 17, 0, tzinfo=timezone.utc) # 02:00 KST + with mock.patch.dict("os.environ", {"OTHER_UNRELATED_ENV": "2026-07-26", "ANY_UNRELATED_ENV": "1"}): + decision = policy.select_policy( + stage="worker", lane="local", grade=8, evaluated_at=night_time + ) + self.assertEqual(decision.rule_id, "worker-local-g07-g08-kst-night") + self.assertEqual(decision.candidates, (policy.PI_LAGUNA, policy.AGY_GEMINI_MEDIUM)) + self.assertEqual(decision.time_window, "kst-night-[23:00,07:00)") + self.assertEqual(decision.candidates[0].target, "iop/laguna-s:2.1") + + def test_worker_grade_matrix_has_no_gaps(self): + daytime = at_utc(3) + expected = { + "local": { + **{ + grade: ("pi", "iop/ornith:35b", True) + for grade in range(1, 7) + }, + 7: ("agy", "Gemini 3.6 Flash (Medium)", False), + 8: ("agy", "Gemini 3.6 Flash (Medium)", False), + 9: ("claude", "claude-opus-4-8", False), + 10: ("claude", "claude-opus-4-8", False), + }, + "cloud": { + **{ + grade: ("codex", "gpt-5.3-codex-spark", False) + for grade in range(1, 3) + }, + **{ + grade: ("agy", "Gemini 3.6 Flash (Medium)", False) + for grade in range(3, 5) + }, + **{ + grade: ("agy", "Gemini 3.6 Flash (High)", False) + for grade in range(5, 7) + }, + 7: ("claude", "claude-opus-4-8", False), + 8: ("claude", "claude-opus-4-8", False), + 9: ("codex", "gpt-5.6-sol", False), + 10: ("codex", "gpt-5.6-sol", False), + }, + } + for lane, grades in expected.items(): + for grade, route in grades.items(): + with self.subTest(lane=lane, grade=grade): + selected = policy.select_policy( + stage="worker", + lane=lane, + grade=grade, + evaluated_at=daytime, + ).candidates[0] + self.assertEqual( + ( + selected.adapter, + selected.target, + selected.selfcheck_required, + ), + route, + ) + + def test_cloud_g01_g02_uses_ordered_spark_gemini_haiku_candidates(self): + for grade in (1, 2): + with self.subTest(grade=grade): + decision = policy.select_policy( + stage="worker", + lane="cloud", + grade=grade, + evaluated_at=at_utc(3), + ) + self.assertEqual( + decision.candidates, + ( + policy.CODEX_SPARK_XHIGH, + policy.AGY_GEMINI_LOW, + policy.CLAUDE_HAIKU_XHIGH, + ), + ) + self.assertEqual( + decision.reason_codes, + ("cloud_spark_priority_grade",), + ) + + def test_review_matrix_is_fixed_to_codex(self): + for lane in ("local", "cloud"): + for grade in range(1, 11): + with self.subTest(lane=lane, grade=grade): + decision = policy.select_policy( + stage="review", + lane=lane, + grade=grade, + evaluated_at=at_utc(3), + ) + self.assertEqual(decision.rule_id, "official-review-codex") + self.assertEqual(decision.candidates, (policy.CODEX_SOL_XHIGH,)) + + def test_local_g07_g08_candidate_order_uses_kst_boundaries(self): + daytime = policy.select_policy( + stage="worker", + lane="local", + grade=8, + evaluated_at=at_utc(3), + ) + nighttime = policy.select_policy( + stage="worker", + lane="local", + grade=8, + evaluated_at=at_utc(15), + ) + self.assertEqual( + [candidate.adapter for candidate in daytime.candidates], + ["agy", "pi"], + ) + self.assertEqual( + [candidate.adapter for candidate in nighttime.candidates], + ["pi", "agy"], + ) + + def test_invalid_inputs_are_rejected(self): + cases = [ + {"stage": "selfcheck", "lane": "local", "grade": 7}, + {"stage": "worker", "lane": "hybrid", "grade": 7}, + {"stage": "worker", "lane": "local", "grade": 0}, + {"stage": "worker", "lane": "local", "grade": 11}, + ] + for values in cases: + with self.subTest(values=values): + with self.assertRaises(ValueError): + policy.select_policy( + **values, + evaluated_at=at_utc(3), + ) + with self.assertRaisesRegex(ValueError, "timezone-aware"): + policy.select_policy( + stage="worker", + lane="local", + grade=7, + evaluated_at=datetime(2026, 7, 25, 12, 0, 0), + ) + + def test_cloud_promotion_matrix(self): + cases = [ + (policy.AGY_GEMINI_LOW, policy.CLAUDE_OPUS), + (policy.AGY_GEMINI_MEDIUM, policy.CLAUDE_OPUS), + (policy.AGY_GEMINI_HIGH, policy.CLAUDE_OPUS), + (policy.CLAUDE_OPUS, policy.CODEX_TERRA_HIGH), + (policy.CLAUDE_HAIKU_XHIGH, None), + (policy.CODEX_SPARK_XHIGH, None), + (policy.CODEX_SOL_XHIGH, None), + (policy.CODEX_TERRA_HIGH, None), + (policy.PI_ORNITH, None), + (policy.PI_LAGUNA, None), + ] + for current, expected in cases: + with self.subTest(current=current): + self.assertEqual(policy.promotion_target(current), expected) + + for target in policy.CANONICAL_TARGETS: + with self.subTest(identity=target.target): + self.assertEqual( + policy.canonical_target(target.adapter, target.target), + target, + ) + self.assertIsNone(policy.canonical_target("codex", "unknown")) + + def test_quota_probe_spec_matrix(self): + cases = [ + (policy.PI_ORNITH, None), + (policy.PI_LAGUNA, None), + ( + policy.AGY_GEMINI_LOW, + policy.QuotaProbeSpec("agy", "Gemini 3.6 Flash (Low)", ("overall", "model:Gemini 3.6 Flash (Low)")), + ), + ( + policy.AGY_GEMINI_MEDIUM, + policy.QuotaProbeSpec("agy", "Gemini 3.6 Flash (Medium)", ("overall", "model:Gemini 3.6 Flash (Medium)")), + ), + ( + policy.AGY_GEMINI_HIGH, + policy.QuotaProbeSpec("agy", "Gemini 3.6 Flash (High)", ("overall", "model:Gemini 3.6 Flash (High)")), + ), + ( + policy.CLAUDE_OPUS, + policy.QuotaProbeSpec("claude", "claude-opus-4-8", ("overall",)), + ), + ( + policy.CLAUDE_HAIKU_XHIGH, + policy.QuotaProbeSpec("claude", "claude-haiku-4-5", ("overall",)), + ), + ( + policy.CODEX_SPARK_XHIGH, + policy.QuotaProbeSpec("codex", "gpt-5.3-codex-spark", ("overall",)), + ), + ( + policy.CODEX_SOL_XHIGH, + policy.QuotaProbeSpec("codex", "gpt-5.6-sol", ("overall",)), + ), + ] + for target, expected in cases: + with self.subTest(target=target.target): + self.assertEqual(policy.quota_probe_spec(target), expected) + + +if __name__ == "__main__": + unittest.main() diff --git a/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_select_execution_target.py b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_select_execution_target.py new file mode 100644 index 0000000..91a1038 --- /dev/null +++ b/agent-ops/skills/common/orchestrate-agent-task-loop/tests/test_select_execution_target.py @@ -0,0 +1,1616 @@ +import copy +import importlib.util +import json +import subprocess +import sys +import unittest +from datetime import datetime +from pathlib import Path +from unittest import mock +from tempfile import TemporaryDirectory +from zoneinfo import ZoneInfo + + +SCRIPT = ( + Path(__file__).resolve().parents[1] + / "scripts" + / "select_execution_target.py" +) +SPEC = importlib.util.spec_from_file_location("select_execution_target", SCRIPT) +selector = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +sys.modules[SPEC.name] = selector +SPEC.loader.exec_module(selector) + +KST = ZoneInfo("Asia/Seoul") + + +def kst(hour: int, minute: int = 0, second: int = 0) -> datetime: + return datetime(2026, 7, 25, hour, minute, second, tzinfo=KST) + + +def go_quota_snapshot( + adapter: str, + target: str, + status: str, + *, + snapshot_id: str = "quota-snap-1", + checked_at: str = "2026-07-25T05:00:00Z", +) -> dict: + remaining = { + "available": 25.0, + "exhausted": 0.0, + "unknown": None, + }[status] + return { + "schema_version": "1.0", + "snapshot_id": snapshot_id, + "source": "iop-node quota-probe", + "checked_at": checked_at, + "targets": [ + {"adapter": adapter, "target": target, "status": status} + ], + "required_caps": [ + { + "name": "overall", + "status": status, + "remaining_percent": remaining, + } + ], + "reason_codes": ["cap_evidence_unknown"] if status == "unknown" else [], + } + + +def write_task_file( + directory: Path, + kind: str, + lane: str, + grade: int, + *, + task: str = "grp/01_unit", + plan: int = 0, + tag: str = "API", + body: str = "body\n", +) -> Path: + path = Path(directory) / f"{kind}-{lane}-G{grade:02d}.md" + path.write_text( + f"\n\n# title\n\n{body}", + encoding="utf-8", + ) + return path + + +_DELETE = object() + + +def _apply_path(prior: dict, path: tuple, value) -> None: + *parents, last = path + node = prior + for key in parents: + node = node[key] + if value is _DELETE: + del node[last] + else: + node[last] = value + + +# (name, path into a valid initial decision, replacement or _DELETE) triples that +# each leave the top-level containers well-typed but break one nested +# field/type/enum the resume path reuses verbatim. +MALFORMED_NESTED_VARIANTS = [ + ("empty_candidate", ("candidates", 0), {}), + ("candidate_missing_quota_mode", ("candidates", 0, "quota_mode"), _DELETE), + ("candidate_bad_eligibility_enum", ("candidates", 0, "eligibility"), "maybe"), + ("candidate_bad_selfcheck_type", ("candidates", 0, "selfcheck_required"), "yes"), + ("candidate_rank_not_consecutive", ("candidates", 0, "candidate_rank"), 5), + ("candidates_empty_list", ("candidates",), []), + ("decision_missing_rule_id", ("decision", "rule_id"), _DELETE), + ("decision_bad_time_window_enum", ("decision", "time_window"), "bogus"), + ("decision_wrong_timezone", ("decision", "timezone"), "UTC"), + ("decision_bad_pinned_type", ("decision", "pinned"), "yes"), + ("decision_reason_codes_scalar", ("decision", "reason_codes"), "kst_day_window"), + ("quota_missing_mode", ("quota", "mode"), _DELETE), + ("quota_bad_mode_enum", ("quota", "mode"), "bogus"), + ("quota_bad_status_enum", ("quota", "status"), "maybe"), + ("quota_bad_snapshot_id_type", ("quota", "snapshot_id"), 5), +] + + +class SelectorContractTests(unittest.TestCase): + def test_worker_contract_shape_and_types(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "cloud", 7) + result = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + self.assertEqual(result["schema_version"], "1.0") + self.assertEqual( + result["work_unit_id"], "grp/01_unit::plan-0::tag-API" + ) + self.assertEqual(result["stage"], "worker") + self.assertEqual(result["lane"], "cloud") + self.assertEqual(result["grade"], 7) + self.assertIsInstance(result["grade"], int) + self.assertEqual( + result["selected"], + { + "adapter": "claude", + "target": "claude-opus-4-8", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + ) + for key in ("rule_id", "policy_priority", "reason_codes", "pinned"): + self.assertIn(key, result["decision"]) + self.assertIs(result["decision"]["pinned"], False) + self.assertEqual(result["decision"]["timezone"], "Asia/Seoul") + self.assertEqual( + set(result["quota"]), + {"snapshot_id", "mode", "status", "source", "checked_at", "targets"}, + ) + self.assertEqual(result["transition"]["trigger"], "initial") + self.assertEqual(result["transition"]["context_transfer"], "none") + + def test_stage_inference_and_mismatch(self): + with TemporaryDirectory() as tmp: + plan_file = write_task_file(Path(tmp), "PLAN", "local", 5) + review_file = write_task_file(Path(tmp), "CODE_REVIEW", "local", 5) + self.assertEqual( + selector.select_execution_target( + plan_file, evaluated_at=kst(12) + )["stage"], + "worker", + ) + self.assertEqual( + selector.select_execution_target( + review_file, evaluated_at=kst(12) + )["stage"], + "review", + ) + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + plan_file, stage="review", evaluated_at=kst(12) + ) + self.assertEqual(ctx.exception.code, "stage_mismatch") + with self.assertRaises(selector.SelectorInputError): + selector.select_execution_target( + review_file, stage="worker", evaluated_at=kst(12) + ) + + def test_invalid_filenames_and_grades_rejected(self): + with TemporaryDirectory() as tmp: + root = Path(tmp) + bad_names = [ + "NOTE-cloud-G07.md", + "PLAN-hybrid-G07.md", + "PLAN-cloud-G7.md", + "PLAN-cloud-G07.txt", + "PLAN-cloud-G00.md", + "PLAN-cloud-G11.md", + ] + for name in bad_names: + path = root / name + path.write_text( + "\n", encoding="utf-8" + ) + with self.subTest(name=name): + with self.assertRaises(selector.SelectorInputError): + selector.select_execution_target( + path, evaluated_at=kst(12) + ) + + def test_malformed_header_rejected(self): + with TemporaryDirectory() as tmp: + path = Path(tmp) / "PLAN-cloud-G05.md" + path.write_text("# no generation header\n", encoding="utf-8") + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target(path, evaluated_at=kst(12)) + self.assertEqual(ctx.exception.code, "malformed_header") + + def test_work_unit_id_stable_across_body_changes(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file( + Path(tmp), "PLAN", "cloud", 7, body="first body\n" + ) + first = selector.select_execution_target( + task_file, evaluated_at=kst(12) + )["work_unit_id"] + task_file.write_text( + "\n\n# title\n\n" + "a much longer body with different content\n", + encoding="utf-8", + ) + second = selector.select_execution_target( + task_file, evaluated_at=kst(12) + )["work_unit_id"] + self.assertEqual(first, second) + # A new plan/tag generation must yield a new identity. + changed = write_task_file(Path(tmp), "PLAN", "cloud", 7, plan=1) + self.assertNotEqual( + first, + selector.select_execution_target( + changed, evaluated_at=kst(12) + )["work_unit_id"], + ) + + def test_deterministic_output_for_fixed_clock(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + first = selector.to_json( + selector.select_execution_target(task_file, evaluated_at=kst(12)) + ) + second = selector.to_json( + selector.select_execution_target(task_file, evaluated_at=kst(12)) + ) + self.assertEqual(first, second) + + def test_repeated_input_is_byte_stable(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "CODE_REVIEW", "cloud", 9) + runs = [ + subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + ], + capture_output=True, + check=True, + ) + for _ in range(2) + ] + self.assertEqual(runs[0].stdout, runs[1].stdout) + self.assertTrue(runs[0].stdout.strip()) + + def test_resume_pins_prior_target_across_time(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + daytime = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + self.assertEqual(daytime["selected"]["adapter"], "agy") + # A new night route changes target, but resume preserves the pin. + night_initial = selector.select_execution_target( + task_file, evaluated_at=kst(2) + ) + self.assertEqual(night_initial["selected"]["adapter"], "pi") + self.assertEqual(night_initial["selected"]["target"], "iop/laguna-s:2.1") + resumed = selector.select_execution_target( + task_file, + evaluated_at=kst(2), + transition="resume", + prior_decision=daytime, + ) + self.assertEqual(resumed["selected"], daytime["selected"]) + self.assertIs(resumed["decision"]["pinned"], True) + self.assertEqual(resumed["transition"]["trigger"], "resume") + self.assertEqual( + resumed["transition"]["previous_target"], + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + ) + + def test_resume_requires_matching_prior_decision(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, evaluated_at=kst(12), transition="resume" + ) + self.assertEqual( + ctx.exception.code, "resume_requires_prior_decision" + ) + other = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + other["work_unit_id"] = "grp/other::plan-0::tag-API" + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="resume", + prior_decision=other, + ) + self.assertEqual(ctx.exception.code, "resume_work_unit_mismatch") + + def test_failover_requires_qualified_failure_class(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, evaluated_at=kst(12), transition="failover" + ) + self.assertEqual(ctx.exception.code, "unqualified_failover_trigger") + + def test_cli_input_error_is_stderr_json_without_stdout(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + proc = subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + "--transition", + "failover", + ], + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 2) + self.assertEqual(proc.stdout, "") + self.assertEqual( + json.loads(proc.stderr)["error"], "unqualified_failover_trigger" + ) + + +class SelectorRouteMatrixTests(unittest.TestCase): + def test_local_g07_g08_use_kst_boundaries(self): + cases = [ + (kst(6, 59, 59), "pi", "iop/laguna-s:2.1"), + (kst(7, 0, 0), "agy", "Gemini 3.6 Flash (Medium)"), + (kst(22, 59, 59), "agy", "Gemini 3.6 Flash (Medium)"), + (kst(23, 0, 0), "pi", "iop/laguna-s:2.1"), + ] + with TemporaryDirectory() as tmp: + for grade in (7, 8): + task_file = write_task_file(Path(tmp), "PLAN", "local", grade) + for evaluated_at, adapter, target in cases: + with self.subTest(grade=grade, evaluated_at=evaluated_at): + result = selector.select_execution_target( + task_file, evaluated_at=evaluated_at + ) + self.assertEqual(result["selected"]["adapter"], adapter) + self.assertEqual(result["selected"]["target"], target) + + def test_worker_route_matrix_through_selector(self): + expected = { + "local": { + **{g: ("pi", "iop/ornith:35b", "local_model", True) + for g in range(1, 7)}, + 7: ("agy", "Gemini 3.6 Flash (Medium)", "cloud_model", False), + 8: ("agy", "Gemini 3.6 Flash (Medium)", "cloud_model", False), + 9: ("claude", "claude-opus-4-8", "cloud_model", False), + 10: ("claude", "claude-opus-4-8", "cloud_model", False), + }, + "cloud": { + 1: ("codex", "gpt-5.3-codex-spark", "cloud_model", False), + 2: ("codex", "gpt-5.3-codex-spark", "cloud_model", False), + 3: ("agy", "Gemini 3.6 Flash (Medium)", "cloud_model", False), + 4: ("agy", "Gemini 3.6 Flash (Medium)", "cloud_model", False), + 5: ("agy", "Gemini 3.6 Flash (High)", "cloud_model", False), + 6: ("agy", "Gemini 3.6 Flash (High)", "cloud_model", False), + 7: ("claude", "claude-opus-4-8", "cloud_model", False), + 8: ("claude", "claude-opus-4-8", "cloud_model", False), + 9: ("codex", "gpt-5.6-sol", "cloud_model", False), + 10: ("codex", "gpt-5.6-sol", "cloud_model", False), + }, + } + with TemporaryDirectory() as tmp: + for lane, grades in expected.items(): + for grade, route in grades.items(): + task_file = write_task_file( + Path(tmp), "PLAN", lane, grade + ) + with self.subTest(lane=lane, grade=grade): + sel = selector.select_execution_target( + task_file, evaluated_at=kst(12) + )["selected"] + self.assertEqual( + ( + sel["adapter"], + sel["target"], + sel["execution_class"], + sel["selfcheck_required"], + ), + route, + ) + + def test_review_route_matrix_is_codex(self): + with TemporaryDirectory() as tmp: + for lane in ("local", "cloud"): + for grade in range(1, 11): + task_file = write_task_file( + Path(tmp), "CODE_REVIEW", lane, grade + ) + with self.subTest(lane=lane, grade=grade): + sel = selector.select_execution_target( + task_file, evaluated_at=kst(12) + )["selected"] + self.assertEqual( + ( + sel["adapter"], + sel["target"], + sel["execution_class"], + sel["selfcheck_required"], + ), + ("codex", "gpt-5.6-sol", "cloud_model", False), + ) + + def test_candidate_rank_is_single_per_time_window(self): + with TemporaryDirectory() as tmp: + dynamic = write_task_file(Path(tmp), "PLAN", "local", 8) + daytime = selector.select_execution_target( + dynamic, evaluated_at=kst(12) + )["candidates"] + nighttime = selector.select_execution_target( + dynamic, evaluated_at=kst(2) + )["candidates"] + self.assertEqual( + [c["candidate_rank"] for c in daytime], [1, 2] + ) + self.assertEqual( + [c["adapter"] for c in daytime], ["agy", "pi"] + ) + self.assertEqual( + [c["adapter"] for c in nighttime], ["pi", "agy"] + ) + single = write_task_file(Path(tmp), "PLAN", "cloud", 5) + candidates = selector.select_execution_target( + single, evaluated_at=kst(12) + )["candidates"] + self.assertEqual([c["candidate_rank"] for c in candidates], [1]) + + +class SelectorQuotaRepresentationTests(unittest.TestCase): + def test_quota_probe_tri_state(self): + snapshots = { + "exhausted": "exhausted", + "available": "available", + "unknown": "unknown", + } + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 7) + for name, status in snapshots.items(): + with self.subTest(status=name): + if status == "exhausted": + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + cloud, + evaluated_at=kst(12), + quota_snapshot={ + "snapshot_id": f"probe-{name}", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + { + "adapter": "claude", + "target": "claude-opus-4-8", + "status": status, + } + ], + }, + ) + self.assertEqual(ctx.exception.code, "no_eligible_target") + continue + result = selector.select_execution_target( + cloud, + evaluated_at=kst(12), + quota_snapshot={ + "snapshot_id": f"probe-{name}", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + { + "adapter": "claude", + "target": "claude-opus-4-8", + "status": status, + } + ], + }, + ) + candidate = result["candidates"][0] + self.assertEqual(candidate["quota_status"], status) + self.assertEqual( + candidate["eligibility"], + "eligible", + ) + + def test_actual_go_snapshot_shape_preserves_tri_state_and_metadata(self): + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 7) + for status in ("available", "exhausted", "unknown"): + with self.subTest(status=status): + snapshot = go_quota_snapshot( + "claude", + "claude-opus-4-8", + status, + snapshot_id=f"quota-{status}", + ) + completed = mock.Mock( + returncode=0, stdout=json.dumps(snapshot) + ) + with mock.patch( + "subprocess.run", return_value=completed + ): + if status == "exhausted": + with self.assertRaises( + selector.SelectorInputError + ) as ctx: + selector.select_execution_target( + cloud, evaluated_at=kst(12) + ) + self.assertEqual( + ctx.exception.code, "no_eligible_target" + ) + continue + result = selector.select_execution_target( + cloud, evaluated_at=kst(12) + ) + self.assertEqual( + result["candidates"][0]["quota_status"], status + ) + self.assertEqual(result["quota"]["status"], status) + self.assertEqual( + result["quota"]["snapshot_id"], snapshot["snapshot_id"] + ) + self.assertEqual( + result["quota"]["source"], snapshot["source"] + ) + self.assertEqual( + result["quota"]["checked_at"], snapshot["checked_at"] + ) + self.assertEqual( + result["quota"]["targets"], snapshot["targets"] + ) + + def test_exhausted_gemini_falls_back_to_laguna(self): + snapshot = { + "snapshot_id": "gemini-exhausted", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + { + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + "status": "exhausted", + } + ], + } + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + result = selector.select_execution_target( + task_file, evaluated_at=kst(12), quota_snapshot=snapshot + ) + self.assertEqual(result["selected"]["adapter"], "pi") + self.assertEqual(result["selected"]["target"], "iop/laguna-s:2.1") + + def test_all_candidates_exhausted_returns_no_eligible_target(self): + snapshot = { + "snapshot_id": "opus-exhausted", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + { + "adapter": "claude", + "target": "claude-opus-4-8", + "status": "exhausted", + } + ], + } + with TemporaryDirectory() as tmp: + root = Path(tmp) + task_file = write_task_file(root, "PLAN", "cloud", 7) + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, evaluated_at=kst(12), quota_snapshot=snapshot + ) + self.assertEqual(ctx.exception.code, "no_eligible_target") + + snapshot_path = root / "quota.json" + snapshot_path.write_text(json.dumps(snapshot), encoding="utf-8") + proc = subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + "--quota-snapshot", + str(snapshot_path), + ], + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 2) + self.assertEqual(proc.stdout, "") + self.assertEqual(json.loads(proc.stderr)["error"], "no_eligible_target") + + def test_unknown_is_admitted_once_per_work_unit(self): + snapshot = { + "snapshot_id": "unknown-1", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + { + "adapter": "claude", + "target": "claude-opus-4-8", + "status": "unknown", + } + ], + } + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 7) + initial = selector.select_execution_target( + cloud, evaluated_at=kst(12), quota_snapshot=snapshot + ) + self.assertEqual(initial["candidates"][0]["eligibility"], "eligible") + # Resume consumes the persisted decision instead of evaluating a + # second unknown admission for the same task/plan/tag generation. + resumed = selector.select_execution_target( + cloud, + evaluated_at=kst(12), + transition="resume", + prior_decision=initial, + quota_snapshot={ + **snapshot, + "snapshot_id": "later-exhausted", + "targets": [ + { + "adapter": "claude", + "target": "claude-opus-4-8", + "status": "exhausted", + } + ], + }, + ) + self.assertEqual(resumed["quota"], initial["quota"]) + self.assertEqual(resumed["candidates"], initial["candidates"]) + + def test_local_route_does_not_call_probe(self): + with TemporaryDirectory() as tmp: + local = write_task_file(Path(tmp), "PLAN", "local", 3) + result = selector.select_execution_target( + local, + evaluated_at=kst(12), + quota_probe_command="probe must not be used for local", + ) + self.assertEqual(result["quota"]["mode"], "unbounded") + self.assertEqual(result["quota"]["status"], "not_applicable") + self.assertEqual(result["quota"]["source"], "local_unbounded") + + def test_generic_stderr_is_not_quota_evidence(self): + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 7) + result = selector.select_execution_target( + cloud, + evaluated_at=kst(12), + quota_probe_command="generic stderr: quota might be exhausted", + ) + self.assertEqual(result["quota"]["status"], "unknown") + self.assertEqual(result["candidates"][0]["eligibility"], "eligible") + + def test_quota_representation_without_snapshot(self): + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 7) + cloud_result = selector.select_execution_target( + cloud, evaluated_at=kst(12) + ) + self.assertEqual(cloud_result["quota"]["mode"], "bounded") + self.assertEqual(cloud_result["quota"]["status"], "unknown") + self.assertEqual( + cloud_result["quota"]["source"], + selector.DEFAULT_QUOTA_PROBE_COMMAND, + ) + + local = write_task_file(Path(tmp), "PLAN", "local", 3) + local_result = selector.select_execution_target( + local, evaluated_at=kst(12) + ) + self.assertEqual(local_result["quota"]["mode"], "unbounded") + self.assertEqual(local_result["quota"]["status"], "not_applicable") + + # Local G07 has Gemini primary candidate and Laguna fallback. + dynamic = write_task_file(Path(tmp), "PLAN", "local", 7) + candidates = selector.select_execution_target( + dynamic, evaluated_at=kst(12) + )["candidates"] + self.assertEqual(len(candidates), 2) + self.assertEqual(candidates[0]["adapter"], "agy") + self.assertEqual(candidates[0]["quota_status"], "unknown") + self.assertEqual(candidates[1]["adapter"], "pi") + + def test_injected_snapshot_is_reflected(self): + snapshot = { + "snapshot_id": "snap-1", + "source": "usage-checker", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (High)", "status": "available"} + ], + } + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 5) + result = selector.select_execution_target( + cloud, evaluated_at=kst(12), quota_snapshot=snapshot + ) + self.assertEqual(result["quota"]["status"], "available") + self.assertEqual(result["quota"]["snapshot_id"], "snap-1") + self.assertEqual(result["quota"]["source"], "usage-checker") + self.assertEqual( + result["candidates"][0]["quota_status"], "available" + ) + + +class SelectorNestedInputContractTests(unittest.TestCase): + def test_resume_rejects_incomplete_selected_schema(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + prior = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + # Reproduce the prior loop: a selected with only adapter/target must + # no longer flow through as a "successful" resume schema. + prior["selected"] = { + "adapter": prior["selected"]["adapter"], + "target": prior["selected"]["target"], + } + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="resume", + prior_decision=prior, + ) + self.assertEqual(ctx.exception.code, "malformed_prior_decision") + + def test_resume_rejects_malformed_nested_prior_schema_variants(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + base = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + # Sanity: the untouched decision resumes cleanly. + self.assertEqual( + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="resume", + prior_decision=copy.deepcopy(base), + )["selected"], + base["selected"], + ) + for name, path, value in MALFORMED_NESTED_VARIANTS: + with self.subTest(variant=name): + prior = copy.deepcopy(base) + _apply_path(prior, path, value) + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="resume", + prior_decision=prior, + ) + self.assertEqual( + ctx.exception.code, "malformed_prior_decision" + ) + + def test_cli_deeply_malformed_prior_uses_json_error_envelope(self): + with TemporaryDirectory() as tmp: + root = Path(tmp) + task_file = write_task_file(root, "PLAN", "local", 7) + prior = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + # Containers stay well-typed object/list; only a nested enum is bad. + prior["quota"]["mode"] = "bogus" + prior_path = root / "prior.json" + prior_path.write_text(json.dumps(prior), encoding="utf-8") + proc = subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + "--transition", + "resume", + "--prior-decision", + str(prior_path), + ], + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 2) + self.assertEqual(proc.stdout, "") + self.assertEqual( + json.loads(proc.stderr)["error"], "malformed_prior_decision" + ) + + def test_cli_malformed_prior_uses_json_error_envelope(self): + with TemporaryDirectory() as tmp: + root = Path(tmp) + task_file = write_task_file(root, "PLAN", "local", 7) + prior = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + # A scalar where a nested object is required must not reach a raw + # TypeError/AttributeError traceback. + prior["decision"] = 1 + prior_path = root / "prior.json" + prior_path.write_text(json.dumps(prior), encoding="utf-8") + proc = subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + "--transition", + "resume", + "--prior-decision", + str(prior_path), + ], + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 2) + self.assertEqual(proc.stdout, "") + self.assertEqual( + json.loads(proc.stderr)["error"], "malformed_prior_decision" + ) + + def test_cli_malformed_quota_uses_json_error_envelope(self): + with TemporaryDirectory() as tmp: + root = Path(tmp) + task_file = write_task_file(root, "PLAN", "cloud", 5) + cases = { + # A bare array instead of the snapshot object. + "array_snapshot": [ + { + "adapter": "claude", + "target": "sonnet", + "status": "available", + } + ], + # A target entry missing the required status field. + "invalid_target_entry": { + "targets": [{"adapter": "claude", "target": "sonnet"}] + }, + } + for name, snapshot in cases.items(): + quota_path = root / f"quota_{name}.json" + quota_path.write_text(json.dumps(snapshot), encoding="utf-8") + with self.subTest(case=name): + proc = subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + "--quota-snapshot", + str(quota_path), + ], + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 2) + self.assertEqual(proc.stdout, "") + self.assertEqual( + json.loads(proc.stderr)["error"], + "malformed_quota_snapshot", + ) + + +class SelectorIdentityAndQuotaRoundtripTests(unittest.TestCase): + _VALID_TARGETS = [ + {"adapter": "claude", "target": "sonnet", "status": "available"} + ] + + def test_resume_rejects_unhashable_stage_and_lane_types(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 7) + base = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + # list/dict identity values must be normalized to a stable selector + # error instead of leaking a raw unhashable-type TypeError/exit 1. + for field, unhashable in (("stage", []), ("lane", {})): + with self.subTest(field=field): + prior = copy.deepcopy(base) + prior[field] = unhashable + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="resume", + prior_decision=prior, + ) + self.assertEqual( + ctx.exception.code, "malformed_prior_decision" + ) + + def test_quota_metadata_is_validated_before_initial_output(self): + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 5) + snapshot_cases = { + "numeric_snapshot_id": { + "snapshot_id": 5, + "targets": self._VALID_TARGETS, + }, + "numeric_checked_at": { + "checked_at": 1690000000, + "targets": self._VALID_TARGETS, + }, + "array_source": { + "source": ["usage-checker"], + "targets": self._VALID_TARGETS, + }, + "empty_source": { + "source": "", + "targets": self._VALID_TARGETS, + }, + } + for name, snapshot in snapshot_cases.items(): + with self.subTest(case=name): + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + cloud, + evaluated_at=kst(12), + quota_snapshot=snapshot, + ) + self.assertEqual( + ctx.exception.code, "malformed_quota_snapshot" + ) + # An empty probe command would emit an empty quota.source that the + # resume validator rejects, so it must fail before any success JSON. + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + cloud, evaluated_at=kst(12), quota_probe_command="" + ) + self.assertEqual( + ctx.exception.code, "invalid_quota_probe_command" + ) + + def test_valid_quota_initial_output_resumes(self): + snapshots = { + "no_snapshot": None, + "targets_only": {"targets": copy.deepcopy(self._VALID_TARGETS)}, + "full_metadata": { + "snapshot_id": "snap-1", + "source": "usage-checker", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": copy.deepcopy(self._VALID_TARGETS), + }, + } + with TemporaryDirectory() as tmp: + cloud = write_task_file(Path(tmp), "PLAN", "cloud", 5) + for name, snapshot in snapshots.items(): + with self.subTest(case=name): + initial = selector.select_execution_target( + cloud, evaluated_at=kst(12), quota_snapshot=snapshot + ) + # A daytime initial must resume verbatim at night without + # being rejected by its own prior-decision validator. + resumed = selector.select_execution_target( + cloud, + evaluated_at=kst(2), + transition="resume", + prior_decision=copy.deepcopy(initial), + ) + self.assertEqual(resumed["selected"], initial["selected"]) + self.assertEqual(resumed["quota"], initial["quota"]) + self.assertIs(resumed["decision"]["pinned"], True) + + def test_cli_malformed_identity_and_quota_metadata_use_json_error_envelope( + self, + ): + with TemporaryDirectory() as tmp: + root = Path(tmp) + task_file = write_task_file(root, "PLAN", "cloud", 5) + prior = selector.select_execution_target( + task_file, evaluated_at=kst(12) + ) + prior["stage"] = [] # unhashable identity type + prior_path = root / "prior.json" + prior_path.write_text(json.dumps(prior), encoding="utf-8") + snapshot_path = root / "quota.json" + snapshot_path.write_text( + json.dumps( + { + "snapshot_id": 5, + "targets": [ + { + "adapter": "claude", + "target": "sonnet", + "status": "available", + } + ], + } + ), + encoding="utf-8", + ) + cases = [ + ( + [ + "--transition", + "resume", + "--prior-decision", + str(prior_path), + ], + "malformed_prior_decision", + ), + ( + ["--quota-snapshot", str(snapshot_path)], + "malformed_quota_snapshot", + ), + ( + ["--quota-probe-command", ""], + "invalid_quota_probe_command", + ), + ] + for extra, code in cases: + with self.subTest(error=code): + proc = subprocess.run( + [ + sys.executable, + str(SCRIPT), + str(task_file), + "--evaluated-at", + "2026-07-25T12:00:00+09:00", + *extra, + ], + capture_output=True, + text=True, + ) + self.assertEqual(proc.returncode, 2) + self.assertEqual(proc.stdout, "") + self.assertEqual(json.loads(proc.stderr)["error"], code) + + + +class SelectorFailoverContractTests(unittest.TestCase): + def test_cloud_g01_g02_quota_failover_follows_spark_gemini_haiku_order(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "cloud", 1) + initial = selector.select_execution_target( + task_file, + evaluated_at=kst(12), + quota_probe_command="missing-probe", + ) + gemini = selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="failover", + prior_decision=initial, + failure_class="provider-quota", + quota_probe_command="missing-probe", + ) + haiku = selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="failover", + prior_decision=gemini, + failure_class="provider-quota", + quota_probe_command="missing-probe", + ) + + self.assertEqual( + [ + (candidate["adapter"], candidate["target"]) + for candidate in initial["candidates"] + ], + [ + ("codex", "gpt-5.3-codex-spark"), + ("agy", "Gemini 3.6 Flash (Low)"), + ("claude", "claude-haiku-4-5"), + ], + ) + self.assertEqual( + (gemini["selected"]["adapter"], gemini["selected"]["target"]), + ("agy", "Gemini 3.6 Flash (Low)"), + ) + self.assertEqual( + (haiku["selected"]["adapter"], haiku["selected"]["target"]), + ("claude", "claude-haiku-4-5"), + ) + self.assertEqual( + haiku["used_candidates"], + [ + {"adapter": "codex", "target": "gpt-5.3-codex-spark"}, + {"adapter": "agy", "target": "Gemini 3.6 Flash (Low)"}, + {"adapter": "claude", "target": "claude-haiku-4-5"}, + ], + ) + with self.assertRaises(selector.SelectorInputError) as exhausted: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="failover", + prior_decision=haiku, + failure_class="provider-quota", + quota_probe_command="missing-probe", + ) + self.assertEqual(exhausted.exception.code, "no_failover_candidate") + + def test_qualified_failover_uses_only_unused_eligible_candidate(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + prior = selector.select_execution_target(task_file, evaluated_at=kst(12)) + self.assertEqual(prior["selected"]["adapter"], "agy") + result = selector.select_execution_target( + task_file, evaluated_at=kst(12), transition="failover", + prior_decision=prior, failure_class="provider-quota", + ) + self.assertEqual(result["selected"]["adapter"], "pi") + self.assertEqual(result["transition"]["context_transfer"], "logical") + self.assertEqual(result["transition"]["trigger"], "provider-quota") + self.assertEqual(len(result["used_candidates"]), 2) + failed = next(item for item in result["candidates"] if item["adapter"] == "agy") + self.assertEqual( + (failed["quota_status"], failed["eligibility"], failed["rejection_reason"]), + ("exhausted", "ineligible", "quota_exhausted"), + ) + + def test_generic_failure_and_exhausted_or_used_candidate_fail_closed(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + prior = selector.select_execution_target(task_file, evaluated_at=kst(12)) + with self.assertRaises(selector.SelectorInputError) as generic: + selector.select_execution_target(task_file, evaluated_at=kst(12), transition="failover", prior_decision=prior, failure_class="generic-error") + self.assertEqual(generic.exception.code, "unqualified_failover_trigger") + first = selector.select_execution_target( + task_file, evaluated_at=kst(12), transition="failover", + prior_decision=prior, failure_class="provider-quota", + ) + with self.assertRaises(selector.SelectorInputError) as exhausted: + selector.select_execution_target( + task_file, evaluated_at=kst(12), transition="failover", prior_decision=first, failure_class="provider-quota", + ) + self.assertEqual(exhausted.exception.code, "no_failover_candidate") + + def test_unknown_is_admitted_once_and_no_bounce_remains(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + prior = selector.select_execution_target(task_file, evaluated_at=kst(12)) + first = selector.select_execution_target(task_file, evaluated_at=kst(12), transition="failover", prior_decision=prior, failure_class="provider-stream-disconnect") + resumed = selector.select_execution_target(task_file, evaluated_at=kst(12), transition="resume", prior_decision=first) + self.assertEqual(resumed["used_candidates"], first["used_candidates"]) + with self.assertRaises(selector.SelectorInputError) as repeated: + selector.select_execution_target(task_file, evaluated_at=kst(12), transition="failover", prior_decision=resumed, failure_class="provider-quota") + self.assertEqual(repeated.exception.code, "no_failover_candidate") + + def test_failover_never_returns_to_an_earlier_candidate_rank(self): + gemini_exhausted_snapshot = { + "snapshot_id": "gemini-exhausted", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T03:00:00+09:00", + "targets": [ + { + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + "status": "exhausted", + } + ], + } + gemini_available_snapshot = { + "snapshot_id": "gemini-recovered", + "source": "iop-node quota-probe", + "checked_at": "2026-07-25T04:00:00+09:00", + "targets": [ + { + "adapter": "agy", + "target": "Gemini 3.6 Flash (Medium)", + "status": "available", + } + ], + } + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + prior = selector.select_execution_target( + task_file, evaluated_at=kst(12), quota_snapshot=gemini_exhausted_snapshot + ) + self.assertEqual(prior["selected"]["adapter"], "pi") + self.assertEqual(prior["selected"]["target"], "iop/laguna-s:2.1") + + with self.assertRaises(selector.SelectorInputError) as ctx: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="failover", + prior_decision=prior, + failure_class="provider-stream-disconnect", + quota_snapshot=gemini_available_snapshot, + ) + self.assertEqual(ctx.exception.code, "no_failover_candidate") + + def test_tampered_prior_decision_rejected(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + prior = selector.select_execution_target(task_file, evaluated_at=kst(12)) + + variants = { + "extra_candidate": lambda p: { + **p, + "candidates": list(p["candidates"]) + + [ + { + "candidate_rank": 3, + "adapter": "codex", + "target": "gpt-5.6-sol", + "execution_class": "cloud_model", + "selfcheck_required": False, + "quota_mode": "bounded", + "quota_status": "unknown", + "eligibility": "eligible", + "rejection_reason": None, + } + ], + }, + "bad_selected": lambda p: { + **p, + "selected": { + "adapter": "codex", + "target": "gpt-5.6-sol", + "execution_class": "cloud_model", + "selfcheck_required": False, + }, + }, + "used_duplicate": lambda p: { + **p, + "used_candidates": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + ], + }, + "used_reordered": lambda p: { + **p, + "selected": {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + "used_candidates": [ + {"adapter": "pi", "target": "iop/laguna-s:2.1"}, + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + ], + }, + "selected_used_tail_mismatch": lambda p: { + **p, + "selected": {"adapter": "pi", "target": "iop/laguna-s:2.1"}, + "used_candidates": [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + ], + }, + "tampered_rule_id": lambda p: { + **p, + "decision": {**p["decision"], "rule_id": "fake-rule"}, + }, + "tampered_policy_priority": lambda p: { + **p, + "decision": {**p["decision"], "policy_priority": 99}, + }, + "tampered_reason_codes": lambda p: { + **p, + "decision": {**p["decision"], "reason_codes": ["fake_reason"]}, + }, + "tampered_time_window": lambda p: { + **p, + "decision": {**p["decision"], "time_window": "kst-night-[23:00,07:00)"}, + }, + "invalid_evaluated_at": lambda p: { + **p, + "decision": {**p["decision"], "evaluated_at": "invalid-iso-datetime"}, + }, + "naive_evaluated_at": lambda p: { + **p, + "decision": {**p["decision"], "evaluated_at": "2026-07-25T12:00:00"}, + }, + } + + for name, modifier in variants.items(): + with self.subTest(variant=name): + tampered = modifier(copy.deepcopy(prior)) + with self.assertRaises(selector.SelectorInputError) as exc: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="failover", + prior_decision=tampered, + failure_class="provider-quota", + ) + self.assertEqual(exc.exception.code, "malformed_prior_decision") + + with self.assertRaises(selector.SelectorInputError) as exc_resume: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="resume", + prior_decision=tampered, + ) + self.assertEqual(exc_resume.exception.code, "malformed_prior_decision") + + def test_cross_boundary_failover_resumes_pinned_decision(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + # 22:59 KST is daytime policy -> agy primary + day_initial = selector.select_execution_target( + task_file, evaluated_at=kst(22, 59, 0) + ) + self.assertEqual(day_initial["selected"]["adapter"], "agy") + + # 23:00 KST is nighttime -> failover to pi + night_failover = selector.select_execution_target( + task_file, + evaluated_at=kst(23, 0, 0), + transition="failover", + prior_decision=day_initial, + failure_class="provider-quota", + ) + self.assertEqual(night_failover["selected"]["adapter"], "pi") + self.assertEqual( + night_failover["used_candidates"], + [ + {"adapter": "agy", "target": "Gemini 3.6 Flash (Medium)"}, + {"adapter": "pi", "target": "iop/laguna-s:2.1"}, + ], + ) + + # 23:01 KST nighttime resume -> preserved pinned pi decision + night_resume = selector.select_execution_target( + task_file, + evaluated_at=kst(23, 1, 0), + transition="resume", + prior_decision=night_failover, + ) + self.assertEqual(night_resume["selected"]["adapter"], "pi") + self.assertIs(night_resume["decision"]["pinned"], True) + self.assertEqual(night_resume["used_candidates"], night_failover["used_candidates"]) + + def test_runtime_probed_cloud_alternate_round_trips_selected_snapshot(self): + snapshot = go_quota_snapshot( + "agy", + "Gemini 3.6 Flash (Medium)", + "available", + snapshot_id="night-gemini-available", + ) + completed = mock.Mock(returncode=0, stdout=json.dumps(snapshot)) + with TemporaryDirectory() as tmp, mock.patch( + "subprocess.run", return_value=completed + ) as run_mock: + task_file = write_task_file(Path(tmp), "PLAN", "local", 8) + prior = selector.select_execution_target( + task_file, evaluated_at=kst(1) + ) + self.assertEqual(prior["selected"]["adapter"], "pi") + result = selector.select_execution_target( + task_file, + evaluated_at=kst(1), + transition="failover", + prior_decision=prior, + failure_class="provider-stream-disconnect", + ) + + self.assertEqual(result["selected"]["adapter"], "agy") + selected_candidate = next( + candidate + for candidate in result["candidates"] + if candidate["adapter"] == "agy" + ) + self.assertEqual(selected_candidate["quota_status"], "available") + self.assertEqual(result["quota"]["status"], "available") + self.assertEqual( + result["quota"]["snapshot_id"], snapshot["snapshot_id"] + ) + self.assertEqual(result["quota"]["targets"], snapshot["targets"]) + self.assertEqual(run_mock.call_count, 2) + + def test_policy_owned_cloud_promotion_chain_and_no_bounce(self): + with TemporaryDirectory() as tmp: + task_file = write_task_file(Path(tmp), "PLAN", "cloud", 5) + initial = selector.select_execution_target( + task_file, + evaluated_at=kst(12), + quota_probe_command="missing-probe", + ) + claude = selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="promotion", + prior_decision=initial, + failure_class="provider-quota", + ) + resumed = selector.select_execution_target( + task_file, + evaluated_at=kst(23), + transition="resume", + prior_decision=claude, + ) + terra = selector.select_execution_target( + task_file, + evaluated_at=kst(23), + transition="promotion", + prior_decision=resumed, + failure_class="context-limit", + ) + + self.assertEqual( + (claude["selected"]["adapter"], claude["selected"]["target"]), + ("claude", "claude-opus-4-8"), + ) + self.assertEqual(claude["transition"]["kind"], "promotion") + self.assertEqual(claude["transition"]["trigger"], "provider-quota") + self.assertEqual( + (terra["selected"]["adapter"], terra["selected"]["target"]), + ("codex", "gpt-5.6-terra"), + ) + self.assertEqual( + terra["promotion_path"], + [ + { + "adapter": "agy", + "target": "Gemini 3.6 Flash (High)", + }, + {"adapter": "claude", "target": "claude-opus-4-8"}, + {"adapter": "codex", "target": "gpt-5.6-terra"}, + ], + ) + with self.assertRaises(selector.SelectorInputError) as exhausted: + selector.select_execution_target( + task_file, + evaluated_at=kst(23), + transition="promotion", + prior_decision=terra, + failure_class="provider-quota", + ) + self.assertEqual(exhausted.exception.code, "no_promotion_target") + with self.assertRaises(selector.SelectorInputError) as generic: + selector.select_execution_target( + task_file, + evaluated_at=kst(12), + transition="promotion", + prior_decision=initial, + failure_class="generic-error", + ) + self.assertEqual( + generic.exception.code, "unqualified_promotion_trigger" + ) + + def test_probe_candidate_quota_argv_and_normalization(self): + eval_time = kst(14, 0, 0) + snapshot = go_quota_snapshot( + "agy", + "Gemini 3.6 Flash (Medium)", + "available", + snapshot_id="snap-99", + ) + snapshot["required_caps"].append( + { + "name": "model:Gemini 3.6 Flash (Medium)", + "status": "available", + "remaining_percent": 40.0, + } + ) + with mock.patch("subprocess.run") as run_mock: + run_mock.return_value = mock.Mock( + returncode=0, + stdout=json.dumps(snapshot), + ) + result = selector.probe_candidate_quota( + target="Gemini 3.6 Flash (Medium)", + adapter="agy", + required_caps=("overall", "model:Gemini 3.6 Flash (Medium)"), + checked_at=eval_time, + quota_probe_command="iop-node quota-probe", + ) + self.assertEqual(result, snapshot) + run_mock.assert_called_once() + cmd = run_mock.call_args[0][0] + self.assertEqual( + cmd, + [ + "iop-node", + "quota-probe", + "--target", + "Gemini 3.6 Flash (Medium)", + "--command", + "agy", + "--required-cap", + "overall", + "--required-cap", + "model:Gemini 3.6 Flash (Medium)", + "--checked-at", + eval_time.isoformat(), + ], + ) + + def test_probe_candidate_quota_error_normalizes_to_unknown(self): + eval_time = kst(14, 0, 0) + error_cases = [ + mock.Mock(returncode=1, stdout=""), + mock.Mock(returncode=0, stdout="invalid json"), + mock.Mock(returncode=0, stdout=json.dumps({"status": "invalid_status"})), + OSError("binary not found"), + ] + for side_effect in error_cases: + with self.subTest(side_effect=side_effect): + with mock.patch("subprocess.run") as run_mock: + if isinstance(side_effect, Exception): + run_mock.side_effect = side_effect + else: + run_mock.return_value = side_effect + result = selector.probe_candidate_quota( + target="claude-opus-4-8", + adapter="claude", + required_caps=("overall",), + checked_at=eval_time, + ) + self.assertEqual(result["targets"][0]["status"], "unknown") + self.assertEqual(result["reason_codes"], ["probe_error"]) + self.assertIsNone(result["snapshot_id"]) + + def test_probe_candidate_quota_accepts_probe_command(self): + eval_time = kst(14, 0, 0) + snapshot = go_quota_snapshot( + "agy", + "Gemini 3.6 Flash (Medium)", + "available", + snapshot_id="snap-100", + ) + with mock.patch("subprocess.run") as run_mock: + run_mock.return_value = mock.Mock( + returncode=0, + stdout=json.dumps(snapshot), + ) + result = selector.probe_candidate_quota( + target="Gemini 3.6 Flash (Medium)", + adapter="agy", + probe_command="antigravity", + required_caps=("overall",), + checked_at=eval_time, + quota_probe_command="iop-node quota-probe", + ) + self.assertEqual(result, snapshot) + run_mock.assert_called_once() + cmd = run_mock.call_args[0][0] + self.assertIn("--command", cmd) + cmd_idx = cmd.index("--command") + self.assertEqual(cmd[cmd_idx + 1], "antigravity") + + +class QuotaBatchProviderTest(unittest.TestCase): + def test_command_profile_axis_is_not_deduplicated(self): + eval_time = kst(14, 0, 0) + provider = selector.QuotaBatchProvider(quota_probe_command="iop-node quota-probe") + key1 = ("agy", "Gemini 3.6 Flash (Medium)", "agy", ("overall",)) + key2 = ("agy", "Gemini 3.6 Flash (Medium)", "antigravity", ("overall",)) + + calls = [] + + def mock_probe(*args, **kwargs): + calls.append(kwargs) + adapter = kwargs["adapter"] + target = kwargs["target"] + cmd = kwargs.get("probe_command", adapter) + return { + "schema_version": "1.0", + "snapshot_id": f"child-{adapter}-{cmd}", + "source": "iop-node quota-probe", + "checked_at": eval_time.isoformat(), + "targets": [{"adapter": adapter, "target": target, "status": "available"}], + "required_caps": [{"name": "overall", "status": "available", "remaining_percent": 90.0}], + "reason_codes": ["ok"], + } + + with mock.patch.object(selector, "probe_candidate_quota", side_effect=mock_probe): + batch_snap = provider.aggregate( + snapshot_id="batch-123", + checked_at=eval_time, + keys=[key1, key2], + ) + + self.assertIsNotNone(batch_snap) + # Verify 2 separate probes were called because probe_command differed (agy vs antigravity) + self.assertEqual(len(calls), 2) + self.assertEqual(calls[0]["probe_command"], "agy") + self.assertEqual(calls[1]["probe_command"], "antigravity") + + # Verify child target evidence preserved in batch snapshot + self.assertEqual(len(batch_snap["targets"]), 2) + self.assertEqual(batch_snap["targets"][0]["command"], "agy") + self.assertEqual(batch_snap["targets"][1]["command"], "antigravity") + self.assertEqual(batch_snap["targets"][0]["child_snapshot_id"], "child-agy-agy") + self.assertEqual(batch_snap["targets"][1]["child_snapshot_id"], "child-agy-antigravity") + + +if __name__ == "__main__": + unittest.main() diff --git a/agent-ops/skills/common/plan/SKILL.md b/agent-ops/skills/common/plan/SKILL.md index e03060e..fa9561b 100644 --- a/agent-ops/skills/common/plan/SKILL.md +++ b/agent-ops/skills/common/plan/SKILL.md @@ -270,7 +270,7 @@ Required sections: - Use repository-relative or canonical absolute file paths. Never use a glob (`*`, `?`, `[]`), directory path, workspace root, URL, path outside the workspace, malformed path, or prose placeholder as a claim. - Enumerate only implementer- or reviewer-owned workspace files, including the active review evidence file and deterministic workspace evidence artifacts. - For generated verification artifacts, choose deterministic exact workspace filenames or write them under a task-specific temporary directory outside the repository. Never substitute a directory or glob claim for dynamic filenames. - - Before writing or returning a prepared pair, validate the rendered PLAN with `python3 agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan `. A prepared in-memory PLAN may be materialized only as a temporary candidate outside the repository for this validation. Require exit code `0`. On failure, do not write or return the pair. + - Before writing or returning a prepared pair, validate the rendered PLAN with `python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan `. A prepared in-memory PLAN may be materialized only as a temporary candidate outside the repository for this validation. Require exit code `0`. On failure, do not write or return the pair. - `Final Verification`: runnable commands and expected outcome. Prefer commands from verified handoff facts when supplied; fill missing coverage from repository manifests, scripts, workflows, domain rules, and related tests, and record the source in `Analysis > Verification Context`. Commands must be exact and deterministic enough for the reviewer to rerun; use stable ordering for searches and state whether cached test output is acceptable. End this section with **"After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`."** Each plan item must include: @@ -343,7 +343,7 @@ Do not write or return a prepared pair when either routing target is not `routed ## Final Checklist - In `write` mode, the routed `PLAN-{build_lane}-GNN.md` and `CODE_REVIEW-{review_lane}-GNN.md` both exist under `agent-task/{task_name}/`. In `prepare-follow-up` mode, neither routed file was written; both exact bodies and basenames were returned while the verdict-appended current pair remained active. -- The rendered PLAN passed `python3 agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan ` before the pair was written or returned. Its single non-empty `Modified Files Summary` contains only exact workspace file claims and no glob or directory claim. +- The rendered PLAN passed `python3 agent-ops/skills/common/orchestrate-agent-task-loop/scripts/dispatch.py --workspace --validate-plan ` before the pair was written or returned. Its single non-empty `Modified Files Summary` contains only exact workspace file claims and no glob or directory claim. - In `write` mode, `.gitignore` has the Agent-Ops managed block that unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores local `agent-roadmap/current.md`. In `prepare-follow-up` mode, the block was only inspected and any needed repair was returned as `gitignore_repair_needed`. - Single-plan work stores active files directly under `agent-task/{task_group}/`. - Split work, if any, uses one shared `agent-task/{task_group}/` parent and one subtask directory per plan/review pair with names like `01_core`, `02+01_edge_integration`, `03+01_node_integration`; dependency details live in the subtask directory name as `NN+PP[,QQ...]_subtask_name`. diff --git a/agent-ops/skills/common/router.md b/agent-ops/skills/common/router.md index efca5aa..cdc1f04 100644 --- a/agent-ops/skills/common/router.md +++ b/agent-ops/skills/common/router.md @@ -47,6 +47,7 @@ | ๊ณ„ํš ์„ธ์›Œ์ค˜, ๊ณ„ํš ์ž‘์„ฑํ•ด, ๊ณ„ํš ๋งŒ๋“ค์–ด์ค˜, ๊ตฌํ˜„ ๊ณ„ํš, PLAN.md, plan, plan ์ž‘์„ฑํ•ด, plan ๋งŒ๋“ค์–ด์ค˜ | `agent-ops/skills/common/plan/SKILL.md` | | ํ˜„์žฌ plan๋“ค ์„ธ๋ถ„ํ™”ํ•ด, ํ˜„์žฌ plan ์„ธ๋ถ„ํ™”, ๊ธฐ์กด plan ๋” ๋‚˜๋ˆ , task ์„ธ๋ถ„ํ™”ํ•ด, plan ๋ถ„๋ฆฌํ•ด | `agent-ops/skills/common/refine-plans/SKILL.md` | | ์ตœ์ข… ๋ผ์šฐํŒ…, task routing, cloud/local ์žฌํ‰๊ฐ€, lane/G ํŒ๋‹จ, G ๋“ฑ๊ธ‰ ์žฌํ‰๊ฐ€, routed filename ๊ฒฐ์ • | `agent-ops/skills/common/finalize-task-routing/SKILL.md` | +| agent-task ์ž‘์—… ์‹คํ–‰, agent-task ์ž‘์—…๋“ค ์‹คํ–‰ํ•ด, agent-task ๋ฌด์ธ ์‹คํ–‰, task-group dry-run/live pass, blocked retry | `agent-ops/skills/common/orchestrate-agent-task-loop/SKILL.md` | | ์ฝ”๋“œ ๋ฆฌ๋ทฐํ•ด์ค˜, ๋ฆฌ๋ทฐ ์ง„ํ–‰ํ•ด, ๋ฆฌ๋ทฐํ•ด์ค˜, code review, CODE_REVIEW.md, ๋ฆฌ๋ทฐ ๋ฃจํ”„ | `agent-ops/skills/common/code-review/SKILL.md` | | ์ปค๋ฐ‹ํ•ด์ค˜, ํ‘ธ์‹œํ•ด์ค˜, commit, push, ๋ฐ˜์˜ํ•ด์ค˜ | `agent-ops/skills/common/commit-push/SKILL.md` | | agent-ops ์‹ฑํฌํ•ด, agent-ops ๋™๊ธฐํ™”ํ•ด, agentic-framework์— ์˜ฌ๋ ค์ค˜, agent-ops๋ฅผ [ํ”„๋กœ์ ํŠธ]๋กœ ์‹ฑํฌํ•ด | `agent-ops/skills/common/sync-push/SKILL.md` |