fix(agent-ops): provider 무응답 재시도를 차단한다

This commit is contained in:
toki 2026-08-13 20:34:50 +09:00
parent 11ba39609f
commit cb01b86f8d
7 changed files with 46 additions and 127 deletions

View file

@ -50,13 +50,11 @@ Each target has:
- `execution_class`: `local_model` or `cloud_model`; - `execution_class`: `local_model` or `cloud_model`;
- optional `selfcheck_required` boolean; - optional `selfcheck_required` boolean;
- `runtime.command`: a non-empty argv template executed without a shell; - `runtime.command`: a non-empty argv template executed without a shell;
- optional `runtime.resume_command`, `preflight_command`, `environment`, `session_path`, `native_session_monitor`, `model_response_stall_seconds`, `terminal_success`, and `auxiliary_logs`; - optional `runtime.resume_command`, `preflight_command`, `environment`, `session_path`, `native_session_monitor`, `terminal_success`, and `auxiliary_logs`;
- optional `runtime.output_format`: `text` or `jsonl`. - optional `runtime.output_format`: `text` or `jsonl`.
Command templates may use only `{agent}`, `{model}`, `{reasoning_effort}`, `{target_id}`, `{workspace}`, `{attempt_dir}`, `{session_id}`, `{resume_session}`, `{resume_session_dir}`, and `{prompt}`. A target with `reasoning_effort` must use `{reasoning_effort}` in its command and resume command when present; a target without the field cannot use that placeholder. `native_session_monitor=true` requires both `resume_command` and `session_path`. `terminal_success=agent_end` requires JSONL output and accepts only a non-retrying final `agent_end` whose last assistant message has `stopReason=stop`. `terminal_success=turn_completed` requires JSONL output and accepts only final `turn.completed`; `turn.failed` or a missing terminal event fails closed. When either declared success event is observed with exit 0, earlier recovered transport diagnostics do not turn the attempt into a failure. The catalog must not embed repository secrets; environment values should refer only to runtime-provided non-secret configuration. Command templates may use only `{agent}`, `{model}`, `{reasoning_effort}`, `{target_id}`, `{workspace}`, `{attempt_dir}`, `{session_id}`, `{resume_session}`, `{resume_session_dir}`, and `{prompt}`. A target with `reasoning_effort` must use `{reasoning_effort}` in its command and resume command when present; a target without the field cannot use that placeholder. `native_session_monitor=true` requires both `resume_command` and `session_path`. `terminal_success=agent_end` requires JSONL output and accepts only a non-retrying final `agent_end` whose last assistant message has `stopReason=stop`. `terminal_success=turn_completed` requires JSONL output and accepts only final `turn.completed`; `turn.failed` or a missing terminal event fails closed. When either declared success event is observed with exit 0, earlier recovered transport diagnostics do not turn the attempt into a failure. The catalog must not embed repository secrets; environment values should refer only to runtime-provided non-secret configuration.
`runtime.model_response_stall_seconds` is a positive integer target-owned silence budget. Omission keeps the dispatcher default of 180 seconds. A runtime that can remain externally silent while performing bounded downstream recovery must set this value above its complete silent recovery window, including the initial attempt, every internal retry, and bounded cleanup overhead. Record the resolved value in each locator and never infer it from provider or model names in dispatcher code.
Each route owns its ordered `candidates` plus optional `rule_id`, `policy_priority`, and `reason_codes`. A route may use catalog-owned `windows` instead of a fixed candidate list; every window supplies an IANA timezone, start/end time, and candidates. Exactly one window must match. Each route owns its ordered `candidates` plus optional `rule_id`, `policy_priority`, and `reason_codes`. A route may use catalog-owned `windows` instead of a fixed candidate list; every window supplies an IANA timezone, start/end time, and candidates. Exactly one window must match.
The bundled review routes vary model and reasoning effort by routed grade instead of fixing every review to one target: G01-G04 use Terra/high, G05-G08 use Sol/high, and G09-G10 use Sol/xhigh. Runtime or project catalog overrides may replace this default tiering. The bundled review routes vary model and reasoning effort by routed grade instead of fixing every review to one target: G01-G04 use Terra/high, G05-G08 use Sol/high, and G09-G10 use Sol/xhigh. Runtime or project catalog overrides may replace this default tiering.
@ -113,6 +111,8 @@ Accept self-check completion only when `## Implementation Checklist` or its supp
- Record the target id, opaque agent/model identity, execution class, runtime contract, catalog evidence, process identity, workspace identity, timestamps, result, and exact failure evidence. - Record the target id, opaque agent/model identity, execution class, runtime contract, catalog evidence, process identity, workspace identity, timestamps, result, and exact failure evidence.
- Treat stderr as terminal diagnostic evidence. For JSONL, recognize generic terminal event fields such as error/fatal type or severity, rejected/failed status with an error code, explicit error flags, and a non-retrying `agent_end` whose last assistant message ends with `error` or `aborted`. - Treat stderr as terminal diagnostic evidence. For JSONL, recognize generic terminal event fields such as error/fatal type or severity, rejected/failed status with an error code, explicit error flags, and a non-retrying `agent_end` whose last assistant message ends with `error` or `aborted`.
- Determine liveness from PID/start-token/process-marker evidence and actual stream or native-session progress. Heartbeat mtime is never agent progress. For Codex JSONL, an unmatched `item.started` `command_execution` is an active tool interval: suspend the model-response silence timer until its matching `item.completed`, then restore normal stall detection. - Determine liveness from PID/start-token/process-marker evidence and actual stream or native-session progress. Heartbeat mtime is never agent progress. For Codex JSONL, an unmatched `item.started` `command_execution` is an active tool interval: suspend the model-response silence timer until its matching `item.completed`, then restore normal stall detection.
- The dispatcher model-silence safety net is 70 seconds. Downstream provider runtimes should emit their bounded terminal before that deadline; do not extend the dispatcher budget per target to cover nested retries.
- Treat a confirmed provider transport terminal as the end of the current dispatch. Do not resume or automatically resend the same native session; an operator may start a fresh dispatch after the provider/runtime state is corrected.
- Never start a duplicate attempt while owned live evidence remains. - Never start a duplicate attempt while owned live evidence remains.
- Keep a 10-consecutive-failure budget per task stage. Reset only that stage's budget after success. - Keep a 10-consecutive-failure budget per task stage. Reset only that stage's budget after success.
- Preserve failed attempt logs. Delete successful attempt logs only after verified archive completion and no live evidence. - Preserve failed attempt logs. Delete successful attempt logs only after verified archive completion and no live evidence.

View file

@ -47,7 +47,6 @@
"output_format": "jsonl", "output_format": "jsonl",
"session_path": "{attempt_dir}/pi-sessions/*{session_id}*.jsonl", "session_path": "{attempt_dir}/pi-sessions/*{session_id}*.jsonl",
"native_session_monitor": true, "native_session_monitor": true,
"model_response_stall_seconds": 1500,
"terminal_success": "agent_end" "terminal_success": "agent_end"
} }
}, },

View file

@ -179,7 +179,7 @@ def validated_max_parallel(value: int) -> int:
STREAM_HEARTBEAT_SECONDS = 30 STREAM_HEARTBEAT_SECONDS = 30
MODEL_RESPONSE_STALL_SECONDS = 3 * 60 MODEL_RESPONSE_STALL_SECONDS = 70
RECOVERY_FAILURE_LIMIT = 10 RECOVERY_FAILURE_LIMIT = 10
GENERIC_FAILURE_LIMIT_PER_TARGET = 3 GENERIC_FAILURE_LIMIT_PER_TARGET = 3
SELF_CHECK_UNCHECKED_RETRY_LIMIT = 10 SELF_CHECK_UNCHECKED_RETRY_LIMIT = 10
@ -209,6 +209,7 @@ RUNTIME_FAILURE_PATTERNS = {
], ],
"provider-connection": [ "provider-connection": [
r"\bprovider[_ -]?tunnel[_ -]?error\b", r"\bprovider[_ -]?tunnel[_ -]?error\b",
r"no provider supports the required output validation capability",
( (
r"(?:provider|backend|inference (?:server|endpoint))" r"(?:provider|backend|inference (?:server|endpoint))"
r".{0,160}(?:connection refused|dial tcp)" r".{0,160}(?:connection refused|dial tcp)"
@ -511,19 +512,6 @@ class AgentSpec:
reasoning_effort: str | None = None reasoning_effort: str | None = None
runtime: dict[str, Any] = field(default_factory=dict) runtime: dict[str, Any] = field(default_factory=dict)
def model_response_stall_seconds(spec: AgentSpec) -> float:
"""Resolve the catalog-owned silence budget for one selected target."""
value = spec.runtime.get("model_response_stall_seconds")
if value is None:
return float(MODEL_RESPONSE_STALL_SECONDS)
if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
raise ExecutionDecisionError(
"model_response_stall_seconds must be a positive integer"
)
return float(value)
def agent_spec_from_record(record: dict[str, Any]) -> AgentSpec | None: def agent_spec_from_record(record: dict[str, Any]) -> AgentSpec | None:
cli = str(record.get("cli") or "") cli = str(record.get("cli") or "")
model = str(record.get("model") or "") model = str(record.get("model") or "")
@ -3446,7 +3434,6 @@ async def invoke(
prompt: str, prompt: str,
resume_locator: Path | None = None, resume_locator: Path | None = None,
) -> tuple[int, str | None, Path]: ) -> tuple[int, str | None, Path]:
response_stall_seconds = model_response_stall_seconds(spec)
worker_signature_before = ( worker_signature_before = (
task_signature(workspace, task) if role == "worker" else None task_signature(workspace, task) if role == "worker" else None
) )
@ -3530,7 +3517,6 @@ async def invoke(
"selfcheck_required": spec.selfcheck_required, "selfcheck_required": spec.selfcheck_required,
"reasoning_effort": spec.reasoning_effort, "reasoning_effort": spec.reasoning_effort,
"runtime": spec.runtime, "runtime": spec.runtime,
"model_response_stall_timeout_seconds": response_stall_seconds,
"agent_process_marker": process_marker, "agent_process_marker": process_marker,
"plan_path": str(task.plan) if task.plan else None, "plan_path": str(task.plan) if task.plan else None,
"review_path": str(task.review) if task.review else None, "review_path": str(task.review) if task.review else None,
@ -3834,13 +3820,13 @@ async def invoke(
record["native_pending_tool_call_ids"] = list( record["native_pending_tool_call_ids"] = list(
native_state.pending_tool_call_ids native_state.pending_tool_call_ids
) )
record["native_stall_timeout_seconds"] = response_stall_seconds record["native_stall_timeout_seconds"] = MODEL_RESPONSE_STALL_SECONDS
record.setdefault("native_activity_state", "starting") record.setdefault("native_activity_state", "starting")
if ( if (
spec.native_resume spec.native_resume
and not is_native_tool_execution and not is_native_tool_execution
and not is_command_execution and not is_command_execution
and native_inactive_seconds >= response_stall_seconds and native_inactive_seconds >= MODEL_RESPONSE_STALL_SECONDS
and session_stall_seconds is None and session_stall_seconds is None
): ):
session_stall_seconds = native_inactive_seconds session_stall_seconds = native_inactive_seconds
@ -3867,7 +3853,7 @@ async def invoke(
not spec.native_resume not spec.native_resume
and not is_command_execution and not is_command_execution
and non_native_inactive_seconds and non_native_inactive_seconds
>= response_stall_seconds >= MODEL_RESPONSE_STALL_SECONDS
and session_stall_seconds is None and session_stall_seconds is None
): ):
session_stall_seconds = non_native_inactive_seconds session_stall_seconds = non_native_inactive_seconds
@ -4660,6 +4646,37 @@ async def run_escalating(
], ],
) )
return False, locator return False, locator
if failure in PROVIDER_TRANSPORT_FAILURES:
reason = (
f"{role} provider transport failure requires a fresh dispatch"
)
selected = (
current_decision.get("selected")
if isinstance(current_decision, dict)
else None
)
store.update_task(
task,
blocked=f"{reason} locator={locator}",
blocker_evidence={
"role": role,
"failure_class": failure,
"locator": str(locator) if locator else None,
"selected": selected,
"work_unit_id": current_decision.get("work_unit_id")
if isinstance(current_decision, dict)
else None,
},
)
banner(
"작업차단",
task.name,
[
"reason=provider-transport-terminal",
*failure_report_lines(failure, locator),
],
)
return False, locator
if spec.native_resume: if spec.native_resume:
if failure in {"context-limit", "session-stall"}: if failure in {"context-limit", "session-stall"}:
native_recovery_retries += 1 native_recovery_retries += 1

View file

@ -22,7 +22,6 @@ VALID_STAGES = {"worker", "review"}
VALID_LANES = {"local", "cloud"} VALID_LANES = {"local", "cloud"}
VALID_EXECUTION_CLASSES = {"local_model", "cloud_model"} VALID_EXECUTION_CLASSES = {"local_model", "cloud_model"}
VALID_OUTPUT_FORMATS = {"jsonl", "text"} VALID_OUTPUT_FORMATS = {"jsonl", "text"}
MAX_MODEL_RESPONSE_STALL_SECONDS = 24 * 60 * 60
ALLOWED_TEMPLATE_FIELDS = { ALLOWED_TEMPLATE_FIELDS = {
"agent", "agent",
"attempt_dir", "attempt_dir",
@ -110,7 +109,6 @@ def _validate_runtime(value: object, label: str) -> dict[str, Any]:
"output_format", "output_format",
"session_path", "session_path",
"native_session_monitor", "native_session_monitor",
"model_response_stall_seconds",
"terminal_success", "terminal_success",
"auxiliary_logs", "auxiliary_logs",
} }
@ -178,19 +176,6 @@ def _validate_runtime(value: object, label: str) -> dict[str, Any]:
raise CatalogError( raise CatalogError(
f"{label}.native_session_monitor requires {missing}" f"{label}.native_session_monitor requires {missing}"
) )
model_response_stall_seconds = value.get("model_response_stall_seconds")
if model_response_stall_seconds is not None:
if (
isinstance(model_response_stall_seconds, bool)
or not isinstance(model_response_stall_seconds, int)
or model_response_stall_seconds <= 0
or model_response_stall_seconds > MAX_MODEL_RESPONSE_STALL_SECONDS
):
raise CatalogError(
f"{label}.model_response_stall_seconds must be an integer "
f"between 1 and {MAX_MODEL_RESPONSE_STALL_SECONDS}"
)
runtime["model_response_stall_seconds"] = model_response_stall_seconds
auxiliary_logs = value.get("auxiliary_logs", []) auxiliary_logs = value.get("auxiliary_logs", [])
if not isinstance(auxiliary_logs, list) or not all( if not isinstance(auxiliary_logs, list) or not all(
isinstance(item, str) and item for item in auxiliary_logs isinstance(item, str) and item for item in auxiliary_logs

View file

@ -393,6 +393,13 @@ class RuntimeCatalogDispatcherTests(unittest.TestCase):
self.assertEqual(failure, "provider-quota") self.assertEqual(failure, "provider-quota")
self.assertIsNotNone(evidence) self.assertIsNotNone(evidence)
def test_output_validation_capability_rejection_is_provider_terminal(self):
failure, evidence = dispatch.classify_failure_with_evidence(
"no provider supports the required output validation capability"
)
self.assertEqual(failure, "provider-connection")
self.assertIsNotNone(evidence)
def test_generic_json_terminal_diagnostic_has_no_agent_branch(self): def test_generic_json_terminal_diagnostic_has_no_agent_branch(self):
diagnostic = dispatch.terminal_diagnostic( diagnostic = dispatch.terminal_diagnostic(
"opaque-agent", "opaque-agent",
@ -754,67 +761,6 @@ class RuntimeCatalogDispatcherTests(unittest.TestCase):
self.assertGreaterEqual(record["session_stall_seconds"], 0.05) self.assertGreaterEqual(record["session_stall_seconds"], 0.05)
self.assertIn("native_silence_inspection", record) self.assertIn("native_silence_inspection", record)
def test_target_silence_budget_survives_default_window_after_turn_start(self):
with TemporaryDirectory() as tmp:
root = Path(tmp)
plan = write_plan(root)
task = task_from_plan(root, plan)
runner = root / "bounded_silent_runner.py"
runner.write_text(
"import json, time\n"
"print(json.dumps({'type': 'turn_start'}), flush=True)\n"
"time.sleep(0.15)\n"
"print(json.dumps({'type': 'agent_end', 'willRetry': False, "
"'messages': [{'role': 'assistant', 'stopReason': 'stop'}]}), "
"flush=True)\n",
encoding="utf-8",
)
agent = dispatch.AgentSpec(
"bounded-silent-runner",
"bounded-model",
"bounded-silent-runner/bounded-model",
native_resume=True,
target_id="bounded-target",
runtime={
"command": [sys.executable, str(runner)],
"output_format": "jsonl",
"native_session_monitor": True,
"session_path": "sessions/{session_id}.jsonl",
"terminal_success": "agent_end",
"model_response_stall_seconds": 1,
},
)
with (
mock.patch.dict(
os.environ,
{"XDG_STATE_HOME": str(root / "state")},
),
mock.patch.object(dispatch, "STREAM_HEARTBEAT_SECONDS", 0.01),
mock.patch.object(dispatch, "MODEL_RESPONSE_STALL_SECONDS", 0.05),
):
store = dispatch.StateStore(root)
try:
return_code, failure, locator = asyncio.run(
dispatch.invoke(
root,
store,
task,
"worker",
agent,
"fake prompt",
)
)
record = json.loads(locator.read_text(encoding="utf-8"))
finally:
store.close()
self.assertEqual(return_code, 0)
self.assertIsNone(failure)
self.assertEqual(record["status"], "succeeded")
self.assertEqual(record["model_response_stall_timeout_seconds"], 1.0)
self.assertEqual(record["native_stall_timeout_seconds"], 1.0)
self.assertNotIn("session_stall_seconds", record)
def test_non_native_active_command_execution_is_not_classified_as_stall(self): def test_non_native_active_command_execution_is_not_classified_as_stall(self):
with TemporaryDirectory() as tmp: with TemporaryDirectory() as tmp:
root = Path(tmp) root = Path(tmp)

View file

@ -207,33 +207,6 @@ class ExecutionTargetPolicyTests(unittest.TestCase):
): ):
policy.load_catalog(write_catalog(Path(tmp), value)) policy.load_catalog(write_catalog(Path(tmp), value))
def test_model_response_stall_budget_is_bounded_positive_integer(self):
valid = catalog_value()
valid["targets"]["target-a"]["runtime"][
"model_response_stall_seconds"
] = 1200
invalid_values = (True, 0, -1, 1.5, 24 * 60 * 60 + 1)
with TemporaryDirectory() as tmp:
root = Path(tmp)
loaded = policy.load_catalog(write_catalog(root, valid))
self.assertEqual(
loaded.targets["target-a"].runtime[
"model_response_stall_seconds"
],
1200,
)
for invalid in invalid_values:
with self.subTest(invalid=invalid):
value = catalog_value()
value["targets"]["target-a"]["runtime"][
"model_response_stall_seconds"
] = invalid
with self.assertRaisesRegex(
policy.CatalogError,
"model_response_stall_seconds must be an integer",
):
policy.load_catalog(write_catalog(root, value))
def test_terminal_success_contract_requires_jsonl_agent_end(self): def test_terminal_success_contract_requires_jsonl_agent_end(self):
valid = catalog_value() valid = catalog_value()
valid["targets"]["target-a"]["runtime"]["terminal_success"] = "agent_end" valid["targets"]["target-a"]["runtime"]["terminal_success"] = "agent_end"

View file

@ -172,7 +172,6 @@ class SelectorTests(unittest.TestCase):
self.assertIn("--thinking", pi.runtime["command"]) self.assertIn("--thinking", pi.runtime["command"])
self.assertIn("{reasoning_effort}", pi.runtime["command"]) self.assertIn("{reasoning_effort}", pi.runtime["command"])
self.assertTrue(pi.runtime["native_session_monitor"]) self.assertTrue(pi.runtime["native_session_monitor"])
self.assertEqual(pi.runtime["model_response_stall_seconds"], 1500)
self.assertEqual(pi.runtime["terminal_success"], "agent_end") self.assertEqual(pi.runtime["terminal_success"], "agent_end")
self.assertIn("--session", pi.runtime["resume_command"]) self.assertIn("--session", pi.runtime["resume_command"])
self.assertIn("{resume_session_dir}", pi.runtime["resume_command"]) self.assertIn("{resume_session_dir}", pi.runtime["resume_command"])