# Runner heartbeat timeout runtime loop

## Metadata

- Status: planned
- Task group: `control_plane_phase_exit_followups`
- Plan id: `cloud-G06`
- Review stub: `CODE_REVIEW-cloud-G06.md`
- Roadmap target: first phase exit follow-up for `agent-roadmap/phase/independent-control-plane/PHASE.md`

## Problem

`runnerregistry.Registry.CheckTimeouts` already marks stale runners offline, and unit tests cover the state transition. The running core server does not call it, so a registered runner can remain `online` forever after heartbeats stop.

This blocks closing the first phase with confidence because the control plane has the timeout rule but not the runtime responsibility that enforces it.

## Rules read

- `agent-ops/rules/project/rules.md`
- `agent-ops/rules/project/domain/agent/rules.md`
- `agent-ops/rules/project/domain/framework/rules.md`
- `agent-test/local/rules.md`
- `agent-test/local/core-smoke.md`

## Current evidence

- `services/core/internal/runnerregistry/registry.go`
  - `CheckTimeouts(timeoutDuration time.Duration)` transitions online runners to `offline`.
  - `Heartbeat` records `LastHeartbeatAt`.
- `services/core/internal/runnerregistry/registry_test.go`
  - Registry-level timeout behavior is covered.
- `services/core/internal/httpserver/server.go`
  - `Server.Start` only starts `ListenAndServe`.
  - `Server.Shutdown` only shuts down the HTTP server.
- `services/core/cmd/oto-core/main.go`
  - The process owns startup/shutdown but does not start a registry maintenance loop.

## Scope

Implement a core-owned background loop that periodically calls `Registry.CheckTimeouts` while the server is running, and stops cleanly during shutdown.

Do not change runner protocol messages, registration compatibility rules, or job state semantics in this task.

## Proposed implementation

1. Add timeout configuration to the core HTTP server boundary.
   - Default heartbeat timeout should be conservative, for example `90s`.
   - Default scan interval should be shorter than the timeout, for example `30s`.
   - Tests should be able to configure small durations without sleeping for production intervals.

2. Start the timeout loop from `Server.Start`.
   - Use a `time.Ticker` and server-owned cancellation channel/context.
   - Call `runnerregistry.Registry.CheckTimeouts` on each tick.
   - Ensure the loop is not started when the registry is nil.

3. Stop the loop from `Server.Shutdown`.
   - Shutdown should unblock the loop before or during HTTP shutdown.
   - Multiple shutdown paths should be idempotent enough for tests and normal process exit.

4. Add tests around the server runtime, not only the registry.
   - Register a runner through the HTTP route.
   - Configure a short timeout/interval.
   - Assert that status eventually reports offline after heartbeats stop.
   - Assert shutdown stops the loop without leaking or hanging.

## Expected files

- `services/core/internal/httpserver/server.go`
- `services/core/internal/httpserver/server_test.go`
- Possibly `services/core/cmd/oto-core/main.go` if defaults are wired from main rather than constructor defaults.

## Verification

- `cd services/core && go test -count=1 ./...`
- If command wiring changes: run `go test` from the same module after formatting.

## Risks

- A too-short default may mark slow or temporarily busy runners offline.
- A loop that does not stop cleanly can make tests flaky or delay process shutdown.
- Timeout state must not mutate job/execution terminal state; this task should only affect runner online/offline state.

## Done when

- Stale runners transition offline in the running core process without an external caller.
- The behavior is covered by HTTP/server-level tests.
- Core tests pass locally.
