<!-- task=m-model-serving-load-routing/03+01,02_e2e_field_validation plan=0 tag=OE2E-E2E -->

# Ollama E2E Smoke And Field Validation Plan

## 이 파일을 읽는 구현 에이전트에게

구현 후 반드시 같은 디렉터리의 `CODE_REVIEW-*-G??.md`에서 구현 에이전트 소유 섹션을 실제 구현 내용과 검증 출력으로 채운다. 검증 명령을 실행하고 실제 stdout/stderr를 남긴다. active 파일은 그대로 두고 리뷰 준비 상태를 보고한다. 최종 판정, log rename, `complete.log`, archive 이동은 code-review 전용이다.

## 배경

보조 fake Ollama smoke는 OpenAI-compatible 입력 표면과 edge-node relay의 최소 생존을 빠르게 확인한다. 하지만 마일스톤 완료 기준은 사용자가 직접 테스트하는 방식과 같은 `bin/edge.sh`, 배포된 Node binary, 실제 `toki-labs` Ollama `gemma4:26b` full-cycle이다. Node/Edge reasoning support가 들어간 뒤 보조 smoke와 field validation 모두 thinking/content 분리를 확인해야 한다.

## 분석 결과

### 읽은 파일

- `scripts/e2e-openai-ollama.sh`
- `docs/deploy-dev.md`
- `apps/node/internal/adapters/ollama/ollama.go`
- `apps/node/internal/adapters/ollama/ollama_test.go`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/server_test.go`
- `apps/node/internal/runtime/types.go`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-ops/skills/project/e2e-smoke/SKILL.md`
- `agent-ops/roadmap/phase/serving-routing-optimization/milestones/model-serving-load-routing.md`

### 테스트 커버리지 공백

- `scripts/e2e-openai-ollama.sh` currently checks content and `[DONE]`, not reasoning passthrough.
- Field environment validity was manually checked: SSH works, Ollama `/api/tags`, `/api/show`, non-streaming and streaming `/api/chat` work. Remote host lacks `go` and `iop-node` in PATH, so field validation needs a built Node binary.
- No automated command can prove remote Node can reach Edge until a reachable Edge host/address is selected.

### 심볼 참조

- renamed/removed symbol: none.
- Script output checks will add `reasoning_content` greps.

### 분할 판단

Split decision policy를 먼저 평가했다. 이 task depends on `01_node_reasoning_event` and `02+01_edge_reasoning_surface`, encoded by directory name `03+01,02_e2e_field_validation`. 보조 smoke script와 실제 field validation은 terminal/bin workflow risk가 커서 Node/Edge unit implementation tasks와 분리한다.

### 범위 결정 근거

- Node reasoning event implementation is excluded; task `01` owns it.
- Edge OpenAI response mapping is excluded; task `02` owns it.
- This task may update `scripts/e2e-openai-ollama.sh` and field validation notes only.
- `configs/*.yaml` must not be overwritten with field values.

### 빌드 등급

`cloud-G07` — bin script orchestration, process control, real remote SSH/Ollama validation, and field binary validation are terminal-agent work.

## 구현 체크리스트

- [ ] `01_node_reasoning_event` and `02+01_edge_reasoning_surface` changes are present or complete.
- [ ] fake Ollama emits thinking chunks before content chunks.
- [ ] `scripts/e2e-openai-ollama.sh` asserts `reasoning_content`, content chunk, and `data: [DONE]`.
- [ ] local auxiliary smoke passes via `make test-openai-ollama`.
- [ ] build a `darwin/arm64` `iop-node` binary for `toki-labs` or record an explicit blocker.
- [ ] run or document the actual field full-cycle with `toki-labs` Ollama `gemma4:26b`.
- [ ] 중간 및 최종 검증 명령을 실행한다.
- [ ] CODE_REVIEW-*-G??.md의 구현 에이전트 소유 섹션을 실제 구현 내용과 검증 출력으로 채운다. 이 항목이 완료되기 전에는 구현이 완료된 것이 아니다.

## 의존 관계 및 구현 순서

이 디렉터리 이름 `03+01,02_e2e_field_validation`은 같은 task group의 `01_node_reasoning_event`와 `02+01_edge_reasoning_surface`를 predecessor로 가진다. 두 predecessor의 `complete.log`가 있거나 그 구현이 현재 브랜치에 통합된 상태에서 시작한다.

### [OE2E-E2E-1] Auxiliary Fake Ollama Reasoning Smoke

#### 문제

[e2e-openai-ollama.sh](/config/workspace/iop/scripts/e2e-openai-ollama.sh:77)는 fake Ollama가 content chunks만 반환한다. [e2e-openai-ollama.sh](/config/workspace/iop/scripts/e2e-openai-ollama.sh:189)는 streaming response에서 content and `[DONE]`만 확인한다. reasoning passthrough regression을 잡지 못한다.

Before:

```bash
77  w.Header().Set("Content-Type", "application/x-ndjson")
78  _, _ = w.Write([]byte(`{"message":{"role":"assistant","content":"IOP_OPENAI_"},"done":false}` + "\n"))
...
82  _, _ = w.Write([]byte(`{"message":{"role":"assistant","content":"OLLAMA_OK"},"done":false}` + "\n"))
```

Before:

```bash
189 STREAM_OUT="$TMP_DIR/stream.txt"
190 curl -fsS -N \
...
194 grep -q '"content":"IOP_OPENAI_"' "$STREAM_OUT"
195 grep -q 'data: \[DONE\]' "$STREAM_OUT"
```

#### 해결 방법

Fake Ollama should emit at least one `message.thinking` chunk before content. Non-streaming Edge response should be checked for `reasoning_content` and final content. Streaming response should be checked for `"reasoning_content":"..."`, `"content":"IOP_OPENAI_"`, and `data: [DONE]`.

#### 수정 파일 및 체크리스트

- [ ] `scripts/e2e-openai-ollama.sh` fake Ollama emits thinking chunk
- [ ] `scripts/e2e-openai-ollama.sh` non-streaming response greps `reasoning_content`
- [ ] `scripts/e2e-openai-ollama.sh` streaming response greps `reasoning_content`, content, and `[DONE]`

#### 테스트 작성

Script update is the regression check. No separate Go test in this task.

#### 중간 검증

```bash
make test-openai-ollama
```

예상: pass. This command starts fake Ollama, Edge via `bin/edge.sh`, Node via `bin/node.sh`, calls `/v1/models`, non-streaming chat, streaming chat.

### [OE2E-E2E-2] Real Field Binary Validation

#### 문제

Testing environment is valid at the Ollama HTTP layer, but remote host does not have `go` or `iop-node` in PATH. User intends to test with an actual built Node. Final validation must not rely on fake smoke only.

Known environment:

- Node host: `ssh toki@toki-labs.com`
- Ollama base URL: `http://192.168.0.97:11434`
- Model: `gemma4:26b`
- Remote OS: Darwin arm64

#### 해결 방법

Build a darwin/arm64 `iop-node` binary from this repo and deploy it to the remote host. Start Edge on a host reachable from `toki-labs`; start remote Node with an `edge_addr` pointing to that Edge. Call Edge `/v1/models`, non-streaming chat, and streaming chat. Confirm reasoning chunks and content chunks are separated.

Commands are exact except `EDGE_HOST` and `FIELD_TOKEN`, which must be set to actual test values:

```bash
GOOS=darwin GOARCH=arm64 go build -trimpath -o /tmp/iop-node-darwin-arm64 ./apps/node/cmd/node
```

```bash
scp /tmp/iop-node-darwin-arm64 toki@toki-labs.com:/tmp/iop-node
```

```bash
ssh toki@toki-labs.com 'chmod +x /tmp/iop-node && curl -fsS --connect-timeout 5 --max-time 10 http://192.168.0.97:11434/api/tags | grep -F "\"name\":\"gemma4:26b\""'
```

Before starting remote Node, verify reachability from remote host. Replace `<reachable-edge-host>` with the actual Edge host/IP:

```bash
EDGE_HOST=<reachable-edge-host>
ssh toki@toki-labs.com "nc -vz ${EDGE_HOST} 9090"
```

If `EDGE_HOST` is not reachable, record blocker in `CODE_REVIEW-cloud-G07.md` instead of claiming field validation passed.

#### 수정 파일 및 체크리스트

- [ ] No default `configs/*.yaml` field values are committed.
- [ ] `docs/deploy-dev.md` remains the source of field env instructions.
- [ ] `CODE_REVIEW-cloud-G07.md` records actual `EDGE_HOST`, token handling without exposing secret token, command outputs, and pass/blocker state.

#### 테스트 작성

No automated test file. This is field validation using a real external Ollama host and built Node binary.

#### 중간 검증

```bash
ssh toki@toki-labs.com 'curl -fsS --connect-timeout 5 --max-time 10 http://192.168.0.97:11434/api/tags | grep -F "\"name\":\"gemma4:26b\""'
```

예상: output contains `"name":"gemma4:26b"`.

## 수정 파일 요약

| 파일 | 항목 |
|---|---|
| `scripts/e2e-openai-ollama.sh` | OE2E-E2E-1 |
| `docs/deploy-dev.md` | OE2E-E2E-2, only if field guide needs small corrections |
| `CODE_REVIEW-cloud-G07.md` | OE2E-E2E-2 verification record |

## 최종 검증

```bash
go test -count=1 ./apps/node/internal/adapters/ollama ./apps/edge/internal/openai
```

예상: pass. Go test cache output은 허용하지 않는다.

```bash
make test-openai-ollama
```

예상: pass and output includes `OpenAI-compatible Ollama serving test PASSED.`

```bash
GOOS=darwin GOARCH=arm64 go build -trimpath -o /tmp/iop-node-darwin-arm64 ./apps/node/cmd/node
```

예상: exit 0 and `/tmp/iop-node-darwin-arm64` exists.

Field full-cycle expected outcome: remote `iop-node` registers to Edge, Edge `/v1/models` includes `gemma4:26b`, non-streaming response includes content and separate `reasoning_content` when model emits thinking, streaming SSE includes `reasoning_content`, `content`, and `data: [DONE]`. If `EDGE_HOST` reachability is unavailable, record blocker with the failed `nc -vz` output.

모든 코드 변경 완료 후 반드시 `CODE_REVIEW-*-G??.md`의 구현 에이전트 소유 섹션을 채운다. 이 파일 작성이 구현의 마지막 단계다.
