proto-socket/README.md

# Proto Socket

Binary socket protocol library for bidirectional, heterogeneous communication across languages and platforms.

Built on Protocol Buffers with TCP length-prefixed framing, WebSocket binary frames, type-based message routing, request-response correlation, and built-in heartbeat.

---

## Current Status

Proto Socket currently treats the Dart, Go, Kotlin, Python, and TypeScript implementations as the available compatibility set for protocol `0.1`.

The project is in a stabilization and maintenance pass before adding more language targets or publishing package registry releases. Git-based consumption from this repository is the primary supported workflow for now. C# and Swift remain planned targets, but implementation work is deferred until there is concrete demand.

Use the local validation matrix as the stability gate for protocol or implementation changes:

```bash
bash agent-ops/skills/project/run-proto-socket-test-matrix/scripts/run_matrix.sh --all
```

Protocol or public API changes should be treated as compatibility work: update the relevant docs, regenerate protobuf bindings when needed, and keep the full validation matrix passing.

---

## Design Principles

- Keep the core transport layer thin and stable
- Provide only the minimum common foundation for cross-language communication
- Standardize framing, serialization, routing, request-response correlation, and heartbeat
- Treat Protocol Buffers as the core protocol dependency. Other runtime dependencies should be native-first, narrowly scoped, and proportional to the feature they implement
- Do not embed application semantics into the protocol core
- Domain-specific concerns such as auth, session, agent workflow, chat features, and game logic belong in upper-layer implementations

---

## Protocol

See [PROTOCOL.md](PROTOCOL.md) for the full wire format specification.

```
[4-byte big-endian length] [PacketBase protobuf bytes]
```

For WebSocket/WSS transports, each binary frame contains one `PacketBase` protobuf payload without the TCP length header.

Protocol compatibility is tracked separately from language package versions. See [VERSIONING.md](VERSIONING.md). Even while packages are not published, the checked-in implementations must preserve the documented protocol contract.

---

## Implementations

| Language | Status | Path | Use case |
|----------|--------|------|----------|
| Dart | Available | [dart/](dart/) | Flutter, Dart server |
| Kotlin | Available | [kotlin/](kotlin/) | Android, JVM |
| Go | Available | [go/](go/) | Server, tooling, scripting |
| TypeScript | Available | [typescript/](typescript/) | Browser, Node.js |
| Python | Available | [python/](python/) | Server, tooling, scripting |
| C# | Planned / deferred | `csharp/` | Unity, .NET |
| Swift | Planned / deferred | `swift/` | iOS, macOS |

New language implementations should start from [PORTING_GUIDE.md](PORTING_GUIDE.md) and the templates in [agent-ops/skills/project/add-proto-socket-crosstest-language/templates/](agent-ops/skills/project/add-proto-socket-crosstest-language/templates/). Mark an implementation available only after its same-language tests and cross-language tests pass.

Keep implementation status here as the public snapshot. Current work context is selected through [agent-ops/roadmap/current.md](agent-ops/roadmap/current.md), and broader roadmap direction lives in [agent-ops/roadmap/ROADMAP.md](agent-ops/roadmap/ROADMAP.md).

---

## Common Commands

| Purpose | Command | Notes |
|---------|---------|-------|
| Full local validation | `bash agent-ops/skills/project/run-proto-socket-test-matrix/scripts/run_matrix.sh --all` | Proto sync, same-language tests, and all available cross-language checks |
| Proto schema sync check | `tools/check_proto_sync.sh` | Verifies language proto copies match `proto/message_common.proto` except allowed language options |
| Regenerate protobuf bindings | `tools/generate_proto.sh` | Run after changing the canonical proto schema |
| Dart tests | `cd dart && dart pub get && dart test && dart compile js test/browser_ws_import_compile.dart -o /tmp/proto_socket_browser_ws_import_compile.js` | Dart VM/IO tests and web import compile check |
| Go tests | `cd go && go test ./...` | Go same-language checks |
| Kotlin tests | `cd kotlin && ./gradlew test` | Kotlin same-language checks |
| Python tests | `cd python && python3 -m pytest -q` | Python same-language checks |
| TypeScript checks | `cd typescript && npm run check && npm test` | TypeScript type check and tests |

---

## Repository Structure

| Path | Role |
|------|------|
| [proto/](proto/) | Canonical protobuf schema source |
| [dart/](dart/) | Dart / Flutter implementation and tests |
| [go/](go/) | Go implementation, tests, and Go-led crosstests |
| [kotlin/](kotlin/) | Kotlin / JVM implementation and crosstests |
| [python/](python/) | Python implementation and crosstests |
| [typescript/](typescript/) | TypeScript implementation and crosstests |
| [tools/](tools/) | Proto generation and sync helpers |
| [examples/](examples/) | Consumer-oriented examples |
| [agent-ops/](agent-ops/) | Project rules, roadmap, and repeatable agent workflows |

---

## Quick Start (Dart)

```dart
import 'package:proto_socket/proto_socket.dart';

// 1. Define your message in message_common.proto, generate with protoc

// 2. Implement a client
class MyClient extends ProtobufClient {
  MyClient(Socket socket) : super(socket, 30, 10, {
    MyMessage.getDefault().info_.qualifiedMessageName: MyMessage.fromBuffer,
  });
}

// 3. Implement a server
class MyServer extends ProtobufServer {
  MyServer() : super('0.0.0.0', 9090, (socket) => MyClient(socket));

  @override
  void onClientConnected(ProtobufClient client) {
    client.addListener<MyMessage>((msg) => print('Received: ${msg}'));
  }
}

// 4. Start
final server = MyServer();
await server.start();

// 5. Connect and send
final socket = await Socket.connect('localhost', 9090);
final client = MyClient(socket);
await client.send(MyMessage()..text = 'hello');
```

---

## Adding Message Types

Edit the canonical proto at `proto/message_common.proto`, then regenerate all checked-in bindings:

```bash
tools/generate_proto.sh
tools/check_proto_sync.sh
```

The Go and Kotlin proto copies are allowed to keep only language-specific options such as `option go_package` or Java package/class options. `tools/check_proto_sync.sh` fails with a diff when their message schema drifts from `proto/message_common.proto`.

---

## Quick Start (Go)

```go
package main

import (
	"context"
	"net"
	"time"

	"google.golang.org/protobuf/proto"

	protoSocket "git.toki-labs.com/toki/proto-socket/go"
	"git.toki-labs.com/toki/proto-socket/go/packets"
)

func parserMap() protoSocket.ParserMap {
	return protoSocket.ParserMap{
		protoSocket.TypeNameOf(&packets.TestData{}): func(b []byte) (proto.Message, error) {
			m := &packets.TestData{}
			return m, proto.Unmarshal(b, m)
		},
	}
}

func main() {
	ctx := context.Background()

	server := protoSocket.NewTcpServer("127.0.0.1", 9090, func(conn net.Conn) *protoSocket.TcpClient {
		return protoSocket.NewTcpClient(conn, 30, 10, parserMap())
	})
	server.OnClientConnected = func(client *protoSocket.TcpClient) {
		protoSocket.AddRequestListenerTyped[*packets.TestData, *packets.TestData](
			&client.Communicator,
			func(req *packets.TestData) (*packets.TestData, error) {
				return &packets.TestData{Index: req.GetIndex(), Message: "echo: " + req.GetMessage()}, nil
			},
		)
	}
	if err := server.Start(ctx); err != nil {
		panic(err)
	}
	defer server.Stop()

	client, err := protoSocket.DialTcp(ctx, "127.0.0.1", 9090, 30, 10, parserMap())
	if err != nil {
		panic(err)
	}
	defer client.Close()

	res, err := protoSocket.SendRequestTyped[*packets.TestData, *packets.TestData](
		&client.Communicator,
		&packets.TestData{Index: 1, Message: "hello"},
		2*time.Second,
	)
	if err != nil {
		panic(err)
	}
	println(res.GetMessage())
}
```

Go also provides:

- TCP: `NewTcpServer`, `DialTcp`, `NewTcpServerTLS`, `DialTcpTLS`
- WebSocket: `NewWsServer`, `DialWs`, `NewWsServerTLS`, `DialWss`
- Shared helpers: `Send`, `SendRequest`, `AddListenerTyped`, `AddRequestListenerTyped`, `Broadcast`

If you want a copyable scaffold for using this module from another repository, see [examples/go-module-consumer/](examples/go-module-consumer/).

---

## Running Tests

Local commands are documented here for development and troubleshooting. The repeatable repository-local validation entry point is:

```bash
bash agent-ops/skills/project/run-proto-socket-test-matrix/scripts/run_matrix.sh --all
```

It runs proto schema sync, same-language tests, and cross-language checks, then prints PASS/FAIL tables with reproduction commands for failures. Use `--proto`, `--unit`, or `--cross` to run a smaller slice.

External CI runner integration is deferred until the parent `../oto` project standardizes its execution path. Until then, the local matrix above is the project stability gate.

Proto schema sync:

```bash
tools/check_proto_sync.sh
```

Same-language checks:

```bash
cd dart
dart pub get
dart test
dart compile js test/browser_ws_import_compile.dart -o /tmp/proto_socket_browser_ws_import_compile.js
```

```bash
cd go
go test ./...
```

```bash
cd kotlin
./gradlew test
```

```bash
cd python
python3 -m pytest -q
```

```bash
cd typescript
npm run check
npm test
```

Cross-language checks:

```bash
cd go
go run ./crosstest/go_dart.go
go run ./crosstest/go_kotlin.go
go run ./crosstest/go_python.go
go run ./crosstest/go_typescript.go
```

```bash
cd dart
dart run crosstest/dart_go.dart
dart run crosstest/dart_kotlin.dart
dart run crosstest/dart_python.dart
dart run crosstest/dart_typescript.dart
```

```bash
cd kotlin
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinDartKt
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinGoKt
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinPythonKt
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinTypescriptKt
```

```bash
cd python
python3 crosstest/python_dart.py
python3 crosstest/python_go.py
python3 crosstest/python_kotlin.py
python3 crosstest/python_typescript.py
```

```bash
cd typescript
./node_modules/.bin/tsx crosstest/typescript_dart.ts
./node_modules/.bin/tsx crosstest/typescript_go.ts
./node_modules/.bin/tsx crosstest/typescript_kotlin.ts
./node_modules/.bin/tsx crosstest/typescript_python.ts
```

### Dart.io vs Dart.web E2E matrix

The `Dart` column above refers to the Dart VM/IO client. `Dart.web` is a separate axis: a browser WS client only — TCP and any server role are not possible in the browser, so those cells are `N/A`. WSS is `Deferred` because there is no automation for trusting the self-signed test certificate in a browser.

| Server | Dart.io TCP | Dart.io WS | Dart.io TLS TCP | Dart.io WSS | Dart.web WS | Dart.web WSS | Dart.web TCP |
|---|---|---|---|---|---|---|---|
| Dart.io | Covered | Covered | Covered | Covered | Covered | Deferred | N/A |
| Go | Covered | Covered | Covered | Covered | Covered | Deferred | N/A |
| Kotlin | Covered | Covered | Covered | Covered | Covered | Deferred | N/A |
| Python | Covered | Covered | Covered | Covered | Covered | Deferred | N/A |
| TypeScript | Covered | Covered | Covered | Covered | Covered | Deferred | N/A |

Each Dart.web cell is exercised by a server-language runner that boots its WS server, launches `dart test -p chrome` against the matching browser test file, and asserts the send-push and request-response scenarios:

```bash
cd dart && dart run crosstest/dart_web.dart
cd go && go run ./crosstest/go_dart_web.go
cd kotlin && ./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinDartWebKt
cd python && python3 crosstest/python_dart_web.py
cd typescript && ./node_modules/.bin/tsx crosstest/typescript_dart_web.ts
```

---

## Work Context

Before making code or protocol changes, check the project rules in [agent-ops/rules/project/rules.md](agent-ops/rules/project/rules.md) and any matching domain rule listed there. The active roadmap window is in [agent-ops/roadmap/current.md](agent-ops/roadmap/current.md).

Stabilization boundaries:

- Keep protocol `0.1` behavior stable unless compatibility work is explicitly requested.
- Maintenance-mode work may include bug fixes, documentation corrections, test hardening, and compatibility-preserving implementation fixes.
- Resume C#/Swift, package registry release, or protocol/API change work only when there is concrete consumer demand, a compatibility plan, and a passing full validation matrix.
- Do not connect external CI/CD runners from this repository until the parent `../oto` workflow is decided.