toki/proto-socket
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
proto-socket/PROTOCOL.md

17 KiB
Raw Blame History

Proto Socket Protocol

Binary TCP and WebSocket protocol using Protocol Buffers for message framing and type-based routing. Designed for bidirectional, heterogeneous communication across languages and platforms.

Current protocol version: 0.1

Scope

This protocol intentionally defines only the transport-level contract shared across implementations.

  • In scope: TCP/WebSocket framing, protobuf payload transport, typeName routing, nonce, responseNonce, TLS/WSS transport variants, and heartbeat
  • Out of scope: authentication, session lifecycle, agent semantics, chat semantics, game rules, room logic, and other application-specific behavior
  • Higher-level concerns should be implemented as messages and conventions on top of this protocol, not inside the protocol core

Transport별 Wire Format

TCP / SSL+TCP

┌──────────────────────────────────────────┐
│  Header  (4 bytes, big-endian integer)   │  ← PacketBase payload length
├──────────────────────────────────────────┤
│  PacketBase  (protobuf, N bytes)         │  ← typeName + nonce + data
└──────────────────────────────────────────┘

Header

  • 4 bytes, big-endian positive integer. Dart reads/writes it as int32; Go reads/writes the same 4 bytes as uint32.
  • Value: byte length of the following PacketBase protobuf payload
  • Value of 0: reserved / no-op, receiver clears buffer
  • Implementations may apply safety limits to payload size. The current Dart, Go, and Kotlin implementations reject TCP packets larger than 64 MiB.

WebSocket / WSS

┌──────────────────────────────────────────┐
│  PacketBase  (protobuf, N bytes)         │  ← typeName + nonce + data
└──────────────────────────────────────────┘
  • Length 헤더 없음 — WebSocket 프로토콜이 메시지 경계를 보장한다
  • Binary frame 사용 (text frame 아님)
  • 연결: ws://host:port/path (plain) / wss://host:port/path (TLS)

PacketBase

message PacketBase {
  string typeName      = 1;   // protobuf full message name used for routing
  int32  nonce         = 2;   // monotonically increasing per-connection counter
  bytes  data          = 3;   // serialized inner message bytes
  int32  responseNonce = 4;   // > 0: this packet is a response to request with that nonce
}
Field Description
typeName Protobuf full message name used as the language-agnostic routing key
nonce Starts at 1, increments by 1 per send call per connection
data Protobuf-serialized bytes of the inner message
responseNonce 0 for regular messages. Set to the request's nonce when sending a response

Built-in Messages

HeartBeat

message HeartBeat {}

Automatically handled by the framework. Applications do not need to register or handle this manually.

Heartbeat Lifecycle

Side A                                    Side B
  │                                          │
  │── (no activity for intervalTime) ──────▶│
  │        send(HeartBeat)                   │
  │                                          │── onHeartBeat() called
  │                                          │   send(HeartBeat) [response]
  │◀─────────────────────────────────────────│
  │  onHeartBeat() called                    │
  │  _waitingHeartbeatResponse = false       │
  │  timer reset                             │
  • After any received message → heartbeat interval timer resets
  • If no data received within heartbeatIntervalTime seconds → send HeartBeat
  • If no response within heartbeatWaitTime seconds → onDisconnected() → dispose()
  • A side that is already waiting for a heartbeat response consumes the echo and does not send another heartbeat. A side that is not waiting replies with HeartBeat.

Changing heartbeat timing semantics, response behavior, or disconnect rules is a breaking protocol candidate. Document the compatibility plan in VERSIONING.md before changing it.

Optional Capability Announcement

Implementations may define an application-level ProtoSocketHello message, or an equivalent project-specific hello message, to announce protocol and implementation capabilities after connection open.

This message is optional and non-blocking in protocol 0.1:

  • It is a normal PacketBase payload sent through the existing typeName routing path.
  • It must not be required before normal messages, requests, responses, or heartbeats can flow.
  • Peers that do not recognize its typeName may ignore it according to their normal unknown-message policy.
  • Implementations must not treat a missing hello message as a protocol error in 0.1.

Recommended payload fields:

message ProtoSocketHello {
  string protocolVersion = 1;        // e.g. "0.1"
  repeated string capabilities = 2;  // stable capability tokens
  map<string, string> implementation = 3; // optional language/library metadata
}

Capability tokens should be stable ASCII strings such as std_error.v1 or ws.wss. Application-specific tokens should use a project prefix to avoid collisions.

Promoting this message to a required handshake is a future protocol-version decision and is not part of protocol 0.1.

typeName Convention

typeName uses the protobuf full message name on the send side. For messages in the common proto package, the canonical values are proto_socket.PacketBase, proto_socket.HeartBeat, and proto_socket.TestData; PacketBase.typeName normally carries the inner message name such as proto_socket.HeartBeat or proto_socket.TestData. On the receive side, use the same canonical value as the registration key.

Proto package names are part of the wire identity. Package names must use lowercase snake_case segments joined by dots, for example proto_socket or example_chat.v1. Language-specific namespace options such as Go go_package or Java/Kotlin package names do not change the wire typeName.

The canonical package for proto/message_common.proto is proto_socket.

Language Registration key
Dart protobuf descriptor/native full name when available, otherwise static metadata aligned with the canonical full name
Go string(proto.MessageName(m)) via TypeNameOf(m)
C# protobuf descriptor full name — verify matches the canonical full name
Kotlin descriptorForType.fullName via typeNameOf(m)
Swift Message.protoMessageName or equivalent descriptor full name — verify matches the canonical full name
TypeScript local MessageType.typeName value — verify matches the canonical full name
Python MessageClass.DESCRIPTOR.full_name

Implementations may accept legacy simple-name values such as TestData and HeartBeat for receive compatibility, but new sends must use the canonical full proto name once the implementation supports this convention.

Important: Verify typeName consistency across languages before connecting heterogeneous clients.

Changing the proto package or typeName derivation rule is a breaking protocol change unless all released implementations keep a compatibility path. See VERSIONING.md.

nonce

  • Starts at 1 per connection
  • Increments by 1 on every send call (including requests and responses)
  • Maximum nonce is 2,147,483,647 (int32 max), the lowest common signed integer max across supported protocol fields
  • After issuing 2,147,483,647, implementations reset the internal counter to 0; the next emitted packet nonce is 1
  • 0 is reserved and must not be emitted as PacketBase.nonce, because responseNonce == 0 means "not a response"
  • Used for request-response correlation via responseNonce

Changing nonce or responseNonce semantics is a breaking protocol change unless older peers can continue to correlate requests and responses correctly.

Request-Response Pattern

Fire-and-forget (send) and request-response (sendRequest) coexist on the same connection.

Requester                                 Responder
  │                                          │
  │── PacketBase { nonce=5, responseNonce=0, │
  │                typeName="GetUser",       │
  │                data=... } ─────────────▶│
  │                                          │── addRequestListener<GetUser, UserData>
  │                                          │   handler called → returns UserData
  │◀─ PacketBase { nonce=12, responseNonce=5,│
  │                typeName="UserData",      │
  │                data=... } ──────────────│
  │  sendRequest Future completes            │

Rules:

  • responseNonce == 0: regular message or outgoing request → routed to addListener or addRequestListener
  • responseNonce > 0: response → matched to the pending sendRequest by responseNonce
  • A response must also have the expected typeName for the waiting sendRequest; mismatches are protocol errors
  • Both sides can be requester and responder simultaneously on the same connection
  • For a given typeName on one connection, register it with either addListener or addRequestListener, not both. Mixed registration is ambiguous and rejected by the Dart and Go implementations.

Standard Error Response Convention

Implementations may define a ProtoSocketError message, or an equivalent project-specific error message, for request handlers that fail before producing the expected response.

Recommended payload fields:

message ProtoSocketError {
  string code = 1;     // stable machine-readable error code
  string message = 2;  // safe human-readable message
  string detail = 3;   // optional non-sensitive detail
  string debug = 4;    // optional local/debug-only detail
}

When used as a request failure response:

  • The packet's responseNonce should be set to the failing request's nonce.
  • The packet's typeName should identify the error message, for example proto_socket.ProtoSocketError if this becomes a shared common message in a future schema.
  • A requester that understands the convention should complete the pending request with a typed error instead of waiting for timeout.
  • A requester that does not understand the convention may see an expected-response type mismatch or may continue to timeout, depending on its current response handling.

Protocol 0.1 does not require every request handler failure to send an error response. Adding a common error message to the canonical proto schema requires parser map updates and cross-language tests before it can become a shared compatibility contract.

Receive Ordering and Backpressure

Inbound Queue

Each connection owns one bounded FIFO inbound queue between the frame parser and stateful dispatch.

  • The read loop parses complete frames and enqueues packet envelopes in arrival order.
  • Enqueue order is the authoritative dispatch order; implementations must not reorder enqueued packets before dispatch.
  • Recommended default capacity: 64 packets. Implementations may choose a different bound but must document it.
  • When the inbound queue is full, the implementation applies backpressure by suspending frame reads from the transport until space is available. The underlying TCP flow control propagates this pressure to the sender.
  • WebSocket Runtime Limitations:
    • Some WebSocket client runtimes (such as Browser WebSocket and Kotlin OkHttp WebSocket) do not support transport-level pause/resume (suspending frame reads).
    • To prevent unbounded promise/coroutine staging backlog under heavy load, these runtimes must implement a Bounded Input Gate with a documented limit (e.g., 64 packets max staging boundary).
    • If the input gate capacity is exceeded (i.e. the dispatcher queue is full and the staging buffer overflows the limit), the runtime must apply a strict discard/backpressure policy: disconnect the connection (forcefully close / destroy) immediately to protect resources from memory exhaustion.

Receive Coordinator

A single receive coordinator dequeues packets one at a time and dispatches them:

  1. If responseNonce > 0: match to the pending sendRequest and complete the waiting future/callback.
  2. If responseNonce == 0 and the typeName is registered with addRequestListener: invoke the handler, then enqueue the automatic response through the write queue in the same coordinator loop iteration.
  3. If responseNonce == 0 and the typeName is registered with addListener: invoke the listener.
  4. HeartBeat is handled internally before any listener dispatch.

Ordering guarantee: for a given connection, listeners and request handlers are invoked strictly in packet arrival order. Automatic responses are enqueued into the write queue in the same order their triggering requests were dispatched.

Response Correlation Path

Packets with responseNonce > 0 complete an already-pending request and do not invoke listeners or request handlers. Implementations may therefore complete those pending requests through a direct correlation path before or instead of the listener/request-handler FIFO queue.

This response path does not weaken the listener/request-handler ordering guarantee: packets that dispatch to addListener or addRequestListener must still be processed in arrival order, and automatic responses produced by request handlers must still be enqueued in request-dispatch order. Implementations that enqueue response packets in the same inbound queue must produce equivalent pending-request completion semantics.

Close, Drain, and Cancel

  • When Close() is called, the implementation stops accepting new inbound packets and drains the already-enqueued packets in the inbound queue before tearing down the receive coordinator.
  • In-flight sendRequest futures that have not yet received a response are cancelled with an error when the connection closes.
  • The inbound queue is not drained after Close() if the transport signals an error (e.g., read error, heartbeat timeout). In that case the queue is discarded immediately.
  • Implementations must ensure Close() is idempotent (see connCloseOnce pattern in PORTING_GUIDE.md).

Worker Gateway and Internal seq

  • Implementations may introduce an optional parallel worker gateway for decoding or preprocessing large/burst packets.
  • In such multi-worker designs, the gateway input/output carries an internal seq assigned by the read loop to track the arrival order.
  • The seq sequence is purely an internal implementation detail and NOT part of the wire format (it is not added to PacketBase or the transport framing).
  • The gateway must perform only stateless operations (e.g., raw bytes decoding or preprocessing) and is prohibited from possessing stateful dispatcher logic.
  • The receive coordinator remains the sole owner of ordering restoration (sorting by seq), pending response matching, listener/request handler dispatching, and automated response enqueuing.

Example Packet (hex)

Sending HeartBeat {}:

00 00 00 18          ← header: PacketBase length = 24 bytes
0A 16 70 72 6F 74 6F 5F 73 6F 63 6B 65 74 2E 48 65 61 72 74 42 65 61 74
  ← PacketBase { typeName: "proto_socket.HeartBeat" }

Multi-language Implementations

Language Status Path
Dart Available dart/
C# (Unity) Planned csharp/
Kotlin Available kotlin/
Swift Planned swift/
Go Available go/
TypeScript Available typescript/
Python Available python/

Proto Source

proto/message_common.proto is the canonical packet definition. All language implementations generate bindings from it.

The canonical package for this proto is proto_socket. New application protos must choose package names using lowercase snake_case segments joined by dots, and those package names define the wire typeName prefix.

Language-specific generation options such as go_package and Java/Kotlin package options are kept in each language's own proto copy under go/packets/ and kotlin/src/main/proto/. These options affect generated code namespaces only; they do not change the protobuf package or wire identity. Keep those copies' message schemas in sync with proto/message_common.proto before regenerating language bindings.