toki/proto-socket
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
proto-socket/README.md

7.1 KiB
Raw Blame History

Proto Socket

Binary socket protocol library for bidirectional, heterogeneous communication across languages and platforms.

Built on Protocol Buffers with TCP length-prefixed framing, WebSocket binary frames, type-based message routing, request-response correlation, and built-in heartbeat.

Design Principles

  • Keep the core transport layer thin and stable
  • Provide only the minimum common foundation for cross-language communication
  • Standardize framing, serialization, routing, request-response correlation, and heartbeat
  • Treat Protocol Buffers as the core protocol dependency. Other runtime dependencies should be native-first, narrowly scoped, and proportional to the feature they implement
  • Do not embed application semantics into the protocol core
  • Domain-specific concerns such as auth, session, agent workflow, chat features, and game logic belong in upper-layer implementations

Protocol

See PROTOCOL.md for the full wire format specification.

[4-byte big-endian length] [PacketBase protobuf bytes]

For WebSocket/WSS transports, each binary frame contains one PacketBase protobuf payload without the TCP length header.

Protocol compatibility is tracked separately from language package versions. See VERSIONING.md. Even while packages are not published, the checked-in implementations must preserve the documented protocol contract.

Implementations

Language Status Path Use case
Dart Available dart/ Flutter, Dart server
C# Planned csharp/ Unity, .NET
Kotlin Available kotlin/ Android, JVM
Swift Planned swift/ iOS, macOS
Go Available go/ Server, tooling, scripting
TypeScript Available typescript/ Browser, Node.js
Python Available python/ Server, tooling, scripting

New language implementations should start from PORTING_GUIDE.md and the templates in agent-ops/skills/project/add-proto-socket-crosstest-language/templates/. Mark an implementation available only after its same-language tests and cross-language tests pass.

Detailed planning now lives in agent-ops/roadmap/ROADMAP.md. Keep implementation status here as the public snapshot; roadmap direction, CI automation, and future C#/Swift work are tracked in the roadmap.

Quick Start (Dart)

import 'package:proto_socket/proto_socket.dart';

// 1. Define your message in message_common.proto, generate with protoc

// 2. Implement a client
class MyClient extends ProtobufClient {
  MyClient(Socket socket) : super(socket, 30, 10, {
    MyMessage.getDefault().info_.qualifiedMessageName: MyMessage.fromBuffer,
  });
}

// 3. Implement a server
class MyServer extends ProtobufServer {
  MyServer() : super('0.0.0.0', 9090, (socket) => MyClient(socket));

  @override
  void onClientConnected(ProtobufClient client) {
    client.addListener<MyMessage>((msg) => print('Received: ${msg}'));
  }
}

// 4. Start
final server = MyServer();
await server.start();

// 5. Connect and send
final socket = await Socket.connect('localhost', 9090);
final client = MyClient(socket);
await client.send(MyMessage()..text = 'hello');

Adding Message Types

Edit the canonical proto at proto/message_common.proto, then regenerate all checked-in bindings:

tools/generate_proto.sh
tools/check_proto_sync.sh

The Go and Kotlin proto copies are allowed to keep only language-specific options such as option go_package or Java package/class options. tools/check_proto_sync.sh fails with a diff when their message schema drifts from proto/message_common.proto.

Quick Start (Go)

package main

import (
	"context"
	"net"
	"time"

	"google.golang.org/protobuf/proto"

	protoSocket "git.toki-labs.com/toki/proto-socket/go"
	"git.toki-labs.com/toki/proto-socket/go/packets"
)

func parserMap() protoSocket.ParserMap {
	return protoSocket.ParserMap{
		protoSocket.TypeNameOf(&packets.TestData{}): func(b []byte) (proto.Message, error) {
			m := &packets.TestData{}
			return m, proto.Unmarshal(b, m)
		},
	}
}

func main() {
	ctx := context.Background()

	server := protoSocket.NewTcpServer("127.0.0.1", 9090, func(conn net.Conn) *protoSocket.TcpClient {
		return protoSocket.NewTcpClient(conn, 30, 10, parserMap())
	})
	server.OnClientConnected = func(client *protoSocket.TcpClient) {
		protoSocket.AddRequestListenerTyped[*packets.TestData, *packets.TestData](
			&client.Communicator,
			func(req *packets.TestData) (*packets.TestData, error) {
				return &packets.TestData{Index: req.GetIndex(), Message: "echo: " + req.GetMessage()}, nil
			},
		)
	}
	if err := server.Start(ctx); err != nil {
		panic(err)
	}
	defer server.Stop()

	client, err := protoSocket.DialTcp(ctx, "127.0.0.1", 9090, 30, 10, parserMap())
	if err != nil {
		panic(err)
	}
	defer client.Close()

	res, err := protoSocket.SendRequestTyped[*packets.TestData, *packets.TestData](
		&client.Communicator,
		&packets.TestData{Index: 1, Message: "hello"},
		2*time.Second,
	)
	if err != nil {
		panic(err)
	}
	println(res.GetMessage())
}

Go also provides:

  • TCP: NewTcpServer, DialTcp, NewTcpServerTLS, DialTcpTLS
  • WebSocket: NewWsServer, DialWs, NewWsServerTLS, DialWss
  • Shared helpers: Send, SendRequest, AddListenerTyped, AddRequestListenerTyped, Broadcast

If you want a copyable scaffold for using this module from another repository, see examples/go-module-consumer/.

Running Tests

Local commands are documented here for development and troubleshooting. Continuous verification automation is tracked in agent-ops/roadmap/ROADMAP.md.

cd dart
dart pub get
dart test

cd go
go test ./...

cd kotlin
./gradlew test

cd python
python3 -m pytest -q

cd typescript
npm run check
npm test

Cross-language checks:

cd go
go run ./crosstest/go_dart.go
go run ./crosstest/go_kotlin.go
go run ./crosstest/go_python.go
go run ./crosstest/go_typescript.go

cd dart
dart run crosstest/dart_go.dart
dart run crosstest/dart_kotlin.dart
dart run crosstest/dart_python.dart
dart run crosstest/dart_typescript.dart

cd kotlin
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinDartKt
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinGoKt
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinPythonKt
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinTypescriptKt

cd python
python3 crosstest/python_dart.py
python3 crosstest/python_go.py
python3 crosstest/python_kotlin.py
python3 crosstest/python_typescript.py

cd typescript
./node_modules/.bin/tsx crosstest/typescript_dart.ts
./node_modules/.bin/tsx crosstest/typescript_go.ts
./node_modules/.bin/tsx crosstest/typescript_kotlin.ts
./node_modules/.bin/tsx crosstest/typescript_python.ts

When proto files change, also run:

tools/check_proto_sync.sh