250 lines
7.4 KiB
Markdown
250 lines
7.4 KiB
Markdown
# Proto Socket
|
|
|
|
Binary socket protocol library for bidirectional, heterogeneous communication across languages and platforms.
|
|
|
|
Built on Protocol Buffers with TCP length-prefixed framing, WebSocket binary frames, type-based message routing, request-response correlation, and built-in heartbeat.
|
|
|
|
---
|
|
|
|
## Design Principles
|
|
|
|
- Keep the core transport layer thin and stable
|
|
- Provide only the minimum common foundation for cross-language communication
|
|
- Standardize framing, serialization, routing, request-response correlation, and heartbeat
|
|
- Treat Protocol Buffers as the core protocol dependency. Other runtime dependencies should be native-first, narrowly scoped, and proportional to the feature they implement
|
|
- Do not embed application semantics into the protocol core
|
|
- Domain-specific concerns such as auth, session, agent workflow, chat features, and game logic belong in upper-layer implementations
|
|
|
|
---
|
|
|
|
## Protocol
|
|
|
|
See [PROTOCOL.md](PROTOCOL.md) for the full wire format specification.
|
|
|
|
```
|
|
[4-byte big-endian length] [PacketBase protobuf bytes]
|
|
```
|
|
|
|
For WebSocket/WSS transports, each binary frame contains one `PacketBase` protobuf payload without the TCP length header.
|
|
|
|
Protocol compatibility is tracked separately from language package versions. See [VERSIONING.md](VERSIONING.md). Even while packages are not published, the checked-in implementations must preserve the documented protocol contract.
|
|
|
|
---
|
|
|
|
## Implementations
|
|
|
|
| Language | Status | Path | Use case |
|
|
|----------|--------|------|----------|
|
|
| Dart | Available | [dart/](dart/) | Flutter, Dart server |
|
|
| C# | Planned | `csharp/` | Unity, .NET |
|
|
| Kotlin | Available | kotlin/ | Android, JVM |
|
|
| Swift | Planned | `swift/` | iOS, macOS |
|
|
| Go | Available | [go/](go/) | Server, tooling, scripting |
|
|
| TypeScript | Available | [typescript/](typescript/) | Browser, Node.js |
|
|
| Python | Available | [python/](python/) | Server, tooling, scripting |
|
|
|
|
New language implementations should start from [PORTING_GUIDE.md](PORTING_GUIDE.md) and the templates in [agent-ops/skills/project/add-proto-socket-crosstest-language/templates/](agent-ops/skills/project/add-proto-socket-crosstest-language/templates/). Mark an implementation available only after its same-language tests and cross-language tests pass.
|
|
|
|
Roadmap direction: every implementation should stay as close to native platform behavior as possible outside the protobuf layer. Prefer built-in TCP/TLS/WebSocket, concurrency, timer, and binary APIs over external packages. A focused library that directly matches a protocol need is acceptable when native support is missing or impractical, but broad frameworks or runtime layers must not be introduced just to cover a narrow feature such as WebSocket transport. Runtime dependencies beyond protobuf must be explicitly justified, protocol-relevant, and kept minimal.
|
|
|
|
---
|
|
|
|
## Quick Start (Dart)
|
|
|
|
```dart
|
|
import 'package:proto_socket/proto_socket.dart';
|
|
|
|
// 1. Define your message in message_common.proto, generate with protoc
|
|
|
|
// 2. Implement a client
|
|
class MyClient extends ProtobufClient {
|
|
MyClient(Socket socket) : super(socket, 30, 10, {
|
|
MyMessage.getDefault().info_.qualifiedMessageName: MyMessage.fromBuffer,
|
|
});
|
|
}
|
|
|
|
// 3. Implement a server
|
|
class MyServer extends ProtobufServer {
|
|
MyServer() : super('0.0.0.0', 9090, (socket) => MyClient(socket));
|
|
|
|
@override
|
|
void onClientConnected(ProtobufClient client) {
|
|
client.addListener<MyMessage>((msg) => print('Received: ${msg}'));
|
|
}
|
|
}
|
|
|
|
// 4. Start
|
|
final server = MyServer();
|
|
await server.start();
|
|
|
|
// 5. Connect and send
|
|
final socket = await Socket.connect('localhost', 9090);
|
|
final client = MyClient(socket);
|
|
await client.send(MyMessage()..text = 'hello');
|
|
```
|
|
|
|
---
|
|
|
|
## Adding Message Types
|
|
|
|
Edit the canonical proto at `proto/message_common.proto`, then regenerate all checked-in bindings:
|
|
|
|
```bash
|
|
tools/generate_proto.sh
|
|
tools/check_proto_sync.sh
|
|
```
|
|
|
|
The Go and Kotlin proto copies are allowed to keep only language-specific options such as `option go_package` or Java package/class options. `tools/check_proto_sync.sh` fails with a diff when their message schema drifts from `proto/message_common.proto`.
|
|
|
|
---
|
|
|
|
## Quick Start (Go)
|
|
|
|
```go
|
|
package main
|
|
|
|
import (
|
|
"context"
|
|
"net"
|
|
"time"
|
|
|
|
"google.golang.org/protobuf/proto"
|
|
|
|
protoSocket "git.toki-labs.com/toki/proto-socket/go"
|
|
"git.toki-labs.com/toki/proto-socket/go/packets"
|
|
)
|
|
|
|
func parserMap() protoSocket.ParserMap {
|
|
return protoSocket.ParserMap{
|
|
protoSocket.TypeNameOf(&packets.TestData{}): func(b []byte) (proto.Message, error) {
|
|
m := &packets.TestData{}
|
|
return m, proto.Unmarshal(b, m)
|
|
},
|
|
}
|
|
}
|
|
|
|
func main() {
|
|
ctx := context.Background()
|
|
|
|
server := protoSocket.NewTcpServer("127.0.0.1", 9090, func(conn net.Conn) *protoSocket.TcpClient {
|
|
return protoSocket.NewTcpClient(conn, 30, 10, parserMap())
|
|
})
|
|
server.OnClientConnected = func(client *protoSocket.TcpClient) {
|
|
protoSocket.AddRequestListenerTyped[*packets.TestData, *packets.TestData](
|
|
&client.Communicator,
|
|
func(req *packets.TestData) (*packets.TestData, error) {
|
|
return &packets.TestData{Index: req.GetIndex(), Message: "echo: " + req.GetMessage()}, nil
|
|
},
|
|
)
|
|
}
|
|
if err := server.Start(ctx); err != nil {
|
|
panic(err)
|
|
}
|
|
defer server.Stop()
|
|
|
|
client, err := protoSocket.DialTcp(ctx, "127.0.0.1", 9090, 30, 10, parserMap())
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
defer client.Close()
|
|
|
|
res, err := protoSocket.SendRequestTyped[*packets.TestData, *packets.TestData](
|
|
&client.Communicator,
|
|
&packets.TestData{Index: 1, Message: "hello"},
|
|
2*time.Second,
|
|
)
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
println(res.GetMessage())
|
|
}
|
|
```
|
|
|
|
Go also provides:
|
|
|
|
- TCP: `NewTcpServer`, `DialTcp`, `NewTcpServerTLS`, `DialTcpTLS`
|
|
- WebSocket: `NewWsServer`, `DialWs`, `NewWsServerTLS`, `DialWss`
|
|
- Shared helpers: `Send`, `SendRequest`, `AddListenerTyped`, `AddRequestListenerTyped`, `Broadcast`
|
|
|
|
If you want a copyable scaffold for using this module from another repository, see [examples/go-module-consumer/](examples/go-module-consumer/).
|
|
|
|
---
|
|
|
|
## Running Tests
|
|
|
|
Local commands are documented here for development and troubleshooting. Continuous verification is expected to run in an external tool, with Jenkins planned to execute the full supported-language and cross-language test suite.
|
|
|
|
```bash
|
|
cd dart
|
|
dart pub get
|
|
dart test
|
|
```
|
|
|
|
```bash
|
|
cd go
|
|
go test ./...
|
|
```
|
|
|
|
```bash
|
|
cd kotlin
|
|
./gradlew test
|
|
```
|
|
|
|
```bash
|
|
cd python
|
|
python3 -m pytest -q
|
|
```
|
|
|
|
```bash
|
|
cd typescript
|
|
npm run check
|
|
npm test
|
|
```
|
|
|
|
Cross-language checks:
|
|
|
|
```bash
|
|
cd go
|
|
go run ./crosstest/go_dart.go
|
|
go run ./crosstest/go_kotlin.go
|
|
go run ./crosstest/go_python.go
|
|
go run ./crosstest/go_typescript.go
|
|
```
|
|
|
|
```bash
|
|
cd dart
|
|
dart run crosstest/dart_go.dart
|
|
dart run crosstest/dart_kotlin.dart
|
|
dart run crosstest/dart_python.dart
|
|
dart run crosstest/dart_typescript.dart
|
|
```
|
|
|
|
```bash
|
|
cd kotlin
|
|
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinDartKt
|
|
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinGoKt
|
|
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinPythonKt
|
|
./gradlew run -PmainClass=com.tokilabs.proto_socket.crosstest.KotlinTypescriptKt
|
|
```
|
|
|
|
```bash
|
|
cd python
|
|
python3 crosstest/python_dart.py
|
|
python3 crosstest/python_go.py
|
|
python3 crosstest/python_kotlin.py
|
|
python3 crosstest/python_typescript.py
|
|
```
|
|
|
|
```bash
|
|
cd typescript
|
|
./node_modules/.bin/tsx crosstest/typescript_dart.ts
|
|
./node_modules/.bin/tsx crosstest/typescript_go.ts
|
|
./node_modules/.bin/tsx crosstest/typescript_kotlin.ts
|
|
./node_modules/.bin/tsx crosstest/typescript_python.ts
|
|
```
|
|
|
|
When proto files change, also run:
|
|
|
|
```bash
|
|
tools/check_proto_sync.sh
|
|
```
|