ai/doc/protocol.md

ZeroMQ Protocol Architecture

Our data transfer protocol uses ZeroMQ with Protobufs. We send a small envelope with a protocol version byte as the first frame, then a type ID as the first byte of the second frame, followed by the protobuf payload also in the second frame.

OHLC periods are represented as seconds.

Data Flow Overview

Relay as Gateway: The Relay is a well-known bind point that all components connect to. It routes messages between clients, ingestors, and Flink.

Historical Data Query Flow (Async Event-Driven Architecture)

• Client generates request_id and/or client_id (both are client-generated)
• Client computes notification topic: RESPONSE:{client_id} or HISTORY_READY:{request_id}
• Client subscribes to notification topic BEFORE sending request (prevents race condition)
• Client sends SubmitHistoricalRequest to Relay (REQ/REP)
• Relay returns immediate SubmitResponse with request_id and notification_topic (for confirmation)
• Relay publishes DataRequest to ingestor work queue with exchange prefix (PUB/SUB)
• Ingestor receives request, fetches data from exchange
• Ingestor writes OHLC data to Kafka with __metadata in first record
• Flink reads from Kafka, processes data, writes to Iceberg
• Flink publishes HistoryReadyNotification to ZMQ PUB socket (port 5557) with deterministic topic
• Relay proxies notification via XSUB → XPUB to clients
• Client receives notification (already subscribed) and queries Iceberg for data

Key Architectural Change: Relay is completely stateless. No request/response correlation needed. All notification routing is topic-based (e.g., "RESPONSE:{client_id}").

Race Condition Prevention: Notification topics are deterministic based on client-generated values (request_id or client_id). Clients MUST subscribe to the notification topic BEFORE submitting the request to avoid missing notifications.

Two Notification Patterns:

1. Per-client topic (RESPONSE:{client_id}): Subscribe once during connection, reuse for all requests from this client. Recommended for most clients.
2. Per-request topic (HISTORY_READY:{request_id}): Subscribe immediately before each request. Use when you need per-request isolation or don't have a persistent client_id.

Realtime Data Flow (Flink → Relay → Clients)

• Ingestors write realtime ticks to Kafka
• Flink reads from Kafka, processes OHLC aggregations, CEP triggers
• Flink publishes market data via ZMQ PUB
• Relay subscribes to Flink (XSUB) and fanouts to clients (XPUB)
• Clients subscribe to specific tickers

Data Processing (Kafka → Flink → Iceberg)

• All market data flows through Kafka (durable event log)
• Flink processes streams for aggregations and CEP
• Flink writes historical data to Apache Iceberg tables
• Clients can query Iceberg for historical data (alternative to ingestor backfill)

Key Design Principles:

• Relay is the well-known bind point - all other components connect to it
• Relay is completely stateless - no request tracking, only topic-based routing
• Exchange prefix filtering allows ingestor specialization (e.g., only BINANCE ingestors)
• Historical data flows through Kafka (durable processing) only - no direct response
• Async event-driven notifications via pub/sub (Flink → Relay → Clients)
• Protobufs over ZMQ for all inter-service communication
• Kafka for durability and Flink stream processing
• Iceberg for long-term historical storage and client queries

ZeroMQ Channels and Patterns

All sockets bind on Relay (well-known endpoint). Components connect to relay.

1. Client Request Channel (Clients → Relay)

Pattern: ROUTER (Relay binds, Clients use REQ)

• Socket Type: Relay uses ROUTER (bind), Clients use REQ (connect)
• Endpoint: tcp://*:5559 (Relay binds)
• Message Types: SubmitHistoricalRequest → SubmitResponse
• Behavior:
  • Client generates request_id and/or client_id
  • Client computes notification topic deterministically
  • Client subscribes to notification topic FIRST (prevents race)
  • Client sends REQ for historical OHLC data
  • Relay validates request and returns immediate acknowledgment
  • Response includes notification_topic for client confirmation
  • Relay publishes DataRequest to ingestor work queue
  • No request tracking - relay is stateless

2. Ingestor Work Queue (Relay → Ingestors)

Pattern: PUB/SUB with exchange prefix filtering

• Socket Type: Relay uses PUB (bind), Ingestors use SUB (connect)
• Endpoint: tcp://*:5555 (Relay binds)
• Message Types: DataRequest (historical or realtime)
• Topic Prefix: Exchange name (e.g., BINANCE:, COINBASE:)
• Behavior:
  • Relay publishes work with exchange prefix from ticker
  • Ingestors subscribe only to exchanges they support
  • Multiple ingestors can compete for same exchange
  • Ingestors write data to Kafka only (no direct response)
  • Flink processes Kafka → Iceberg → notification

3. Market Data Fanout (Relay ↔ Flink ↔ Clients)

Pattern: XPUB/XSUB proxy

• Socket Type:
  • Relay XPUB (bind) ← Clients SUB (connect) - Port 5558
  • Relay XSUB (connect) → Flink PUB (bind) - Port 5557
• Message Types: Tick, OHLC, HistoryReadyNotification
• Topic Formats:
  • Market data: {ticker}|{data_type} (e.g., BINANCE:BTC/USDT|tick)
  • Notifications: RESPONSE:{client_id} or HISTORY_READY:{request_id}
• Behavior:
  • Clients subscribe to ticker topics and notification topics via Relay XPUB
  • Relay forwards subscriptions to Flink via XSUB
  • Flink publishes processed market data and notifications
  • Relay proxies data to subscribed clients (stateless forwarding)
  • Dynamic subscription management (no pre-registration)

4. Ingestor Control Channel (Optional - Future Use)

Pattern: PUB/SUB (Broadcast control)

• Socket Type: Relay uses PUB, Ingestors use SUB
• Endpoint: tcp://*:5557 (Relay binds)
• Message Types: IngestorControl (cancel, config updates)
• Behavior:
  • Broadcast control messages to all ingestors
  • Used for realtime subscription cancellation
  • Configuration updates

Message Envelope Format

The core protocol uses two ZeroMQ frames:

Frame 1: [1 byte: protocol version]
Frame 2: [1 byte: message type ID][N bytes: protobuf message]

This two-frame approach allows receivers to check the protocol version before parsing the message type and protobuf payload.

Important: Some ZeroMQ socket patterns (PUB/SUB, XPUB/XSUB) may prepend additional frames for routing purposes. For example:

• PUB/SUB with topic filtering: SUB sockets receive [topic frame][version frame][message frame]
• ROUTER sockets: Prepend identity frames before the message

Components must handle these additional frames appropriately:

• SUB sockets: Skip the first frame (topic), then parse the remaining frames as the standard 2-frame envelope
• ROUTER sockets: Extract identity frames, then parse the standard 2-frame envelope

The two-frame envelope is the logical protocol format, but physical transmission may include additional ZeroMQ transport frames.

Message Type IDs

Type ID	Message Type	Description
0x01	DataRequest	Request for historical or realtime data
0x02	DataResponse (deprecated)	Historical data response (no longer used)
0x03	IngestorControl	Control messages for ingestors
0x04	Tick	Individual trade tick data
0x05	OHLC	Single OHLC candle with volume
0x06	Market	Market metadata
0x07	OHLCRequest (deprecated)	Client request (replaced by SubmitHistorical)
0x08	Response (deprecated)	Generic response (replaced by SubmitResponse)
0x09	CEPTriggerRequest	Register CEP trigger
0x0A	CEPTriggerAck	CEP trigger acknowledgment
0x0B	CEPTriggerEvent	CEP trigger fired callback
0x0C	OHLCBatch	Batch of OHLC rows with metadata (Kafka)
0x10	SubmitHistoricalRequest	Client request for historical data (async)
0x11	SubmitResponse	Immediate ack with notification topic
0x12	HistoryReadyNotification	Notification that data is ready in Iceberg

Error Handling

Async Architecture Error Handling:

• Failed historical requests: ingestor writes error marker to Kafka
• Flink reads error marker and publishes HistoryReadyNotification with ERROR status
• Client timeout: if no notification received within timeout, assume failure
• Realtime requests cancelled via control channel if ingestor fails
• REQ/REP timeouts: 30 seconds default for client request submission
• PUB/SUB has no delivery guarantees (Kafka provides durability)
• No response routing needed - all notifications via topic-based pub/sub

Durability:

• All data flows through Kafka for durability
• Flink checkpointing ensures exactly-once processing
• Client can retry request with new request_id if notification not received