chart data loading
This commit is contained in:
@@ -37,6 +37,14 @@ OHLC periods are represented as seconds.
|
||||
* Relay subscribes to Flink (XSUB) and fanouts to clients (XPUB)
|
||||
* Clients subscribe to specific tickers
|
||||
|
||||
### Symbol Metadata Update Flow (Flink → Gateways)
|
||||
* Ingestors write symbol metadata to Kafka
|
||||
* Flink reads from Kafka, writes to Iceberg symbol_metadata table
|
||||
* After committing to Iceberg, Flink publishes SymbolMetadataUpdated notification on MARKET_DATA_PUB
|
||||
* Gateways subscribe to METADATA_UPDATE topic on startup
|
||||
* Upon receiving notification, gateways reload symbol metadata from Iceberg
|
||||
* This prevents race conditions where gateways start before symbol metadata is available
|
||||
|
||||
### Data Processing (Kafka → Flink → Iceberg)
|
||||
* All market data flows through Kafka (durable event log)
|
||||
* Flink processes streams for aggregations and CEP
|
||||
@@ -90,10 +98,11 @@ All sockets bind on **Relay** (well-known endpoint). Components connect to relay
|
||||
- **Socket Type**:
|
||||
- Relay XPUB (bind) ← Clients SUB (connect) - Port 5558
|
||||
- Relay XSUB (connect) → Flink PUB (bind) - Port 5557
|
||||
- **Message Types**: `Tick`, `OHLC`, `HistoryReadyNotification`
|
||||
- **Message Types**: `Tick`, `OHLC`, `HistoryReadyNotification`, `SymbolMetadataUpdated`
|
||||
- **Topic Formats**:
|
||||
- Market data: `{ticker}|{data_type}` (e.g., `BINANCE:BTC/USDT|tick`)
|
||||
- Notifications: `RESPONSE:{client_id}` or `HISTORY_READY:{request_id}`
|
||||
- System notifications: `METADATA_UPDATE` (for symbol metadata updates)
|
||||
- **Behavior**:
|
||||
- Clients subscribe to ticker topics and notification topics via Relay XPUB
|
||||
- Relay forwards subscriptions to Flink via XSUB
|
||||
@@ -150,6 +159,7 @@ The two-frame envelope is the **logical protocol format**, but physical transmis
|
||||
| 0x10 | SubmitHistoricalRequest | Client request for historical data (async) |
|
||||
| 0x11 | SubmitResponse | Immediate ack with notification topic |
|
||||
| 0x12 | HistoryReadyNotification | Notification that data is ready in Iceberg |
|
||||
| 0x13 | SymbolMetadataUpdated | Notification that symbol metadata refreshed |
|
||||
|
||||
## User Container Event System
|
||||
|
||||
@@ -355,6 +365,11 @@ enum AckStatus {
|
||||
}
|
||||
```
|
||||
|
||||
### Language Notes
|
||||
- JavaScript protobufs will convert field names to camelCase.
|
||||
- Python will retain snake_case.
|
||||
|
||||
|
||||
### Delivery Examples
|
||||
|
||||
```python
|
||||
|
||||
Reference in New Issue
Block a user