chart data loading

This commit is contained in:
2026-03-24 21:37:49 -04:00
parent f6bd22a8ef
commit c76887ab92
65 changed files with 6350 additions and 713 deletions

View File

@@ -37,6 +37,14 @@ OHLC periods are represented as seconds.
* Relay subscribes to Flink (XSUB) and fanouts to clients (XPUB)
* Clients subscribe to specific tickers
### Symbol Metadata Update Flow (Flink → Gateways)
* Ingestors write symbol metadata to Kafka
* Flink reads from Kafka, writes to Iceberg symbol_metadata table
* After committing to Iceberg, Flink publishes SymbolMetadataUpdated notification on MARKET_DATA_PUB
* Gateways subscribe to METADATA_UPDATE topic on startup
* Upon receiving notification, gateways reload symbol metadata from Iceberg
* This prevents race conditions where gateways start before symbol metadata is available
### Data Processing (Kafka → Flink → Iceberg)
* All market data flows through Kafka (durable event log)
* Flink processes streams for aggregations and CEP
@@ -90,10 +98,11 @@ All sockets bind on **Relay** (well-known endpoint). Components connect to relay
- **Socket Type**:
- Relay XPUB (bind) ← Clients SUB (connect) - Port 5558
- Relay XSUB (connect) → Flink PUB (bind) - Port 5557
- **Message Types**: `Tick`, `OHLC`, `HistoryReadyNotification`
- **Message Types**: `Tick`, `OHLC`, `HistoryReadyNotification`, `SymbolMetadataUpdated`
- **Topic Formats**:
- Market data: `{ticker}|{data_type}` (e.g., `BINANCE:BTC/USDT|tick`)
- Notifications: `RESPONSE:{client_id}` or `HISTORY_READY:{request_id}`
- System notifications: `METADATA_UPDATE` (for symbol metadata updates)
- **Behavior**:
- Clients subscribe to ticker topics and notification topics via Relay XPUB
- Relay forwards subscriptions to Flink via XSUB
@@ -150,6 +159,7 @@ The two-frame envelope is the **logical protocol format**, but physical transmis
| 0x10 | SubmitHistoricalRequest | Client request for historical data (async) |
| 0x11 | SubmitResponse | Immediate ack with notification topic |
| 0x12 | HistoryReadyNotification | Notification that data is ready in Iceberg |
| 0x13 | SymbolMetadataUpdated | Notification that symbol metadata refreshed |
## User Container Event System
@@ -355,6 +365,11 @@ enum AckStatus {
}
```
### Language Notes
- JavaScript protobufs will convert field names to camelCase.
- Python will retain snake_case.
### Delivery Examples
```python