dexorder/ai
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add Ticker24h support: hourly market snapshots with USD-normalized volume filtering

Browse Source
This commit is contained in:
Tim Olson
2026-04-26 18:39:52 -04:00
parent 85fcbe1330
commit 0178b5d29d
45 changed files with 1995 additions and 170 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
gateway/knowledge/api-reference.md
View File

@@ -247,6 +247,63 @@ class DataAPI(ABC):
"""
pass
@abstractmethod
async def get_ticker_24h(
self,
exchange: str,
limit: Optional[int] = None,
min_std_quote_volume: Optional[float] = None,
market_type: Optional[str] = None,
base_asset_contains: Optional[str] = None,
) -> pd.DataFrame:
"""
Retrieve 24h rolling market stats for all symbols on an exchange.
Data is refreshed hourly by the ingestor pipeline. Use this to build a
pre-filtered symbol universe before running a scanner — it avoids requesting
per-symbol OHLC data for thousands of symbols.
Args:
exchange: Exchange name (e.g., "BINANCE", "COINBASE", "KRAKEN")
limit: If set, return only the Top N symbols By Volume. None = return all.
min_std_quote_volume: Exclude symbols with USD volume below this threshold.
market_type: Filter by market type: "spot" or "perp". None = return all.
base_asset_contains: Filter to symbols whose base asset contains this string
(case-insensitive). E.g., "BTC" matches "BTC/USDT".
Returns:
DataFrame sorted by std_quote_volume descending (NULLs last). Columns:
- ticker: Full ticker (e.g., "BTC/USDT.BINANCE")
- exchange_id: Exchange name
- base_asset: Base currency (e.g., "BTC")
- quote_asset: Quote currency (e.g., "USDT")
- last_price: Last traded price in quote currency
- price_change_pct: 24h price change as percentage (e.g. 2.5 = +2.5%)
- quote_volume_24h: Raw 24h volume in quote asset
- std_quote_volume: quote_volume_24h normalized to USD (NaN if conversion unknown)
- bid_price, ask_price: Current best bid/ask (NaN if not provided by exchange)
- open_24h, high_24h, low_24h: 24h OHLC prices (NaN if not provided)
- volume_24h: Base-asset volume (NaN if not provided)
- num_trades: 24h trade count (NaN if not provided)
- timestamp_ms: Snapshot timestamp in milliseconds
Returns empty DataFrame if no data is available (e.g., not yet fetched).
Examples:
# Top 50 most liquid Binance spot symbols
df = await api.data.get_ticker_24h("BINANCE", limit=50, market_type="spot")
# All BTC pairs with at least $10M daily volume
df = await api.data.get_ticker_24h("BINANCE",
base_asset_contains="BTC",
min_std_quote_volume=10_000_000)
# Build a scanner universe: all Binance symbols, sorted by volume
universe = await api.data.get_ticker_24h("BINANCE")
top_100 = universe.head(100)["ticker"].tolist()
"""
pass
```

55
gateway/knowledge/usage-examples.md
View File

@@ -92,6 +92,61 @@ All columns below are fully populated for Binance data. Other exchanges provide
- `"ticker"` - Market identifier
- `"period_seconds"` - Period in seconds
## Building a Scanner Universe with get_ticker_24h
**Always pre-filter symbols before fetching OHLC data for scanners.** Fetching OHLC for all ~1800 Binance symbols would exhaust the 2M-bar per-script budget instantly. Use `get_ticker_24h` to get a ranked list of all symbols for free (no OHLC budget cost), then run per-symbol analysis only on the filtered set.
```python
from dexorder.api import get_api
import asyncio
api = get_api()
# Get top 50 most liquid Binance spot symbols (no OHLC budget used)
universe = asyncio.run(api.data.get_ticker_24h(
"BINANCE",
limit=50,
market_type="spot",
min_std_quote_volume=10_000_000 # $10M+ daily volume
))
print(f"Universe: {len(universe)} symbols")
print(universe[["ticker", "std_quote_volume", "price_change_pct"]].head(10))
# Now fetch OHLC only for these symbols
tickers = universe["ticker"].tolist()
results = {}
for ticker in tickers:
df = asyncio.run(api.data.historical_ohlc(
ticker=ticker,
period_seconds=3600,
start_time="2024-01-01",
end_time="2025-01-01",
extra_columns=["volume"]
))
print(f"[Data] {ticker}: {len(df)} bars")
results[ticker] = df
```
### get_ticker_24h filter parameters
```python
# All BTC pairs on Binance (spot + perp)
df = asyncio.run(api.data.get_ticker_24h("BINANCE", base_asset_contains="BTC"))
# Top 100 perp markets with at least $50M daily volume
df = asyncio.run(api.data.get_ticker_24h(
"BINANCE",
limit=100,
market_type="perp",
min_std_quote_volume=50_000_000
))
# All Coinbase symbols (for a cross-exchange scan)
df = asyncio.run(api.data.get_ticker_24h("COINBASE"))
```
The returned DataFrame is sorted by `std_quote_volume` (USD-normalized volume) descending. Symbols without a USD conversion path have `std_quote_volume = NaN` and appear last. Columns: `ticker`, `exchange_id`, `base_asset`, `quote_asset`, `last_price`, `price_change_pct`, `quote_volume_24h`, `std_quote_volume`, `bid_price`, `ask_price`, `open_24h`, `high_24h`, `low_24h`, `volume_24h`, `num_trades`, `timestamp_ms`.
## Using the Charting API
The charting API provides styled financial charts with OHLC candlesticks and technical indicators.

2
gateway/prompt/agent-main.md
View File

@@ -89,7 +89,7 @@ After patching, confirm the change to the user.
## Symbol Resolution
Always use `SymbolLookup` to resolve tickers before passing them to research or chart tools. Symbols must be in `SYMBOL.EXCHANGE` format (e.g., `BTC/USDT.BINANCE`). If the user says "ETHUSDT", "ETH", or any ambiguous ticker, resolve it first. If not specified by the user, prefer to use the most prominent exchange available (e.g. BINANCE not KRAKEN)
Always use `SymbolLookup` to resolve tickers before passing them to research or chart tools. Symbols must be in `SYMBOL.EXCHANGE` format (e.g., `BTC/USDT.BINANCE`). If the user says "ETHUSDT", "ETH", or any ambiguous ticker, resolve it first. `SymbolLookup` results are sorted by 24h volume descending — pick the top result when the user hasn't specified an exchange.
## Raw Data Retrieval

45
gateway/prompt/agent-research.md
View File

@@ -66,6 +66,27 @@ Quick reference — approximate bars per resolution at various windows:
**When to shorten the window**: only if 5 years at the chosen resolution would far exceed 200,000 bars (e.g., 5m over 5 years ≈ 525k → shorten to ~2 years). Otherwise always use the full 5 years.
## Multi-Symbol Analysis
When scanning many symbols, scale the per-symbol time window so total bars stay within the **2,000,000-bar script limit**. The API enforces this — exceeding it raises a `ValueError` with the limit number and suggestions.
Budget rule: `bars_per_symbol ≈ 2,000,000 / num_symbols` (never exceed 200,000 per symbol)
| Symbol count | Recommended period | Approx max window |
|---|---|---|
| ≤ 10 | any | 5 years |
| 10–100 | 1h or coarser | scale to budget |
| 100–500 | 1d (86400s) | ~1–2 years |
| 500+ | 1d (86400s) | ≤ 1 year |
**Strategy for large symbol lists**:
1. **Filter first**: scan all symbols with a short window (90–180 days, daily bars) to rank/screen candidates
2. **Zoom in**: fetch full history only for the top N (≤ 20) finalists
3. **Never use intraday periods for > 50 symbols** in one script
4. **Print progress** every 50 symbols so the output log shows the script is alive
If you hit a `ValueError` about the bar budget, read the limit and suggestions in the error message, then adjust the period or window accordingly.
## Tool Behavior Notes
- **`PythonWrite` / `PythonEdit` for research**: auto-executes the script and returns all output (stdout, stderr) and captured images. **Do not call `ExecuteResearch` afterward** — the script has already run.
@@ -89,6 +110,30 @@ The API provides two main components:
See the knowledge base sections below for complete API documentation, examples, and the full pandas-ta indicator reference.
### Scanner Pre-filtering with get_ticker_24h
**Before fetching OHLC data for multiple symbols, always build a pre-filtered universe first.**
Scanners must not blindly fetch OHLC for all symbols on an exchange — Binance has ~1800 symbols and the script budget is 2M bars total. Use `api.data.get_ticker_24h()` to get a ranked, filterable list of all symbols without consuming any OHLC budget:
```python
# Get top 50 most liquid Binance spot symbols by USD volume
universe = asyncio.run(api.data.get_ticker_24h(
"BINANCE",
limit=50,
market_type="spot",
min_std_quote_volume=10_000_000 # $10M+ daily volume
))
tickers = universe["ticker"].tolist()
print(f"Universe: {len(tickers)} symbols")
# Now fetch OHLC only for these symbols
for ticker in tickers:
df = asyncio.run(api.data.historical_ohlc(ticker, period_seconds=3600, ...))
```
`get_ticker_24h` returns a DataFrame sorted by `std_quote_volume` (USD-normalized) descending, with columns: `ticker`, `exchange_id`, `base_asset`, `quote_asset`, `last_price`, `price_change_pct`, `quote_volume_24h`, `std_quote_volume`, `bid_price`, `ask_price`, `open_24h`, `high_24h`, `low_24h`, `volume_24h`, `num_trades`, `timestamp_ms`. See the full docstring in the knowledge base `api-reference.md`.
## Technical Indicators — pandas-ta
Use `import pandas_ta as ta` for all indicator calculations. Never write manual rolling/ewm implementations. The full indicator catalog, calling conventions, column naming patterns, and default parameters are in the pandas-ta-reference section of your knowledge base.

1
gateway/prompt/tools.md
View File

@@ -22,6 +22,7 @@ Available to all agents:
| `PythonList` | List existing scripts by category (`strategy`, `indicator`, or `research`) |
| `SymbolLookup` | Resolve a ticker to the correct `SYMBOL.EXCHANGE` format |
| `GetChartData` | Fetch raw OHLC data (casual retrieval only — use `Spawn` research for analysis) |
| `GetTicker24h` | Fetch 24h market stats for all symbols on an exchange, sorted by USD volume — use this to build scanner universes without burning OHLC bar budget |
| `WebSearch` | Search the web (Tavily) |
| `FetchPage` | Fetch and read a web page or PDF |
| `ArxivSearch` | Search arXiv for academic papers |

55
gateway/src/channels/websocket-handler.ts
View File

@@ -57,6 +57,7 @@ interface BarSubscription {
ticker: string;
periodSeconds: number;
callback: BarUpdateCallback;
openBars: boolean;
}
export class WebSocketHandler {
@@ -65,6 +66,8 @@ export class WebSocketHandler {
private workspaces = new Map<string, WorkspaceManager>();
/** Per-session realtime bar subscriptions for cleanup on disconnect */
private barSubscriptions = new Map<string, BarSubscription[]>();
/** "sessionId:pandas_ta_name" → active request_id; supersedes stale requests on scroll */
private activeEvaluations = new Map<string, string>();
constructor(config: WebSocketHandlerConfig) {
this.config = config;
@@ -501,13 +504,19 @@ export class WebSocketHandler {
const sessionId = authContext.sessionId;
const subs = this.barSubscriptions.get(sessionId);
if (subs && this.config.ohlcService) {
for (const { ticker, periodSeconds, callback } of subs) {
this.config.ohlcService.unsubscribeFromTicker(ticker, periodSeconds, callback);
for (const { ticker, periodSeconds, callback, openBars } of subs) {
this.config.ohlcService.unsubscribeFromTicker(ticker, periodSeconds, callback, openBars);
}
this.barSubscriptions.delete(sessionId);
logger.info({ sessionId, count: subs.length }, 'Cleaned up realtime bar subscriptions');
}
// Cleanup active indicator evaluations for this session
const evalPrefix = `${sessionId}:`;
for (const key of this.activeEvaluations.keys()) {
if (key.startsWith(evalPrefix)) this.activeEvaluations.delete(key);
}
// Cleanup workspace
await workspace!.shutdown();
this.workspaces.delete(authContext.sessionId);
@@ -623,18 +632,11 @@ export class WebSocketHandler {
case 'search_symbols': {
logger.info({ query: payload.query, limit: payload.limit }, 'Handling search_symbols');
// Use SymbolIndexService if available, otherwise fallback to OHLCService stub
const symbolIndexService = this.config.symbolIndexService;
logger.info({ hasSymbolIndexService: !!symbolIndexService }, 'Service check for search');
const results = symbolIndexService
? await symbolIndexService.search(payload.query, payload.limit || 30)
: (ohlcService ? await ohlcService.searchSymbols(
payload.query,
payload.symbol_type,
payload.exchange,
payload.limit || 30
) : []);
: [];
logger.info({ resultsCount: results.length }, 'Search complete');
socket.send(
@@ -649,13 +651,11 @@ export class WebSocketHandler {
case 'resolve_symbol': {
logger.info({ symbol: payload.symbol }, 'Handling resolve_symbol');
// Use SymbolIndexService if available, otherwise fallback to OHLCService stub
const symbolIndexService = this.config.symbolIndexService;
logger.info({ hasSymbolIndexService: !!symbolIndexService }, 'Service check for resolve');
const symbolInfo = symbolIndexService
? await symbolIndexService.resolveSymbol(payload.symbol)
: (ohlcService ? await ohlcService.resolveSymbol(payload.symbol) : null);
: null;
logger.info({ found: !!symbolInfo }, 'Symbol resolution complete');
@@ -723,6 +723,8 @@ export class WebSocketHandler {
const subTicker: string = payload.symbol;
const subPeriod: number = payload.period_seconds ?? payload.resolution ?? 60;
// 'open' = in-progress bar snapshots every tick (chart); 'closed' = completed bars only (strategies)
const openBars: boolean = (payload.bar_type ?? 'open') === 'open';
const sessionId = authContext.sessionId;
// Create a per-subscription callback that forwards bars to this socket
@@ -733,6 +735,7 @@ export class WebSocketHandler {
subscription_id: payload.subscription_id,
ticker: bar.ticker,
period_seconds: bar.periodSeconds,
is_closed: bar.isClosed,
bar: {
// Convert nanoseconds → seconds for client compatibility
time: Number(bar.timestamp / 1_000_000_000n),
@@ -745,7 +748,7 @@ export class WebSocketHandler {
}));
};
ohlcService.subscribeToTicker(subTicker, subPeriod, barCallback);
ohlcService.subscribeToTicker(subTicker, subPeriod, barCallback, openBars);
// Track for cleanup on disconnect
if (!this.barSubscriptions.has(sessionId)) {
@@ -755,6 +758,7 @@ export class WebSocketHandler {
ticker: subTicker,
periodSeconds: subPeriod,
callback: barCallback,
openBars,
});
logger.info({ sessionId, ticker: subTicker, period: subPeriod }, 'Subscribed to realtime bars');
@@ -782,7 +786,7 @@ export class WebSocketHandler {
);
if (idx >= 0) {
const [removed] = subs.splice(idx, 1);
ohlcService.unsubscribeFromTicker(unsubTicker, unsubPeriod, removed.callback);
ohlcService.unsubscribeFromTicker(unsubTicker, unsubPeriod, removed.callback, removed.openBars);
logger.info({ sessionId, ticker: unsubTicker, period: unsubPeriod }, 'Unsubscribed from realtime bars');
}
}
@@ -807,6 +811,19 @@ export class WebSocketHandler {
}));
break;
}
// Supersede any in-flight request for the same indicator (e.g. rapid scrolling)
const evalKey = `${authContext.sessionId}:${payload.pandas_ta_name}`;
const prevRequestId = this.activeEvaluations.get(evalKey);
if (prevRequestId) {
socket.send(JSON.stringify({
type: 'evaluate_indicator_result',
request_id: prevRequestId,
error: 'superseded',
}));
}
this.activeEvaluations.set(evalKey, requestId);
try {
const mcpResult = await harness.callMcpTool('EvaluateIndicator', {
symbol: payload.symbol,
@@ -816,6 +833,11 @@ export class WebSocketHandler {
pandas_ta_name: payload.pandas_ta_name,
parameters: payload.parameters ?? {},
}) as any;
// Discard result if a newer request arrived while we were awaiting
if (this.activeEvaluations.get(evalKey) !== requestId) break;
this.activeEvaluations.delete(evalKey);
// MCP returns { content: [{type: 'text', text: '...json...'}] }
// When the tool raises an exception, the MCP framework sets isError: true
// and puts the raw exception text in content[0].text (not JSON-wrapped).
@@ -849,6 +871,9 @@ export class WebSocketHandler {
...data,
}));
} catch (err: any) {
if (this.activeEvaluations.get(evalKey) === requestId) {
this.activeEvaluations.delete(evalKey);
}
logger.error({ err: err?.message, pandas_ta_name: payload.pandas_ta_name }, 'evaluate_indicator handler error');
socket.send(JSON.stringify({
type: 'evaluate_indicator_result',

98
gateway/src/clients/zmq-protocol.ts
View File

@@ -20,11 +20,15 @@ import type {
NotificationStatus,
} from '../types/ohlc.js';
export const OHLC_BAR_TOPIC_PATTERN = /^(.+)\|ohlc:(\d+)$/;
// Matches both "{ticker}|ohlc:{period}" (closed) and "{ticker}|ohlc:{period}:open" (open bar)
export const OHLC_BAR_TOPIC_PATTERN = /^(.+)\|ohlc:(\d+)(:open)?$/;
// Matches "{exchange_id}|ticker24h" (e.g., "BINANCE|ticker24h")
export const TICKER24H_TOPIC_PATTERN = /^([A-Z0-9]+)\|ticker24h$/;
/** Decoded realtime OHLC bar received from the XPUB market data stream */
export interface RealtimeBar {
topic: string; // e.g., "BTC/USDT.BINANCE|ohlc:60"
topic: string; // e.g., "BTC/USDT.BINANCE|ohlc:60" or "BTC/USDT.BINANCE|ohlc:60:open"
ticker: string; // e.g., "BTC/USDT.BINANCE"
periodSeconds: number;
/** Window open time in nanoseconds since epoch */
@@ -34,6 +38,35 @@ export interface RealtimeBar {
low: number;
close: number;
volume: number;
/** True if this bar's time window has fully closed (strategies); false if still accumulating (chart). */
isClosed: boolean;
}
/** Single ticker 24h stats from a Ticker24h snapshot */
export interface Ticker24hStats {
ticker: string;
exchange_id: string;
base_asset: string;
quote_asset: string;
last_price: number;
price_change_pct: number;
quote_volume_24h: number;
timestamp_ms: number; // milliseconds (converted from nanoseconds)
bid_price?: number;
ask_price?: number;
open_24h?: number;
high_24h?: number;
low_24h?: number;
volume_24h?: number;
std_quote_volume?: number; // null if conversion unknown
num_trades?: number;
}
/** Decoded Ticker24h snapshot received from the XPUB market data stream */
export interface Ticker24hSnapshot {
exchange_id: string;
tickers: Ticker24hStats[];
generated_at_ms: number; // milliseconds (converted from nanoseconds)
}
const __filename = fileURLToPath(import.meta.url);
@@ -58,14 +91,17 @@ const root = new protobuf.Root();
// Load proto files
const ingestorProto = readFileSync(join(protoDir, 'ingestor.proto'), 'utf8');
const ohlcProto = readFileSync(join(protoDir, 'ohlc.proto'), 'utf8');
const ticker24hProto = readFileSync(join(protoDir, 'ticker24h.proto'), 'utf8');
protobuf.parse(ingestorProto, root);
protobuf.parse(ohlcProto, root);
protobuf.parse(ticker24hProto, root);
// Export message types
const SubmitHistoricalRequestType = root.lookupType('SubmitHistoricalRequest');
const SubmitResponseType = root.lookupType('SubmitResponse');
const HistoryReadyNotificationType = root.lookupType('HistoryReadyNotification');
const OHLCType = root.lookupType('OHLC');
const Ticker24hType = root.lookupType('Ticker24h');
/**
* Encode SubmitHistoricalRequest to ZMQ frames
@@ -198,6 +234,62 @@ export function decodeHistoryReadyNotification(frames: Buffer[]): HistoryReadyNo
};
}
/**
* Decode a Ticker24h snapshot from ZMQ SUB frames.
* Frame layout: [topic][version][0x0D Ticker24h type + Ticker24h protobuf bytes]
*
* Returns null if the topic doesn't match the ticker24h pattern or decoding fails.
*/
export function decodeTicker24h(frames: Buffer[]): Ticker24hSnapshot | null {
if (frames.length < 3) return null;
const topic = frames[0].toString();
const match = TICKER24H_TOPIC_PATTERN.exec(topic);
if (!match) return null;
const messageFrame = frames[2];
if (messageFrame[0] !== 0x0D) return null; // Must be TICKER_24H type
try {
const payloadBuffer = messageFrame.slice(1);
const decoded = Ticker24hType.decode(payloadBuffer);
const snapshot = Ticker24hType.toObject(decoded, {
longs: String,
defaults: false,
});
const tickers: Ticker24hStats[] = (snapshot.tickers ?? []).map((ts: any) => {
const row: Ticker24hStats = {
ticker: ts.ticker ?? '',
exchange_id: ts.exchangeId ?? '',
base_asset: ts.baseAsset ?? '',
quote_asset: ts.quoteAsset ?? '',
last_price: Number(ts.lastPrice ?? 0),
price_change_pct: Number(ts.priceChangePct ?? 0),
quote_volume_24h: Number(ts.quoteVolume24H ?? 0),
timestamp_ms: Math.round(Number(BigInt(ts.timestamp ?? '0')) / 1e6),
};
if (ts.bidPrice != null) row.bid_price = Number(ts.bidPrice);
if (ts.askPrice != null) row.ask_price = Number(ts.askPrice);
if (ts.open24H != null) row.open_24h = Number(ts.open24H);
if (ts.high24H != null) row.high_24h = Number(ts.high24H);
if (ts.low24H != null) row.low_24h = Number(ts.low24H);
if (ts.volume24H != null) row.volume_24h = Number(ts.volume24H);
if (ts.stdQuoteVolume != null) row.std_quote_volume = Number(ts.stdQuoteVolume);
if (ts.numTrades != null) row.num_trades = Number(ts.numTrades);
return row;
});
return {
exchange_id: snapshot.exchangeId ?? match[1],
tickers,
generated_at_ms: Math.round(Number(BigInt(snapshot.generatedAt ?? '0')) / 1e6),
};
} catch (e) {
return null;
}
}
/**
* Decode a realtime OHLC bar from ZMQ SUB frames.
* Frame layout: [topic][version][0x04 OHLC type + OHLC protobuf bytes]
@@ -213,6 +305,7 @@ export function decodeRealtimeBar(frames: Buffer[]): RealtimeBar | null {
const ticker = match[1];
const periodSeconds = parseInt(match[2], 10);
const isClosed = !match[3]; // ":open" suffix absent → closed bar
const messageFrame = frames[2];
if (messageFrame[0] !== 0x04) return null; // Must be OHLC type
@@ -231,5 +324,6 @@ export function decodeRealtimeBar(frames: Buffer[]): RealtimeBar | null {
low: Number(ohlc.low ?? 0),
close: Number(ohlc.close ?? 0),
volume: Number(ohlc.volume ?? 0),
isClosed,
};
}

49
gateway/src/clients/zmq-relay-client.ts
View File

@@ -18,8 +18,11 @@ import {
decodeSubmitResponse,
decodeHistoryReadyNotification,
decodeRealtimeBar,
decodeTicker24h,
OHLC_BAR_TOPIC_PATTERN,
TICKER24H_TOPIC_PATTERN,
type RealtimeBar,
type Ticker24hSnapshot,
} from './zmq-protocol.js';
import type {
SubmitHistoricalRequest,
@@ -31,7 +34,8 @@ import {
} from '../types/ohlc.js';
export type BarUpdateCallback = (bar: RealtimeBar) => void;
export type { RealtimeBar };
export type Ticker24hCallback = (snapshot: Ticker24hSnapshot) => void;
export type { RealtimeBar, Ticker24hSnapshot };
export interface ZMQRelayConfig {
relayRequestEndpoint: string; // e.g., "tcp://relay:5559"
@@ -39,6 +43,7 @@ export interface ZMQRelayConfig {
clientId?: string; // Optional client ID, will generate if not provided
requestTimeout?: number; // Request timeout in ms (default: 120000)
onMetadataUpdate?: () => Promise<void>; // Callback when symbol metadata updates
onTicker24h?: (snapshot: Ticker24hSnapshot) => void; // Callback when Ticker24h snapshot arrives
}
interface PendingRequest {
@@ -79,6 +84,7 @@ export class ZMQRelayClient {
clientId: config.clientId || `gateway-${randomUUID().slice(0, 8)}`,
requestTimeout: config.requestTimeout || 120000,
onMetadataUpdate: config.onMetadataUpdate || (async () => {}),
onTicker24h: config.onTicker24h || (() => {}),
};
this.logger = logger;
this.notificationTopic = `RESPONSE:${this.config.clientId}`;
@@ -275,6 +281,17 @@ export class ZMQRelayClient {
continue;
}
// Handle Ticker24h snapshot updates (topic pattern: "{exchange}|ticker24h")
if (TICKER24H_TOPIC_PATTERN.test(topic)) {
if (this.config.onTicker24h) {
const snapshot = decodeTicker24h(Array.from(frames));
if (snapshot) {
try { this.config.onTicker24h(snapshot); } catch (e) { /* ignore callback errors */ }
}
}
continue;
}
// Handle realtime OHLC bar updates (topic pattern: "{ticker}|ohlc:{period}")
if (OHLC_BAR_TOPIC_PATTERN.test(topic)) {
const bar = decodeRealtimeBar(Array.from(frames));
@@ -339,9 +356,12 @@ export class ZMQRelayClient {
* This triggers the relay XPUB → Flink subscription detection → ingestor activation.
*
* @param callback Called whenever a new bar arrives for this topic
* @param openBars If true, subscribe to the ":open" topic (live in-progress bar updates
* for charts). If false (default), subscribe to closed bars only
* (completed candles for strategies/triggers).
*/
subscribeToTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback): void {
const topic = `${ticker}|ohlc:${periodSeconds}`;
subscribeToTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback, openBars = false): void {
const topic = `${ticker}|ohlc:${periodSeconds}${openBars ? ':open' : ''}`;
// Register callback
if (!this.barCallbacks.has(topic)) {
@@ -361,9 +381,10 @@ export class ZMQRelayClient {
/**
* Unsubscribe a callback from realtime OHLC bars.
* ZMQ unsubscribe is only called on the 1→0 transition (last subscriber).
* @param openBars Must match the value used in the corresponding subscribeToTicker call.
*/
unsubscribeFromTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback): void {
const topic = `${ticker}|ohlc:${periodSeconds}`;
unsubscribeFromTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback, openBars = false): void {
const topic = `${ticker}|ohlc:${periodSeconds}${openBars ? ':open' : ''}`;
const callbacks = this.barCallbacks.get(topic);
if (callbacks) {
@@ -385,13 +406,25 @@ export class ZMQRelayClient {
}
}
/**
* Subscribe to Ticker24h snapshots for an exchange.
* Only calls ZMQ subscribe — receipt is handled via the onTicker24h config callback.
*/
subscribeToTicker24h(exchange: string): void {
const topic = `${exchange.toUpperCase()}|ticker24h`;
if (this.subSocket) {
this.subSocket.subscribe(topic);
this.logger.info({ topic }, 'ZMQ subscribed to Ticker24h topic');
}
}
/**
* Remove all subscriptions for a set of (topic, callback) pairs.
* Convenience method for WebSocket disconnect cleanup.
*/
cleanupSubscriptions(subscriptions: Array<{ ticker: string; periodSeconds: number; callback: BarUpdateCallback }>): void {
for (const { ticker, periodSeconds, callback } of subscriptions) {
this.unsubscribeFromTicker(ticker, periodSeconds, callback);
cleanupSubscriptions(subscriptions: Array<{ ticker: string; periodSeconds: number; callback: BarUpdateCallback; openBars?: boolean }>): void {
for (const { ticker, periodSeconds, callback, openBars } of subscriptions) {
this.unsubscribeFromTicker(ticker, periodSeconds, callback, openBars ?? false);
}
}

2
gateway/src/harness/spawn/spawn-service.ts
View File

@@ -2,7 +2,7 @@ import type { BaseChatModel } from '@langchain/core/language_models/chat_models'
import { SystemMessage, HumanMessage } from '@langchain/core/messages';
/** All platform tool names available to every subagent. */
const ALL_PLATFORM_TOOLS = ['SymbolLookup', 'GetChartData', 'WebSearch', 'FetchPage', 'ArxivSearch'];
const ALL_PLATFORM_TOOLS = ['SymbolLookup', 'GetChartData', 'GetTicker24h', 'WebSearch', 'FetchPage', 'ArxivSearch'];
import type { FastifyBaseLogger } from 'fastify';
import { createReactAgent } from '@langchain/langgraph/prebuilt';
import type { HarnessEvent, SubagentChunkEvent, SubagentThinkingEvent } from '../harness-events.js';

1
gateway/src/harness/tool-labels.ts
View File

@@ -9,6 +9,7 @@ const TOOL_LABELS: Record<string, string> = {
MemoryLookup: 'Checking docs...',
memory_lookup: 'Checking docs...',
GetChartData: 'Fetching chart data...',
GetTicker24h: 'Fetching market data...',
SymbolLookup: 'Searching symbol...',
WebSearch: 'Searching the web...',
FetchPage: 'Fetching page...',

49
gateway/src/main.ts
View File

@@ -13,7 +13,7 @@ import { WebSocketHandler } from './channels/websocket-handler.js';
import { TelegramHandler } from './channels/telegram-handler.js';
import { KubernetesClient } from './k8s/client.js';
import { ContainerManager } from './k8s/container-manager.js';
import { ZMQRelayClient } from './clients/zmq-relay-client.js';
import { ZMQRelayClient, type Ticker24hSnapshot } from './clients/zmq-relay-client.js';
import { IcebergClient } from './clients/iceberg-client.js';
import { ConversationStore } from './harness/memory/conversation-store.js';
import { BlobStore } from './harness/memory/blob-store.js';
@@ -141,10 +141,13 @@ function loadConfig() {
conversationsBucket: configData.iceberg?.conversations_bucket || process.env.CONVERSATIONS_S3_BUCKET,
},
// Relay configuration (for historical data)
// Relay configuration (for historical data and market snapshots)
relay: {
requestEndpoint: configData.relay?.request_endpoint || process.env.RELAY_REQUEST_ENDPOINT || 'tcp://relay:5559',
notificationEndpoint: configData.relay?.notification_endpoint || process.env.RELAY_NOTIFICATION_ENDPOINT || 'tcp://relay:5558',
// Exchanges to subscribe for Ticker24h snapshots
supportedExchanges: (configData.relay?.supported_exchanges || process.env.RELAY_SUPPORTED_EXCHANGES || 'BINANCE')
.split(',').map((s: string) => s.trim().toUpperCase()).filter(Boolean) as string[],
},
// Kubernetes configuration
@@ -258,12 +261,28 @@ const onMetadataUpdate = async () => {
}
};
// Initialize ZMQ Relay client (for historical data)
// Pass onMetadataUpdate callback so it's registered before connection
// Ticker24h in-memory cache (primary serving layer; Redis is warm-start only)
const ticker24hCache = new Map<string, Ticker24hSnapshot>();
const TICKER24H_REDIS_TTL = 5400; // 90 minutes
function onTicker24hReceived(snapshot: Ticker24hSnapshot): void {
const exchange = snapshot.exchange_id;
ticker24hCache.set(exchange, snapshot);
app.log.info({ exchange, count: snapshot.tickers.length }, 'Ticker24h snapshot updated in memory');
// Write to Redis as JSON (fire-and-forget warm-start cache)
redis.setex(`ticker24h:${exchange}`, TICKER24H_REDIS_TTL, JSON.stringify(snapshot))
.catch(err => app.log.warn({ exchange, err }, 'Failed to write Ticker24h to Redis'));
}
// Initialize ZMQ Relay client (for historical data and market snapshots)
// Pass onMetadataUpdate and onTicker24h callbacks so they're registered before connection
const zmqRelayClient = new ZMQRelayClient({
relayRequestEndpoint: config.relay.requestEndpoint,
relayNotificationEndpoint: config.relay.notificationEndpoint,
onMetadataUpdate,
onTicker24h: onTicker24hReceived,
}, app.log);
app.log.info({
@@ -478,6 +497,24 @@ try {
try {
await zmqRelayClient.connect();
app.log.info('ZMQ Relay connected');
// Subscribe to Ticker24h topics for all configured exchanges
for (const exchange of config.relay.supportedExchanges) {
zmqRelayClient.subscribeToTicker24h(exchange);
}
// Warm Ticker24h in-memory cache from Redis (best-effort; gateway ZMQ is the primary source)
for (const exchange of config.relay.supportedExchanges) {
try {
const cached = await redis.get(`ticker24h:${exchange}`);
if (cached) {
ticker24hCache.set(exchange, JSON.parse(cached));
app.log.info({ exchange }, 'Ticker24h warm-started from Redis');
}
} catch (err) {
app.log.debug({ exchange, err }, 'No Ticker24h cache in Redis (will populate on first ZMQ update)');
}
}
} catch (error) {
app.log.warn({ error }, 'ZMQ Relay connection failed - historical data will not be available');
}
@@ -490,6 +527,7 @@ try {
ohlcService: () => ohlcService,
symbolIndexService: () => symbolIndexService,
workspaceManager: undefined, // Will be set per-session
ticker24hGetter: (exchange: string) => ticker24hCache.get(exchange),
tavilyApiKey: config.tavilyApiKey,
});
@@ -497,7 +535,7 @@ try {
// Main agent: platform tools + user's general MCP tools
toolRegistry.registerAgentTools({
agentName: 'main',
platformTools: ['SymbolLookup', 'GetChartData'],
platformTools: ['SymbolLookup', 'GetChartData', 'GetTicker24h'],
mcpTools: ['PythonList', 'PythonDelete', 'BacktestStrategy', 'ListActiveStrategies'],
});
@@ -548,6 +586,7 @@ try {
const indexService = new SymbolIndexService({
icebergClient,
logger: app.log,
getTicker24h: (exchange: string) => ticker24hCache.get(exchange),
});
// Assign to module-level variable so onMetadataUpdate callback can use it

76
gateway/src/services/ohlc-service.ts
View File

@@ -20,8 +20,6 @@ import type { ZMQRelayClient, BarUpdateCallback } from '../clients/zmq-relay-cli
export type { BarUpdateCallback } from '../clients/zmq-relay-client.js';
import type {
HistoryResult,
SymbolInfo,
SearchResult,
DatafeedConfig,
TradingViewBar,
} from '../types/ohlc.js';
@@ -58,17 +56,21 @@ export class OHLCService {
* Subscribe to realtime OHLC bar updates for a ticker+period.
* ZMQ subscribe is issued on the first call for a given topic; subsequent calls
* for the same topic only add the callback (no extra ZMQ events).
*
* @param openBars If true, subscribe to in-progress bar snapshots (every tick, for charts).
* If false (default), subscribe to closed bars only (for strategies/triggers).
*/
subscribeToTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback): void {
this.relayClient.subscribeToTicker(ticker, periodSeconds, callback);
subscribeToTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback, openBars = false): void {
this.relayClient.subscribeToTicker(ticker, periodSeconds, callback, openBars);
}
/**
* Unsubscribe a callback from realtime OHLC bar updates.
* ZMQ unsubscribe is issued when the last callback for a topic is removed.
* @param openBars Must match the value used in the corresponding subscribeToTicker call.
*/
unsubscribeFromTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback): void {
this.relayClient.unsubscribeFromTicker(ticker, periodSeconds, callback);
unsubscribeFromTicker(ticker: string, periodSeconds: number, callback: BarUpdateCallback, openBars = false): void {
this.relayClient.unsubscribeFromTicker(ticker, periodSeconds, callback, openBars);
}
/**
@@ -223,66 +225,4 @@ export class OHLCService {
supports_time: false,
};
}
/**
* Search symbols
*
* For now, stub with default symbol
*/
async searchSymbols(
query: string,
type?: string,
exchange?: string,
limit: number = 30
): Promise<SearchResult[]> {
this.logger.debug({ query, type, exchange, limit }, 'Searching symbols');
// TODO: Implement central symbol registry
// For now, return default symbol if query matches
if (query.toLowerCase().includes('btc') || query.toLowerCase().includes('binance')) {
return [{
symbol: 'BTC/USDT',
full_name: 'BTC/USDT (BINANCE)',
description: 'Bitcoin / Tether USD',
exchange: 'BINANCE',
ticker: 'BTC/USDT.BINANCE',
type: 'crypto',
}];
}
return [];
}
/**
* Resolve symbol metadata
*
* For now, stub with default symbol
*/
async resolveSymbol(symbol: string): Promise<SymbolInfo> {
this.logger.debug({ symbol }, 'Resolving symbol');
// TODO: Implement central symbol registry
// For now, return default symbol info for BTC/USDT.BINANCE
if (symbol === 'BTC/USDT.BINANCE' || symbol === 'BTC/USDT') {
return {
symbol: 'BTC/USDT',
name: 'BTC/USDT',
ticker: 'BTC/USDT.BINANCE',
description: 'Bitcoin / Tether USD',
type: 'crypto',
session: '24x7',
timezone: 'Etc/UTC',
exchange: 'BINANCE',
minmov: 1,
pricescale: 100,
has_intraday: true,
has_daily: true,
has_weekly_and_monthly: true,
supported_resolutions: DEFAULT_SUPPORTED_RESOLUTIONS,
data_status: 'streaming',
};
}
throw new Error(`Symbol not found: ${symbol}`);
}
}

36
gateway/src/services/symbol-index-service.ts
View File

@@ -7,12 +7,14 @@
import type { FastifyBaseLogger } from 'fastify';
import type { IcebergClient } from '../clients/iceberg-client.js';
import type { Ticker24hSnapshot } from '../clients/zmq-relay-client.js';
import type { SearchResult, SymbolInfo, SymbolMetadata } from '../types/ohlc.js';
import { DEFAULT_SUPPORTED_RESOLUTIONS } from '../types/ohlc.js';
export interface SymbolIndexServiceConfig {
icebergClient: IcebergClient;
logger: FastifyBaseLogger;
getTicker24h?: (exchange: string) => Ticker24hSnapshot | undefined;
}
/**
@@ -23,6 +25,7 @@ export interface SymbolIndexServiceConfig {
export class SymbolIndexService {
private icebergClient: IcebergClient;
private logger: FastifyBaseLogger;
private getTicker24h?: (exchange: string) => Ticker24hSnapshot | undefined;
private symbols: Map<string, SymbolMetadata> = new Map(); // key: "MARKET_ID.EXCHANGE" (Nautilus format)
private initialized: boolean = false;
private initPromise: Promise<void> | null = null;
@@ -30,6 +33,7 @@ export class SymbolIndexService {
constructor(config: SymbolIndexServiceConfig) {
this.icebergClient = config.icebergClient;
this.logger = config.logger;
this.getTicker24h = config.getTicker24h;
}
/**
@@ -114,10 +118,9 @@ export class SymbolIndexService {
}
const queryLower = query.toLowerCase();
const results: SearchResult[] = [];
const matched: SymbolMetadata[] = [];
for (const [key, metadata] of this.symbols) {
// Match against various fields
const ticker = key;
const base = metadata.base_asset || '';
const quote = metadata.quote_asset || '';
@@ -131,15 +134,36 @@ export class SymbolIndexService {
desc.toLowerCase().includes(queryLower) ||
marketId.toLowerCase().includes(queryLower)
) {
results.push(this.metadataToSearchResult(metadata));
matched.push(metadata);
}
}
if (results.length >= limit) {
break;
// Build a volume lookup from Ticker24h cache keyed by "ticker" field (e.g. "BTC/USDT")
// within each exchange snapshot. Map: "MARKET_ID.EXCHANGE" → std_quote_volume
const volumeMap = new Map<string, number>();
if (this.getTicker24h) {
const exchanges = new Set(matched.map(m => m.exchange_id));
for (const exchange of exchanges) {
const snapshot = this.getTicker24h(exchange);
if (snapshot) {
for (const ts of snapshot.tickers) {
const key = `${ts.ticker}.${exchange}`;
const vol = ts.std_quote_volume ?? ts.quote_volume_24h ?? 0;
volumeMap.set(key, vol);
}
}
}
}
this.logger.debug({ query, count: results.length }, 'Symbol search completed');
matched.sort((a, b) => {
const ka = `${a.market_id}.${a.exchange_id}`;
const kb = `${b.market_id}.${b.exchange_id}`;
return (volumeMap.get(kb) ?? 0) - (volumeMap.get(ka) ?? 0);
});
const results = matched.slice(0, limit).map(m => this.metadataToSearchResult(m));
this.logger.debug({ query, matched: matched.length, returned: results.length }, 'Symbol search completed');
return results;
}

90
gateway/src/tools/platform/get-ticker24h.tool.ts Normal file
View File

@@ -0,0 +1,90 @@
import { DynamicStructuredTool } from '@langchain/core/tools';
import { z } from 'zod';
import type { FastifyBaseLogger } from 'fastify';
import type { Ticker24hSnapshot } from '../../clients/zmq-relay-client.js';
export interface GetTicker24hToolConfig {
getTicker24h: (exchange: string) => Ticker24hSnapshot | undefined;
logger: FastifyBaseLogger;
}
export function createGetTicker24hTool(config: GetTicker24hToolConfig): DynamicStructuredTool {
const { getTicker24h, logger } = config;
return new DynamicStructuredTool({
name: 'GetTicker24h',
description: `Retrieve 24h rolling market stats for all symbols on an exchange. Data is refreshed hourly. Returns symbols sorted by USD volume (std_quote_volume) descending.
Use this to build a pre-filtered symbol universe before fetching OHLC data for a scanner — it avoids requesting per-symbol OHLC data for thousands of symbols and consuming the bar budget.
Returned columns per symbol: ticker, exchange_id, base_asset, quote_asset, last_price, price_change_pct, quote_volume_24h, std_quote_volume (USD-normalized, null if unknown), bid_price, ask_price, open_24h, high_24h, low_24h, volume_24h, num_trades, timestamp_ms.
Parameters:
- exchange: Exchange name (BINANCE, COINBASE, or KRAKEN)
- limit (optional): Top N by volume. If omitted, returns all symbols.
- min_std_quote_volume (optional): Minimum USD-normalized 24h volume; excludes symbols below this threshold.
- market_type (optional): "spot" or "perp" (perpetual futures). If omitted, returns all.
- base_asset_contains (optional): Filter to symbols whose base asset contains this string (case-insensitive), e.g. "BTC".`,
schema: z.object({
exchange: z.string().describe('Exchange name: BINANCE, COINBASE, or KRAKEN'),
limit: z.number().int().optional().describe('Top N By Volume. If omitted, returns all symbols.'),
min_std_quote_volume: z.number().optional().describe('Minimum USD-normalized 24h volume. Excludes symbols below this threshold.'),
market_type: z.enum(['spot', 'perp']).optional().describe('Filter by market type: spot or perp. Omit for all.'),
base_asset_contains: z.string().optional().describe('Filter to symbols whose base asset contains this string (case-insensitive). E.g. "BTC".'),
}),
func: async ({ exchange, limit, min_std_quote_volume, market_type, base_asset_contains }) => {
const exchangeUpper = exchange.toUpperCase();
logger.debug({ exchange: exchangeUpper, limit, min_std_quote_volume, market_type, base_asset_contains }, 'GetTicker24h called');
const snapshot = getTicker24h(exchangeUpper);
if (!snapshot) {
return JSON.stringify({
exchange: exchangeUpper,
count: 0,
tickers: [],
note: 'No data available. Ticker24h is refreshed hourly — it may not be ready yet.',
});
}
let tickers = snapshot.tickers;
if (market_type === 'spot') {
tickers = tickers.filter(t => !t.ticker.includes(':'));
} else if (market_type === 'perp') {
tickers = tickers.filter(t => t.ticker.includes(':'));
}
if (base_asset_contains) {
const lower = base_asset_contains.toLowerCase();
tickers = tickers.filter(t => t.base_asset.toLowerCase().includes(lower));
}
if (min_std_quote_volume !== undefined) {
tickers = tickers.filter(
t => t.std_quote_volume !== undefined && !isNaN(t.std_quote_volume) && t.std_quote_volume >= min_std_quote_volume
);
}
// Sort by std_quote_volume descending (null/undefined/NaN last)
tickers = [...tickers].sort((a, b) => {
const av = a.std_quote_volume;
const bv = b.std_quote_volume;
if (av === undefined || av === null || isNaN(av)) return 1;
if (bv === undefined || bv === null || isNaN(bv)) return -1;
return bv - av;
});
if (limit !== undefined) {
tickers = tickers.slice(0, limit);
}
logger.info({ exchange: exchangeUpper, returned: tickers.length }, 'GetTicker24h result');
return JSON.stringify({
exchange: exchangeUpper,
count: tickers.length,
tickers,
});
},
});
}

15
gateway/src/tools/tool-registry.ts
View File

@@ -4,8 +4,10 @@ import type { MCPClientConnector } from '../harness/mcp-client.js';
import type { OHLCService } from '../services/ohlc-service.js';
import type { SymbolIndexService } from '../services/symbol-index-service.js';
import type { WorkspaceManager } from '../workspace/workspace-manager.js';
import type { Ticker24hSnapshot } from '../clients/zmq-relay-client.js';
import { createSymbolLookupTool } from './platform/symbol-lookup.tool.js';
import { createGetChartDataTool } from './platform/get-chart-data.tool.js';
import { createGetTicker24hTool } from './platform/get-ticker24h.tool.js';
import { createWebSearchTool } from './platform/web-search.tool.js';
import { createFetchPageTool } from './platform/fetch-page.tool.js';
import { createArxivSearchTool } from './platform/arxiv-search.tool.js';
@@ -34,6 +36,7 @@ export interface PlatformServices {
ohlcService?: OHLCService | (() => OHLCService | undefined);
symbolIndexService?: SymbolIndexService | (() => SymbolIndexService | undefined);
workspaceManager?: WorkspaceManager | (() => WorkspaceManager | undefined);
ticker24hGetter?: (exchange: string) => Ticker24hSnapshot | undefined;
tavilyApiKey?: string;
}
@@ -185,6 +188,18 @@ export class ToolRegistry {
break;
}
case 'GetTicker24h': {
if (this.platformServices.ticker24hGetter) {
tool = createGetTicker24hTool({
getTicker24h: this.platformServices.ticker24hGetter,
logger: this.logger,
});
} else {
this.logger.warn('ticker24hGetter not configured — GetTicker24h tool unavailable');
}
break;
}
case 'WebSearch': {
if (this.platformServices.tavilyApiKey) {
tool = createWebSearchTool({ apiKey: this.platformServices.tavilyApiKey, logger: this.logger });