backend redesign

doc.old/design.md

@@ -0,0 +1,65 @@
+# AI-Native Trading Platform
+
+## Summary
+
+A full-stack, AI-first algorithmic trading platform in which LLMs act as strategy *authors* rather than strategy *executors*. Human traders express intent through natural language, chart drawings, and annotated support/resistance levels via chat or TradingView; an LLM agent interprets this intent and composes a fully self-contained **strategy executable** — a versioned, schema-validated artifact that runs deterministically with zero LLM involvement at runtime or in backtesting.
+
+The execution architecture is split into two nested kernels. The **strategy kernel** owns the complete mechanics of a strategy: signal DAG evaluation, position state, risk rules, and regime awareness, operating at strategy latency. It drives the **order kernel** — a minimal, fully-specified order state machine designed for deterministic, low-latency execution, in principle deployable to FPGA or kernel-bypass hardware. The two kernels communicate across a hard schema boundary, keeping the hot path free of any high-level logic.
+
+Data is handled through an abstract **feed specification** that treats historical and realtime, structured and unstructured sources uniformly. All signal computation — including NLP-based sentiment and on-chain analytics — uses local models only; the LLM is strictly a design-time tool.
+
+The platform is designed to be run **locally by the trader** for execution security, while strategy generation may be cloud-assisted. It leverages the Python data science ecosystem throughout, integrates with TradingView for chart-driven strategy authoring, and exposes multiple interaction surfaces including chat bots and JupyterLab notebooks. The local executable may still have strong dependencies on a centralized data cluster and logging facility.
+
+## Data
+
+* **Data source spec** — Abstract interface for historical (bulk/batch) and incremental (realtime) data, supporting both push and pull models, fully async. Sources are typed and versioned; any conforming adapter (REST, WebSocket, FIX, file, queue) plugs in identically.
+* **ETL pipeline** — Reusable transform/load tooling for normalizing raw source data into platform schemas, and as agent-callable tools for composing bespoke derived series on demand.
+* **Common schemas** — Canonical normalized forms: OHLCV, tick, order book (L1/L2/L3), trade prints, funding rates, open interest. Unstructured feeds (news, social, on-chain) carry a typed envelope with a payload and a platform-standard timestamp/symbol context.
+* **Feed registry** — Central catalog of available feeds with their schemas, latency class, and asset coverage. Both the agent layer and the execution kernel subscribe by feed ID, decoupled from transport.
+
+## Indicators & Signals
+
+* **Standard indicator library** — Common indicators (moving averages, VWAP, RSI, ATR, Bollinger, volume profile, etc.) implemented against the canonical schema. Stateless functions and stateful streaming variants both available.
+* **Custom signals in Python** — Arbitrary user/agent-defined signal nodes. Must conform to a standard `SignalNode` interface (inputs, outputs, params, `evaluate()`) so they are composable in the signal DAG and serializable into strategy executables.
+* **Unstructured signal pipeline** — Adapters that transform unstructured feeds into scalar or categorical signal outputs (e.g. FinBERT sentiment → float, on-chain anomaly detector → bool). Runs local models only; no LLM at signal evaluation time.
+
+## Strategy Kernel
+
+The outermost execution layer. Owns the complete mechanics of a strategy: signal DAG evaluation, position and risk state, regime awareness, and re-parameterization of the order kernel. Operates at human/strategy latency (milliseconds to seconds).
+
+* **Signal DAG** — Directed acyclic graph of signal nodes wired to feed subscriptions. Evaluated incrementally on new data. Nodes are typed, versioned, and serializable.
+* **Strategy state machine** — Tracks lifecycle: `IDLE → LOOKING → ENTERED → MANAGING → EXITED`. Transitions driven by signal DAG outputs and risk constraints.
+* **Risk layer** — Position sizing, exposure limits, drawdown circuit breakers, correlation guards. Evaluated before any order is emitted downward to the order kernel.
+* **Re-parameterization interface** — The strategy kernel may update order kernel parameters (price, size, urgency, validity) in response to new signal outputs without tearing down open orders — a clean boundary between strategic intent and mechanical execution.
+* **Strategy executable format** — The complete strategy (DAG topology, parameters, risk rules, feed subscriptions) is serialized to a self-contained artifact (YAML + optional Python modules). Generated by the agent layer; runnable with zero LLM dependency.
+* **Strategy Log** — A structured, append-only log of strategy events, including signal outputs, risk evaluations, and order parameter changes. Facilitates debugging, backtesting, and compliance auditing.
+
+## Order Kernel
+
+The innermost execution layer. Represents a single, fully specified order intent and manages its lifecycle mechanically. Designed for deterministic, low-latency operation — in principle implementable in an FPGA or a kernel-bypass network stack.
+
+* **Order specification** — Fully parameterized: symbol, side, type (market/limit/stop/conditional), price(s), size, time-in-force, validity window, venue routing hint. No ambiguity; all fields resolved before submission.
+* **Order lifecycle** — State machine: `PENDING → SUBMITTED → PARTIALLY_FILLED → FILLED | CANCELLED | REJECTED`. Transitions driven by exchange acknowledgements and fill reports.
+* **Timer primitives** — Deadline timers (cancel-on-expire), heartbeat timers (re-quote), and interval timers (TWAP/VWAP slice scheduling). All timers are pure data, evaluatable without process context.
+* **Venue adapter interface** — Thin, typed interface to exchange connectivity (FIX, REST, WebSocket, on-chain). The order kernel emits a normalized order; the adapter translates to venue wire format.
+* **FPGA / low-latency boundary** — The order spec and lifecycle state machine are designed with a hard schema so they can be frozen and handed off to a hardware execution path. The strategy kernel communicates to this boundary via a defined message contract, not shared memory or function calls.
+* **Order Log** — A structured, append-only log of order events, including submission, fill, and cancellation details. Facilitates debugging, backtesting, and compliance auditing.
+* **Credentials Store** — Secure storage for API keys, wallet keys, secrets, and other sensitive information. Available only to the order kernel and no other layer.
+
+## Agent Layer
+
+The LLM-powered strategy authoring environment. Cloud-hosted or local. Produces strategy executables; never participates in live execution or backtesting hot paths.
+
+* **Tool framework** — Typed, schema-validated tools callable by the LLM: `query_chart`, `compute_indicator`, `detect_sr_levels`, `fetch_news_sentiment`, `annotate_drawing`, `compose_signal_node`, `emit_strategy`. Each tool has a `to_executable_node()` method so its output can be frozen into the strategy artifact.
+* **TradingView integration** — Drawing/annotation export (trend lines, SR levels, zones, Fibonacci) ingested as structured JSON, interpreted by the agent into signal DAG nodes (e.g. `sr_proximity`, `trendline_cross`).
+* **Context manager** — Maintains the trader's working session: active symbol, timeframe, chart drawings, prior signal definitions, risk preferences. Persisted across turns; fed as structured context to each LLM call.
+* **Strategy compiler** — Takes the agent's composed tool outputs and renders a validated, schema-checked strategy executable. Validation errors are reflected back to the agent for self-correction before the artifact is finalized.
+* **Prompt/strategy versioning** — Every generated strategy artifact is version-controlled (git). The generating prompt, model version, and tool call trace are stored alongside the artifact for full auditability.
+
+## Interaction Layer
+
+How traders communicate with the system. Multiple surfaces; all feed the same agent layer.
+
+* **Chat interface** — Discord, Telegram, or Slack bot. Accepts natural language, images of charts, TradingView drawing exports. Returns chart images, strategy summaries, backtest results. Primary low-friction UI.
+* **Notebook interface** — JupyterLab environment for power users. Full access to the data layer, indicator library, and agent tools. Chart generation (Plotly/Bokeh) for rich visual feedback, with outputs forwardable to chat channels.
+* **TradingView plugin** — Embeds into an existing TradingView-based website