dexorder/ai
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

backend redesign

Browse Source
This commit is contained in:
Tim Olson
2026-03-11 18:47:11 -04:00
parent 8ff277c8c6
commit e99ef5d2dd
210 changed files with 12147 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
doc/agent_redesign.md Normal file
View File

@@ -0,0 +1,9 @@
Generally use skills instead of subagents, except for the analysis subagent.
## User-specific files
* Indicators
* Strategies
* Watchlists
* Preferences
* Trading style
* Charting / colors

192
doc/auth.md
View File

@@ -1,192 +0,0 @@
# Authentication & Secrets Management Guide
## Overview
Your system now has a complete encrypted secrets management solution with WebSocket authentication. All secrets (like API keys) are stored in an encrypted file, protected by a master password that users enter when connecting.
## Architecture
### Backend Components
1. **Secrets Store** (`backend/src/secrets_manager/`)
- `crypto.py` - Argon2id password hashing + Fernet (AES-256) encryption
- `store.py` - SecretsStore class for managing encrypted secrets
- `cli.py` - Command-line interface for secrets management
2. **Encrypted Storage**
- `backend/data/secrets.enc` - Encrypted secrets file
- `backend/data/.master.key` - Salt + verification hash (never stores actual password)
- Both files are created with 0600 permissions (owner-only access)
3. **WebSocket Authentication** (`backend/src/main.py`)
- First message must be `auth` message
- On first use: requires password + confirmation → initializes secrets store
- Subsequent uses: requires password → unlocks secrets store
- Failed auth closes connection immediately
### Frontend Components
1. **Login Screen** (`web/src/components/LoginScreen.vue`)
- Shows before WebSocket connection
- Detects first-time setup and shows confirmation field
- Displays error messages for failed authentication
2. **WebSocket Manager** (`web/src/composables/useWebSocket.ts`)
- Updated to send auth message on connect
- Returns auth result (success/failure)
- Prevents reconnection on auth failure
3. **App Integration** (`web/src/App.vue`)
- Shows login screen until authenticated
- Initializes state sync only after successful auth
## Security Features
✓ **Password-based encryption** - Argon2id (OWASP recommended)
✓ **AES-256 encryption** - Industry-standard Fernet cipher
✓ **Salted passwords** - Unique salt per installation
✓ **No plaintext storage** - Master password never stored
✓ **Restricted permissions** - Secrets files are 0600 (owner-only)
✓ **Constant-time verification** - Prevents timing attacks
✓ **Auto-lock on disconnect** - Secrets cleared from memory
## Usage
### First Time Setup
1. Start backend: `cd backend && python -m uvicorn src.main:app --reload --port 8080`
2. Start frontend: `cd web && npm run dev`
3. Open browser - you'll see "Welcome" screen
4. Create a master password (with confirmation)
5. System automatically migrates `ANTHROPIC_API_KEY` from `.env` to encrypted store
### Subsequent Logins
1. Start backend and frontend
2. Enter your master password
3. System unlocks and connects
### Managing Secrets (CLI)
```bash
cd backend
# List all secrets
python -m secrets_manager.cli list
# Add a new secret
python -m secrets_manager.cli set MY_SECRET "secret-value"
# Get a secret
python -m secrets_manager.cli get ANTHROPIC_API_KEY
# Change master password
python -m secrets_manager.cli change-password
# Backup secrets (encrypted)
python -m secrets_manager.cli export backup.enc
# Migrate from .env file
python -m secrets_manager.cli migrate-from-env
```
### Managing Secrets (Python)
```python
from secrets_manager import SecretsStore
# Initialize (first time)
store = SecretsStore()
store.initialize("my-password")
# Unlock (subsequent times)
store = SecretsStore()
store.unlock("my-password")
# Use secrets
api_key = store.get("ANTHROPIC_API_KEY")
store.set("NEW_SECRET", "value")
store.delete("OLD_SECRET")
# Change password
store.change_master_password("old-password", "new-password")
```
## Protocol
### Authentication Flow
```
Client → Server: { type: "auth", password: "...", confirm_password: "..." }
Server → Client: { type: "auth_response", success: true, message: "..." }
# If initialization needed:
Client → Server: { type: "auth", password: "...", confirm_password: "..." }
Server → Client: { type: "auth_response", success: false, needs_confirmation: true, ... }
Client → Server: { type: "auth", password: "same", confirm_password: "same" }
Server → Client: { type: "auth_response", success: true, message: "Initialized" }
# After successful auth, normal protocol continues:
Client → Server: { type: "hello", seqs: {...} }
Server → Client: { type: "snapshot", ... }
```
### Error Codes
- `1008` - Authentication failed (invalid password)
- `1011` - Internal error during authentication
## Migration from .env
The system automatically migrates `ANTHROPIC_API_KEY` from `.env` when you first initialize the secrets store through the web interface. You can also use the CLI:
```bash
python -m secrets_manager.cli migrate-from-env
# This will ask if you want to delete .env after migration
```
## Security Considerations
1. **Master Password Strength** - Use a strong password (8+ characters recommended)
2. **Backup** - Export encrypted backups regularly: `python -m secrets_manager.cli export backup.enc`
3. **Environment** - Can still fall back to `.env` if secrets store not unlocked (for development)
4. **Transport** - Use HTTPS/WSS in production (currently using HTTP/WS for development)
## File Locations
```
backend/
├── data/
│ ├── secrets.enc # Encrypted secrets (created on first auth)
│ ├── .master.key # Salt + verification (created on first auth)
│ └── checkpoints.db # Agent state (existing)
└── src/
└── secrets/ # Secrets management module
├── __init__.py
├── crypto.py # Cryptographic primitives
├── store.py # SecretsStore class
└── cli.py # Command-line interface
web/
└── src/
├── components/
│ └── LoginScreen.vue # Authentication UI
└── composables/
└── useWebSocket.ts # Updated with auth support
```
## Development Tips
1. **Testing First-Time Setup**: Delete `backend/data/.master.key` to simulate first-time setup
2. **Reset Password**: Delete both `.master.key` and `secrets.enc`, then reconnect
3. **Debug Auth**: Check backend logs for authentication attempts
4. **Bypass Auth (Dev)**: Set `ANTHROPIC_API_KEY` in `.env` and don't initialize secrets store
## Next Steps
Consider adding:
- [ ] Password reset mechanism (security questions or backup codes)
- [ ] Session timeout / auto-lock
- [ ] Multi-user support with different passwords
- [ ] Secret versioning / audit log
- [ ] Integration with external secret managers (Vault, AWS Secrets Manager)

110
doc/backend_redesign.md Normal file
View File

@@ -0,0 +1,110 @@
# aiignore
# This is not implemented yet and are just notes for Tim
# Overview
We need a realtime data system that is scalable and durable, so we have the following architecture:
* Protobufs over ZeroMQ for data streaming
* Ingestors
* Realtime data subscriptions (tick data)
* Historical data queries (OHLC)
* Everything pushes to Kafka topics
* Kafka
* Durable append logs for incoming and in-process data
* Topics maintained by Flink in redesign/flink/src/main/resources/topics.yaml
* Flink
* Raw ingestor streams are read from Kafka
* Deduplication
* Builds OHLCs
* Apache Iceberg
* Historical data storage
# Configuration
All systems should use two YAML configuration files that are mounted by k8s from a ConfigMap and / or Secrets. Keep secrets separate from config.
When a configuration or secrets item is needed, describe it in resdesign/doc/config.md
# Ingest
Ingestion API
* all symbols
* exchange id (BINANCE)
* market_id (BTC/USDT)
* market_type
* Spot
* description (Bitcoin/Tether on Binance)
* column names ( ['open', 'high', 'low', 'close', 'volume', 'taker_vol', 'maker_vol'])
* name
* exchange
* base asset
* quote asset
* earliest time
* tick size
* supported periods
* Centralized data streaming backend
* Ingestion of tick, ohlc, news, etc. into Kafka by worker gatherers
* Flink with:
* zmq pubsub
* (seq, time) key for every row in a tick series
* every series also has seq->time and time->seq indexes
* Sequence tickers with strict seq's AND time index (seq can just be row counter autoincrement)
* Historical data
* Apache Iceberg
* Clients query here first
* Backfill service
* Quote Server
* Realtime current prices for selected quote currencies
* Workspace
* Current chart, indicators, drawings, etc.
* Always in context, must be brief. Data series are a reference not the actual data.
* Analysis
* Analysis engines are short-running and always tied to a user
* Free users lose pod and data when session times out
* Conda available with many preinstalled packages
* Pip & Conda configured to install
* Src dir r/w with git
* Indicators
* Strategies
* Analysis
* Request Context
* User ID
* Workspace ID
* Channel
* Telegram
* Web
* Website
* Current vue site
* Gateway
* Websocket gateway
* Authentication
* User Featureset / License Info added to requests/headers
* Relays data pub/sub to web/mobile clients
* Routes agent chat to/from user container
* Active channel features
* TV Chart
* Text chat
* Plot out
* Voice/Audio
* Static file server
* Kafka
* Temp Gateway files (image responses, etc.)
* Logs
* Kafka
* Strategy Logs
* Order/Execution Logs
* Chat Logs
* User ID Topic has TTL based on license
* Agent Framework
* Soul file
* Tool set (incl subagents)
* LLM choice
* RAG namespace
* Agents
* Top-level coordinator
* TradingView agent
* Indicators, Drawings, Annotations
* Research Agent
* Pandas/Polars analysis
* Plot generation
* License Manager
* Kafka Topics Doc w/ schemas
*

18
doc/config.md Normal file
View File

@@ -0,0 +1,18 @@
This file describes all the configuration options used by all components. All configuration is divided into regular config and secrets, and k8s will mount either or both as a yaml file accessible to the process.
# Configuration
* `flink_hostname`
* ... various zmq ports for flink ...
* `iceberg_catalog_hostname`
* `iceberg_catalog_port`
* `iceberg_catalog_database`
* etc
# Secrets
* `iceberg_catalog_username`
* `iceberg_catalog_password`
* etc.

21
doc/data.md
View File

@@ -1,21 +0,0 @@
# Data Sources and ETL
We'll use Pandas and ArcticDB as our data foundation. This gives easy access to numpy and a wide spectrum of Python analysis libraries.
ArcticDB
[ArcticDB](https://arcticdb.com/)
We'll use unstructured dynamic schemas for flexibility and ad-hoc adjustment by the assistant.
## Canonical Formats
* OHLCV
* columns
* open_time high_time etc optional columns
* Tick Data (later not now)
* Order Book Data (later not now)
## Symbol Definition
Symbols must follow a canonical formatting that provides a unique string identifier:
type|...(type dependent)...
ohlc|exchange:base/quote

65
doc/design.md
View File

@@ -1,65 +0,0 @@
# AI-Native Trading Platform
## Summary
A full-stack, AI-first algorithmic trading platform in which LLMs act as strategy *authors* rather than strategy *executors*. Human traders express intent through natural language, chart drawings, and annotated support/resistance levels via chat or TradingView; an LLM agent interprets this intent and composes a fully self-contained **strategy executable** — a versioned, schema-validated artifact that runs deterministically with zero LLM involvement at runtime or in backtesting.
The execution architecture is split into two nested kernels. The **strategy kernel** owns the complete mechanics of a strategy: signal DAG evaluation, position state, risk rules, and regime awareness, operating at strategy latency. It drives the **order kernel** — a minimal, fully-specified order state machine designed for deterministic, low-latency execution, in principle deployable to FPGA or kernel-bypass hardware. The two kernels communicate across a hard schema boundary, keeping the hot path free of any high-level logic.
Data is handled through an abstract **feed specification** that treats historical and realtime, structured and unstructured sources uniformly. All signal computation — including NLP-based sentiment and on-chain analytics — uses local models only; the LLM is strictly a design-time tool.
The platform is designed to be run **locally by the trader** for execution security, while strategy generation may be cloud-assisted. It leverages the Python data science ecosystem throughout, integrates with TradingView for chart-driven strategy authoring, and exposes multiple interaction surfaces including chat bots and JupyterLab notebooks. The local executable may still have strong dependencies on a centralized data cluster and logging facility.
## Data
* **Data source spec** — Abstract interface for historical (bulk/batch) and incremental (realtime) data, supporting both push and pull models, fully async. Sources are typed and versioned; any conforming adapter (REST, WebSocket, FIX, file, queue) plugs in identically.
* **ETL pipeline** — Reusable transform/load tooling for normalizing raw source data into platform schemas, and as agent-callable tools for composing bespoke derived series on demand.
* **Common schemas** — Canonical normalized forms: OHLCV, tick, order book (L1/L2/L3), trade prints, funding rates, open interest. Unstructured feeds (news, social, on-chain) carry a typed envelope with a payload and a platform-standard timestamp/symbol context.
* **Feed registry** — Central catalog of available feeds with their schemas, latency class, and asset coverage. Both the agent layer and the execution kernel subscribe by feed ID, decoupled from transport.
## Indicators & Signals
* **Standard indicator library** — Common indicators (moving averages, VWAP, RSI, ATR, Bollinger, volume profile, etc.) implemented against the canonical schema. Stateless functions and stateful streaming variants both available.
* **Custom signals in Python** — Arbitrary user/agent-defined signal nodes. Must conform to a standard `SignalNode` interface (inputs, outputs, params, `evaluate()`) so they are composable in the signal DAG and serializable into strategy executables.
* **Unstructured signal pipeline** — Adapters that transform unstructured feeds into scalar or categorical signal outputs (e.g. FinBERT sentiment → float, on-chain anomaly detector → bool). Runs local models only; no LLM at signal evaluation time.
## Strategy Kernel
The outermost execution layer. Owns the complete mechanics of a strategy: signal DAG evaluation, position and risk state, regime awareness, and re-parameterization of the order kernel. Operates at human/strategy latency (milliseconds to seconds).
* **Signal DAG** — Directed acyclic graph of signal nodes wired to feed subscriptions. Evaluated incrementally on new data. Nodes are typed, versioned, and serializable.
* **Strategy state machine** — Tracks lifecycle: `IDLE → LOOKING → ENTERED → MANAGING → EXITED`. Transitions driven by signal DAG outputs and risk constraints.
* **Risk layer** — Position sizing, exposure limits, drawdown circuit breakers, correlation guards. Evaluated before any order is emitted downward to the order kernel.
* **Re-parameterization interface** — The strategy kernel may update order kernel parameters (price, size, urgency, validity) in response to new signal outputs without tearing down open orders — a clean boundary between strategic intent and mechanical execution.
* **Strategy executable format** — The complete strategy (DAG topology, parameters, risk rules, feed subscriptions) is serialized to a self-contained artifact (YAML + optional Python modules). Generated by the agent layer; runnable with zero LLM dependency.
* **Strategy Log** — A structured, append-only log of strategy events, including signal outputs, risk evaluations, and order parameter changes. Facilitates debugging, backtesting, and compliance auditing.
## Order Kernel
The innermost execution layer. Represents a single, fully specified order intent and manages its lifecycle mechanically. Designed for deterministic, low-latency operation — in principle implementable in an FPGA or a kernel-bypass network stack.
* **Order specification** — Fully parameterized: symbol, side, type (market/limit/stop/conditional), price(s), size, time-in-force, validity window, venue routing hint. No ambiguity; all fields resolved before submission.
* **Order lifecycle** — State machine: `PENDING → SUBMITTED → PARTIALLY_FILLED → FILLED | CANCELLED | REJECTED`. Transitions driven by exchange acknowledgements and fill reports.
* **Timer primitives** — Deadline timers (cancel-on-expire), heartbeat timers (re-quote), and interval timers (TWAP/VWAP slice scheduling). All timers are pure data, evaluatable without process context.
* **Venue adapter interface** — Thin, typed interface to exchange connectivity (FIX, REST, WebSocket, on-chain). The order kernel emits a normalized order; the adapter translates to venue wire format.
* **FPGA / low-latency boundary** — The order spec and lifecycle state machine are designed with a hard schema so they can be frozen and handed off to a hardware execution path. The strategy kernel communicates to this boundary via a defined message contract, not shared memory or function calls.
* **Order Log** — A structured, append-only log of order events, including submission, fill, and cancellation details. Facilitates debugging, backtesting, and compliance auditing.
* **Credentials Store** — Secure storage for API keys, wallet keys, secrets, and other sensitive information. Available only to the order kernel and no other layer.
## Agent Layer
The LLM-powered strategy authoring environment. Cloud-hosted or local. Produces strategy executables; never participates in live execution or backtesting hot paths.
* **Tool framework** — Typed, schema-validated tools callable by the LLM: `query_chart`, `compute_indicator`, `detect_sr_levels`, `fetch_news_sentiment`, `annotate_drawing`, `compose_signal_node`, `emit_strategy`. Each tool has a `to_executable_node()` method so its output can be frozen into the strategy artifact.
* **TradingView integration** — Drawing/annotation export (trend lines, SR levels, zones, Fibonacci) ingested as structured JSON, interpreted by the agent into signal DAG nodes (e.g. `sr_proximity`, `trendline_cross`).
* **Context manager** — Maintains the trader's working session: active symbol, timeframe, chart drawings, prior signal definitions, risk preferences. Persisted across turns; fed as structured context to each LLM call.
* **Strategy compiler** — Takes the agent's composed tool outputs and renders a validated, schema-checked strategy executable. Validation errors are reflected back to the agent for self-correction before the artifact is finalized.
* **Prompt/strategy versioning** — Every generated strategy artifact is version-controlled (git). The generating prompt, model version, and tool call trace are stored alongside the artifact for full auditability.
## Interaction Layer
How traders communicate with the system. Multiple surfaces; all feed the same agent layer.
* **Chat interface** — Discord, Telegram, or Slack bot. Accepts natural language, images of charts, TradingView drawing exports. Returns chart images, strategy summaries, backtest results. Primary low-friction UI.
* **Notebook interface** — JupyterLab environment for power users. Full access to the data layer, indicator library, and agent tools. Chart generation (Plotly/Bokeh) for rich visual feedback, with outputs forwardable to chat channels.
* **TradingView plugin** — Embeds into an existing TradingView-based website

3
doc/libraries.md
View File

@@ -1,3 +0,0 @@
ArcticDB
[ArcticDB](https://arcticdb.com/)
We'll use unstructured dynamic schemas.

32
doc/mvp.md
View File

@@ -1,32 +0,0 @@
# Minimum Viable Prototype
* Chat interface: slack/telegram/whatsapp etc
* Agent loop
* Memory
* RAG
* System Prompt
* Soul / Mission
* Current chart candles
* TV Shapes
* AI self-prompt
* User Prompt
* Agent tools
* Edit strategy code
* Read/Write TradingView shapes
* Fetch data
* TradingView Snapshot
* Plotting
* Cron
* Docker, never host access
* Edit AI self-prompt
* One candle data source: Dexorder
* Python data science stack with charts
* Indicator library
* Standard historical backtest
* Strategy wrapper / sandbox
* One execution engine: Dexorder
* Website
* TradingView interaction
* Authentication
* Settings / Prompts
* Chat area with image support and zoom