backend redesign

doc.old/auth.md

@@ -0,0 +1,192 @@
+# Authentication & Secrets Management Guide
+
+## Overview
+
+Your system now has a complete encrypted secrets management solution with WebSocket authentication. All secrets (like API keys) are stored in an encrypted file, protected by a master password that users enter when connecting.
+
+## Architecture
+
+### Backend Components
+
+1. **Secrets Store** (`backend/src/secrets_manager/`)
+   - `crypto.py` - Argon2id password hashing + Fernet (AES-256) encryption
+   - `store.py` - SecretsStore class for managing encrypted secrets
+   - `cli.py` - Command-line interface for secrets management
+
+2. **Encrypted Storage**
+   - `backend/data/secrets.enc` - Encrypted secrets file
+   - `backend/data/.master.key` - Salt + verification hash (never stores actual password)
+   - Both files are created with 0600 permissions (owner-only access)
+
+3. **WebSocket Authentication** (`backend/src/main.py`)
+   - First message must be `auth` message
+   - On first use: requires password + confirmation → initializes secrets store
+   - Subsequent uses: requires password → unlocks secrets store
+   - Failed auth closes connection immediately
+
+### Frontend Components
+
+1. **Login Screen** (`web/src/components/LoginScreen.vue`)
+   - Shows before WebSocket connection
+   - Detects first-time setup and shows confirmation field
+   - Displays error messages for failed authentication
+
+2. **WebSocket Manager** (`web/src/composables/useWebSocket.ts`)
+   - Updated to send auth message on connect
+   - Returns auth result (success/failure)
+   - Prevents reconnection on auth failure
+
+3. **App Integration** (`web/src/App.vue`)
+   - Shows login screen until authenticated
+   - Initializes state sync only after successful auth
+
+## Security Features
+
+✓ **Password-based encryption** - Argon2id (OWASP recommended)
+✓ **AES-256 encryption** - Industry-standard Fernet cipher
+✓ **Salted passwords** - Unique salt per installation
+✓ **No plaintext storage** - Master password never stored
+✓ **Restricted permissions** - Secrets files are 0600 (owner-only)
+✓ **Constant-time verification** - Prevents timing attacks
+✓ **Auto-lock on disconnect** - Secrets cleared from memory
+
+## Usage
+
+### First Time Setup
+
+1. Start backend: `cd backend && python -m uvicorn src.main:app --reload --port 8080`
+2. Start frontend: `cd web && npm run dev`
+3. Open browser - you'll see "Welcome" screen
+4. Create a master password (with confirmation)
+5. System automatically migrates `ANTHROPIC_API_KEY` from `.env` to encrypted store
+
+### Subsequent Logins
+
+1. Start backend and frontend
+2. Enter your master password
+3. System unlocks and connects
+
+### Managing Secrets (CLI)
+
+```bash
+cd backend
+
+# List all secrets
+python -m secrets_manager.cli list
+
+# Add a new secret
+python -m secrets_manager.cli set MY_SECRET "secret-value"
+
+# Get a secret
+python -m secrets_manager.cli get ANTHROPIC_API_KEY
+
+# Change master password
+python -m secrets_manager.cli change-password
+
+# Backup secrets (encrypted)
+python -m secrets_manager.cli export backup.enc
+
+# Migrate from .env file
+python -m secrets_manager.cli migrate-from-env
+```
+
+### Managing Secrets (Python)
+
+```python
+from secrets_manager import SecretsStore
+
+# Initialize (first time)
+store = SecretsStore()
+store.initialize("my-password")
+
+# Unlock (subsequent times)
+store = SecretsStore()
+store.unlock("my-password")
+
+# Use secrets
+api_key = store.get("ANTHROPIC_API_KEY")
+store.set("NEW_SECRET", "value")
+store.delete("OLD_SECRET")
+
+# Change password
+store.change_master_password("old-password", "new-password")
+```
+
+## Protocol
+
+### Authentication Flow
+
+```
+Client → Server: { type: "auth", password: "...", confirm_password: "..." }
+Server → Client: { type: "auth_response", success: true, message: "..." }
+
+# If initialization needed:
+Client → Server: { type: "auth", password: "...", confirm_password: "..." }
+Server → Client: { type: "auth_response", success: false, needs_confirmation: true, ... }
+Client → Server: { type: "auth", password: "same", confirm_password: "same" }
+Server → Client: { type: "auth_response", success: true, message: "Initialized" }
+
+# After successful auth, normal protocol continues:
+Client → Server: { type: "hello", seqs: {...} }
+Server → Client: { type: "snapshot", ... }
+```
+
+### Error Codes
+
+- `1008` - Authentication failed (invalid password)
+- `1011` - Internal error during authentication
+
+## Migration from .env
+
+The system automatically migrates `ANTHROPIC_API_KEY` from `.env` when you first initialize the secrets store through the web interface. You can also use the CLI:
+
+```bash
+python -m secrets_manager.cli migrate-from-env
+# This will ask if you want to delete .env after migration
+```
+
+## Security Considerations
+
+1. **Master Password Strength** - Use a strong password (8+ characters recommended)
+2. **Backup** - Export encrypted backups regularly: `python -m secrets_manager.cli export backup.enc`
+3. **Environment** - Can still fall back to `.env` if secrets store not unlocked (for development)
+4. **Transport** - Use HTTPS/WSS in production (currently using HTTP/WS for development)
+
+## File Locations
+
+```
+backend/
+├── data/
+│   ├── secrets.enc           # Encrypted secrets (created on first auth)
+│   ├── .master.key           # Salt + verification (created on first auth)
+│   └── checkpoints.db        # Agent state (existing)
+└── src/
+    └── secrets/              # Secrets management module
+        ├── __init__.py
+        ├── crypto.py         # Cryptographic primitives
+        ├── store.py          # SecretsStore class
+        └── cli.py            # Command-line interface
+
+web/
+└── src/
+    ├── components/
+    │   └── LoginScreen.vue   # Authentication UI
+    └── composables/
+        └── useWebSocket.ts   # Updated with auth support
+```
+
+## Development Tips
+
+1. **Testing First-Time Setup**: Delete `backend/data/.master.key` to simulate first-time setup
+2. **Reset Password**: Delete both `.master.key` and `secrets.enc`, then reconnect
+3. **Debug Auth**: Check backend logs for authentication attempts
+4. **Bypass Auth (Dev)**: Set `ANTHROPIC_API_KEY` in `.env` and don't initialize secrets store
+
+## Next Steps
+
+Consider adding:
+- [ ] Password reset mechanism (security questions or backup codes)
+- [ ] Session timeout / auto-lock
+- [ ] Multi-user support with different passwords
+- [ ] Secret versioning / audit log
+- [ ] Integration with external secret managers (Vault, AWS Secrets Manager)

doc.old/data.md

@@ -0,0 +1,21 @@
+# Data Sources and ETL
+
+We'll use Pandas and ArcticDB as our data foundation. This gives easy access to numpy and a wide spectrum of Python analysis libraries.
+
+ArcticDB
+[ArcticDB](https://arcticdb.com/)
+We'll use unstructured dynamic schemas for flexibility and ad-hoc adjustment by the assistant.
+
+## Canonical Formats
+* OHLCV
+  * columns
+  * open_time high_time etc optional columns
+* Tick Data (later not now)
+* Order Book Data (later not now)
+
+## Symbol Definition
+Symbols must follow a canonical formatting that provides a unique string identifier:
+
+type|...(type dependent)...
+ohlc|exchange:base/quote
+

doc.old/design.md

@@ -0,0 +1,65 @@
+# AI-Native Trading Platform
+
+## Summary
+
+A full-stack, AI-first algorithmic trading platform in which LLMs act as strategy *authors* rather than strategy *executors*. Human traders express intent through natural language, chart drawings, and annotated support/resistance levels via chat or TradingView; an LLM agent interprets this intent and composes a fully self-contained **strategy executable** — a versioned, schema-validated artifact that runs deterministically with zero LLM involvement at runtime or in backtesting.
+
+The execution architecture is split into two nested kernels. The **strategy kernel** owns the complete mechanics of a strategy: signal DAG evaluation, position state, risk rules, and regime awareness, operating at strategy latency. It drives the **order kernel** — a minimal, fully-specified order state machine designed for deterministic, low-latency execution, in principle deployable to FPGA or kernel-bypass hardware. The two kernels communicate across a hard schema boundary, keeping the hot path free of any high-level logic.
+
+Data is handled through an abstract **feed specification** that treats historical and realtime, structured and unstructured sources uniformly. All signal computation — including NLP-based sentiment and on-chain analytics — uses local models only; the LLM is strictly a design-time tool.
+
+The platform is designed to be run **locally by the trader** for execution security, while strategy generation may be cloud-assisted. It leverages the Python data science ecosystem throughout, integrates with TradingView for chart-driven strategy authoring, and exposes multiple interaction surfaces including chat bots and JupyterLab notebooks. The local executable may still have strong dependencies on a centralized data cluster and logging facility.
+
+## Data
+
+* **Data source spec** — Abstract interface for historical (bulk/batch) and incremental (realtime) data, supporting both push and pull models, fully async. Sources are typed and versioned; any conforming adapter (REST, WebSocket, FIX, file, queue) plugs in identically.
+* **ETL pipeline** — Reusable transform/load tooling for normalizing raw source data into platform schemas, and as agent-callable tools for composing bespoke derived series on demand.
+* **Common schemas** — Canonical normalized forms: OHLCV, tick, order book (L1/L2/L3), trade prints, funding rates, open interest. Unstructured feeds (news, social, on-chain) carry a typed envelope with a payload and a platform-standard timestamp/symbol context.
+* **Feed registry** — Central catalog of available feeds with their schemas, latency class, and asset coverage. Both the agent layer and the execution kernel subscribe by feed ID, decoupled from transport.
+
+## Indicators & Signals
+
+* **Standard indicator library** — Common indicators (moving averages, VWAP, RSI, ATR, Bollinger, volume profile, etc.) implemented against the canonical schema. Stateless functions and stateful streaming variants both available.
+* **Custom signals in Python** — Arbitrary user/agent-defined signal nodes. Must conform to a standard `SignalNode` interface (inputs, outputs, params, `evaluate()`) so they are composable in the signal DAG and serializable into strategy executables.
+* **Unstructured signal pipeline** — Adapters that transform unstructured feeds into scalar or categorical signal outputs (e.g. FinBERT sentiment → float, on-chain anomaly detector → bool). Runs local models only; no LLM at signal evaluation time.
+
+## Strategy Kernel
+
+The outermost execution layer. Owns the complete mechanics of a strategy: signal DAG evaluation, position and risk state, regime awareness, and re-parameterization of the order kernel. Operates at human/strategy latency (milliseconds to seconds).
+
+* **Signal DAG** — Directed acyclic graph of signal nodes wired to feed subscriptions. Evaluated incrementally on new data. Nodes are typed, versioned, and serializable.
+* **Strategy state machine** — Tracks lifecycle: `IDLE → LOOKING → ENTERED → MANAGING → EXITED`. Transitions driven by signal DAG outputs and risk constraints.
+* **Risk layer** — Position sizing, exposure limits, drawdown circuit breakers, correlation guards. Evaluated before any order is emitted downward to the order kernel.
+* **Re-parameterization interface** — The strategy kernel may update order kernel parameters (price, size, urgency, validity) in response to new signal outputs without tearing down open orders — a clean boundary between strategic intent and mechanical execution.
+* **Strategy executable format** — The complete strategy (DAG topology, parameters, risk rules, feed subscriptions) is serialized to a self-contained artifact (YAML + optional Python modules). Generated by the agent layer; runnable with zero LLM dependency.
+* **Strategy Log** — A structured, append-only log of strategy events, including signal outputs, risk evaluations, and order parameter changes. Facilitates debugging, backtesting, and compliance auditing.
+
+## Order Kernel
+
+The innermost execution layer. Represents a single, fully specified order intent and manages its lifecycle mechanically. Designed for deterministic, low-latency operation — in principle implementable in an FPGA or a kernel-bypass network stack.
+
+* **Order specification** — Fully parameterized: symbol, side, type (market/limit/stop/conditional), price(s), size, time-in-force, validity window, venue routing hint. No ambiguity; all fields resolved before submission.
+* **Order lifecycle** — State machine: `PENDING → SUBMITTED → PARTIALLY_FILLED → FILLED | CANCELLED | REJECTED`. Transitions driven by exchange acknowledgements and fill reports.
+* **Timer primitives** — Deadline timers (cancel-on-expire), heartbeat timers (re-quote), and interval timers (TWAP/VWAP slice scheduling). All timers are pure data, evaluatable without process context.
+* **Venue adapter interface** — Thin, typed interface to exchange connectivity (FIX, REST, WebSocket, on-chain). The order kernel emits a normalized order; the adapter translates to venue wire format.
+* **FPGA / low-latency boundary** — The order spec and lifecycle state machine are designed with a hard schema so they can be frozen and handed off to a hardware execution path. The strategy kernel communicates to this boundary via a defined message contract, not shared memory or function calls.
+* **Order Log** — A structured, append-only log of order events, including submission, fill, and cancellation details. Facilitates debugging, backtesting, and compliance auditing.
+* **Credentials Store** — Secure storage for API keys, wallet keys, secrets, and other sensitive information. Available only to the order kernel and no other layer.
+
+## Agent Layer
+
+The LLM-powered strategy authoring environment. Cloud-hosted or local. Produces strategy executables; never participates in live execution or backtesting hot paths.
+
+* **Tool framework** — Typed, schema-validated tools callable by the LLM: `query_chart`, `compute_indicator`, `detect_sr_levels`, `fetch_news_sentiment`, `annotate_drawing`, `compose_signal_node`, `emit_strategy`. Each tool has a `to_executable_node()` method so its output can be frozen into the strategy artifact.
+* **TradingView integration** — Drawing/annotation export (trend lines, SR levels, zones, Fibonacci) ingested as structured JSON, interpreted by the agent into signal DAG nodes (e.g. `sr_proximity`, `trendline_cross`).
+* **Context manager** — Maintains the trader's working session: active symbol, timeframe, chart drawings, prior signal definitions, risk preferences. Persisted across turns; fed as structured context to each LLM call.
+* **Strategy compiler** — Takes the agent's composed tool outputs and renders a validated, schema-checked strategy executable. Validation errors are reflected back to the agent for self-correction before the artifact is finalized.
+* **Prompt/strategy versioning** — Every generated strategy artifact is version-controlled (git). The generating prompt, model version, and tool call trace are stored alongside the artifact for full auditability.
+
+## Interaction Layer
+
+How traders communicate with the system. Multiple surfaces; all feed the same agent layer.
+
+* **Chat interface** — Discord, Telegram, or Slack bot. Accepts natural language, images of charts, TradingView drawing exports. Returns chart images, strategy summaries, backtest results. Primary low-friction UI.
+* **Notebook interface** — JupyterLab environment for power users. Full access to the data layer, indicator library, and agent tools. Chart generation (Plotly/Bokeh) for rich visual feedback, with outputs forwardable to chat channels.
+* **TradingView plugin** — Embeds into an existing TradingView-based website

doc.old/libraries.md

@@ -0,0 +1,3 @@
+ArcticDB
+[ArcticDB](https://arcticdb.com/)
+We'll use unstructured dynamic schemas.

doc.old/mvp.md

@@ -0,0 +1,32 @@
+# Minimum Viable Prototype
+
+* Chat interface: web+telegram
+* Agent loop
+  * Memory
+  * RAG
+  * System Prompt
+    * Soul / Mission
+    * Current chart candles
+    * TV Shapes
+    * AI self-prompt
+    * User Prompt
+* Agent tools
+  * Edit strategy code
+  * Read/Write TradingView shapes
+  * Fetch data
+  * TradingView Snapshot
+  * Plotting
+  * Cron
+  * Docker, never host access
+  * Edit AI self-prompt
+* One candle data source: Dexorder
+* Python data science stack with charts
+* Indicator library
+* Standard historical backtest
+* Strategy wrapper / sandbox
+* One execution engine: Dexorder
+* Website
+  * TradingView interaction
+  * Authentication
+  * Settings / Prompts
+  * Chat area with image support and zoom

doc.old/trendspider.md

@@ -0,0 +1,38 @@
+Four Main Functions
+* Charting and Analysis
+* Strategy Development and Testing
+* Idea Generation
+* Trade Timing and Execution
+
+# Charting and Analysis
+* "Smart Charts" autodetects chart patterns, trendlines, and plots them
+* Multi-timeframe plots on same chart (e.g. 200 SMA from the daily)
+* Multi-symbol view
+* Sidebar data
+* Mobile app
+* Browser extension detects symbols across pages
+
+# Strategy Development and Testing
+* Strategies seem to be lego components:
+    * "Price Close" "Less than" "Wedge Falling (Top)"
+    * "Any of the following" "happened within" "15 candles" on "Current symbol"
+* The rest is obvious
+
+# Idea Generation
+* Scanners provided filtered symbol lists according to whatever conditions
+* News feeds like recent halts, WallStBets symbols, retail flow alerts, social media chatter
+
+# Trade Timing and Execution
+* Alerts on chart elements have "cadence":
+  * "Break Through" Trigger when price opens & closes, or closes & opens on different sides of the buffer area.
+  * "Touch" Trigger when price touches or trades through the buffer area, but does not Break Through it.
+  * "Bounce" Triggers when price Touches the buffer area but does not Break Through on the next candle.
+* Alerts on any visual element
+* Bots can either trade or sent alerts
+* Condition tree structure of strategies allows a quick red/green view of what sections are currently true/false, allowing a discretionary trader to maybe make an exceptional decision
+
+# Assistant
+* "Quick Commands" finds UI elements
+  * Quick search box basically with minor AI assistance for "add to chat" etc
+  * Maybe use lightweight model agent to choose from a range of UI actions like "add indicator" etc.
+*