redesign fully scaffolded and web login works

2026-03-17 20:10:47 -04:00
parent b9cc397e05
commit f6bd22a8ef
143 changed files with 17317 additions and 693 deletions
--- a/gateway/src/harness/README.md
+++ b/gateway/src/harness/README.md
@@ -0,0 +1,351 @@
+# Agent Harness
+
+Comprehensive agent orchestration system for Dexorder AI platform, built on LangChain.js and LangGraph.js.
+
+## Architecture Overview
+
+```
+gateway/src/harness/
+├── memory/              # Storage layer (Redis + Iceberg + Qdrant)
+├── skills/              # Individual capabilities (markdown + TypeScript)
+├── subagents/           # Specialized agents with multi-file memory
+├── workflows/           # LangGraph state machines
+├── tools/               # Platform tools (non-MCP)
+├── config/              # Configuration files
+└── index.ts             # Main exports
+```
+
+## Core Components
+
+### 1. Memory Layer (`memory/`)
+
+Tiered storage architecture as per [architecture discussion](/chat/harness-rag.txt):
+
+- **Redis**: Hot state (active sessions, checkpoints)
+- **Iceberg**: Cold storage (durable conversations, analytics)
+- **Qdrant**: Vector search (RAG, semantic memory)
+
+**Key Files:**
+- `checkpoint-saver.ts`: LangGraph checkpoint persistence
+- `conversation-store.ts`: Message history management
+- `rag-retriever.ts`: Vector similarity search
+- `embedding-service.ts`: Text→vector conversion
+- `session-context.ts`: User context with channel metadata
+
+### 2. Skills (`skills/`)
+
+Self-contained capabilities with markdown definitions:
+
+- `*.skill.md`: Human-readable documentation
+- `*.ts`: Implementation extending `BaseSkill`
+- Input validation and error handling
+- Can use LLM, MCP tools, or platform tools
+
+**Example:**
+```typescript
+import { MarketAnalysisSkill } from './skills';
+
+const skill = new MarketAnalysisSkill(logger, model);
+const result = await skill.execute({
+  context: userContext,
+  parameters: { ticker: 'BTC/USDT', period: '4h' }
+});
+```
+
+See [skills/README.md](skills/README.md) for authoring guide.
+
+### 3. Subagents (`subagents/`)
+
+Specialized agents with multi-file memory:
+
+```
+subagents/
+  code-reviewer/
+    config.yaml              # Model, memory files, capabilities
+    system-prompt.md         # System instructions
+    memory/                  # Multi-file knowledge base
+      review-guidelines.md
+      common-patterns.md
+      best-practices.md
+    index.ts                 # Implementation
+```
+
+**Features:**
+- Dedicated system prompts
+- Split memory into logical files (better organization)
+- Model overrides
+- Capability tagging
+
+**Example:**
+```typescript
+const codeReviewer = await createCodeReviewerSubagent(model, logger, basePath);
+const review = await codeReviewer.execute({ userContext }, strategyCode);
+```
+
+### 4. Workflows (`workflows/`)
+
+LangGraph state machines with:
+
+- Validation loops (retry with fixes)
+- Human-in-the-loop (approval gates)
+- Multi-step orchestration
+- Error recovery
+
+**Example Workflows:**
+- `strategy-validation/`: Code review → backtest → risk → approval
+- `trading-request/`: Analysis → risk → approval → execute
+
+See individual workflow READMEs for details.
+
+### 5. Configuration (`config/`)
+
+YAML-based configuration:
+
+- `models.yaml`: LLM providers, routing, rate limits
+- `subagent-routing.yaml`: When to use which subagent
+
+## User Context
+
+Enhanced session context with channel awareness for multi-channel support:
+
+```typescript
+interface UserContext {
+  userId: string;
+  sessionId: string;
+  license: UserLicense;
+
+  activeChannel: {
+    type: 'websocket' | 'telegram' | 'slack' | 'discord';
+    channelUserId: string;
+    capabilities: {
+      supportsMarkdown: boolean;
+      supportsImages: boolean;
+      supportsButtons: boolean;
+      maxMessageLength: number;
+    };
+  };
+
+  conversationHistory: BaseMessage[];
+  relevantMemories: MemoryChunk[];
+  workspaceState: WorkspaceContext;
+}
+```
+
+This allows workflows to:
+- Route responses to correct channel
+- Format output for channel capabilities
+- Handle channel-specific interactions (buttons, voice, etc.)
+
+## Storage Architecture
+
+Based on [harness-rag.txt discussion](../../chat/harness-rag.txt):
+
+### Hot Path (Redis)
+- Active checkpoints (TTL: 1 hour)
+- Recent messages (last 50)
+- Session metadata
+- Fast reads for active conversations
+
+### Cold Path (Iceberg)
+- Full conversation history (partitioned by user_id, session_id)
+- Checkpoint snapshots
+- Time-travel queries
+- GDPR-compliant deletion with compaction
+
+### Vector Search (Qdrant)
+- Conversation embeddings
+- Long-term memory
+- RAG retrieval
+- Payload-indexed by user_id for fast GDPR deletion
+- **Global knowledge base** (user_id="0") loaded from markdown files
+
+### GDPR Compliance
+
+```typescript
+// Delete user data across all stores
+await conversationStore.deleteUserData(userId);
+await ragRetriever.deleteUserData(userId);
+await checkpointSaver.delete(userId);
+await containerManager.deleteContainer(userId);
+
+// Iceberg physical delete
+await icebergTable.expire_snapshots();
+await icebergTable.rewrite_data_files();
+```
+
+## Standard Patterns
+
+### Validation Loop (Retry with Fixes)
+
+```typescript
+graph.addConditionalEdges('validate', (state) => {
+  if (state.errors.length > 0 && state.retryCount < 3) {
+    return 'fix_errors';  // Loop back
+  }
+  return state.errors.length === 0 ? 'approve' : 'reject';
+});
+```
+
+### Human-in-the-Loop (Approval Gates)
+
+```typescript
+const approvalNode = async (state) => {
+  // Send to user's channel
+  await sendToChannel(state.userContext.activeChannel, {
+    type: 'approval_request',
+    data: { /* details */ }
+  });
+
+  // LangGraph pauses here via Interrupt
+  // Resume with user input: graph.invoke(state, { ...resumeConfig })
+
+  return { approvalRequested: true };
+};
+```
+
+## Getting Started
+
+### 1. Install Dependencies
+
+Already in `gateway/package.json`:
+```json
+{
+  "@langchain/core": "^0.3.24",
+  "@langchain/langgraph": "^0.2.26",
+  "@langchain/anthropic": "^0.3.8",
+  "ioredis": "^5.4.2"
+}
+```
+
+### 2. Initialize Memory Layer
+
+```typescript
+import Redis from 'ioredis';
+import {
+  TieredCheckpointSaver,
+  ConversationStore,
+  EmbeddingService,
+  RAGRetriever
+} from './harness/memory';
+
+const redis = new Redis(process.env.REDIS_URL);
+
+const checkpointSaver = new TieredCheckpointSaver(redis, logger);
+const conversationStore = new ConversationStore(redis, logger);
+const embeddings = new EmbeddingService({ provider: 'openai', apiKey }, logger);
+const ragRetriever = new RAGRetriever({ url: QDRANT_URL }, logger);
+
+await ragRetriever.initialize();
+```
+
+### 3. Create Subagents
+
+```typescript
+import { createCodeReviewerSubagent } from './harness/subagents';
+import { ModelRouter } from './llm/router';
+
+const model = await modelRouter.route(query, license);
+const codeReviewer = await createCodeReviewerSubagent(
+  model,
+  logger,
+  'gateway/src/harness/subagents/code-reviewer'
+);
+```
+
+### 4. Build Workflows
+
+```typescript
+import { createStrategyValidationWorkflow } from './harness/workflows';
+
+const workflow = await createStrategyValidationWorkflow(
+  model,
+  codeReviewer,
+  mcpBacktestFn,
+  logger,
+  'gateway/src/harness/workflows/strategy-validation/config.yaml'
+);
+
+const result = await workflow.execute({
+  userContext,
+  strategyCode: '...',
+  ticker: 'BTC/USDT',
+  timeframe: '4h'
+});
+```
+
+### 5. Use Skills
+
+```typescript
+import { MarketAnalysisSkill } from './harness/skills';
+
+const skill = new MarketAnalysisSkill(logger, model);
+const analysis = await skill.execute({
+  context: userContext,
+  parameters: { ticker: 'BTC/USDT', period: '1h' }
+});
+```
+
+## Global Knowledge System
+
+The harness includes a document loader that automatically loads markdown files from `gateway/knowledge/` into Qdrant as global knowledge (user_id="0").
+
+### Directory Structure
+```
+gateway/knowledge/
+  ├── platform/          # Platform capabilities and architecture
+  ├── trading/           # Trading concepts and fundamentals
+  ├── indicators/        # Indicator development guides
+  └── strategies/        # Strategy patterns and examples
+```
+
+### How It Works
+
+1. **Startup**: Documents are loaded automatically when gateway starts
+2. **Chunking**: Intelligent splitting by markdown headers (~1000 tokens/chunk)
+3. **Embedding**: Chunks are embedded using configured embedding service
+4. **Storage**: Stored in Qdrant with user_id="0" (global namespace)
+5. **Updates**: Content hashing detects changes for incremental updates
+
+### RAG Query Flow
+
+When a user sends a message:
+1. Query is embedded using same embedding service
+2. Qdrant searches vectors with filter: `user_id = current_user OR user_id = "0"`
+3. Results include both user-specific and global knowledge
+4. Relevant chunks are added to LLM context
+5. LLM generates response with platform knowledge
+
+### Managing Knowledge
+
+**Add new documents**:
+```bash
+# Create markdown file in appropriate directory
+echo "# New Topic" > gateway/knowledge/platform/new-topic.md
+
+# Reload knowledge (development)
+curl -X POST http://localhost:3000/admin/reload-knowledge
+```
+
+**Check stats**:
+```bash
+curl http://localhost:3000/admin/knowledge-stats
+```
+
+**In production**: Just deploy updated markdown files - they'll be loaded on startup.
+
+See [gateway/knowledge/README.md](../../knowledge/README.md) for detailed documentation.
+
+## Next Steps
+
+1. **Implement Iceberg Integration**: Complete TODOs in checkpoint-saver.ts and conversation-store.ts
+2. **Add More Subagents**: Risk analyzer, market analyst, etc.
+3. **Implement Interrupts**: Full human-in-the-loop with LangGraph interrupts
+4. **Add Platform Tools**: Market data queries, chart rendering, etc.
+5. **Expand Knowledge Base**: Add more platform documentation to knowledge/
+
+## References
+
+- Architecture discussion: [chat/harness-rag.txt](../../chat/harness-rag.txt)
+- LangGraph docs: https://langchain-ai.github.io/langgraphjs/
+- Qdrant docs: https://qdrant.tech/documentation/
+- Apache Iceberg: https://iceberg.apache.org/docs/latest/