ai/gateway/knowledge/README.md

Dexorder Knowledge Base

This directory contains global knowledge documents that are automatically loaded into the RAG system as platform-wide knowledge (user_id="0").

Structure

• platform/: Platform architecture and capabilities
• trading/: Trading concepts and fundamentals
• indicators/: Indicator development and usage
• strategies/: Strategy development and patterns

Document Format

Documents should be in Markdown format with:

• Clear headings for chunking
• Optional YAML frontmatter for tags
• Code examples where relevant
• Cross-references to other docs

Example with Frontmatter

---
tags: [trading, risk-management, position-sizing]
---

# Risk Management

Content here...

How It Works

1. At gateway startup, the DocumentLoader scans this directory
2. Each markdown file is chunked by headers (max ~1000 tokens per chunk)
3. Chunks are embedded using the configured embedding service
4. Embeddings are stored in Qdrant with user_id="0" (global namespace)
5. Content hash tracking enables incremental updates

Updating Documents

During Development

• Edit markdown files
• Restart gateway or call reload endpoint: POST /admin/reload-knowledge

In Production

• Update markdown files in git
• Deploy new version
• Gateway will detect changes and update vectors automatically

RAG Integration

When users query the agent:

1. Their query is embedded
2. Qdrant searches both global (user_id="0") and user-specific vectors
3. Relevant chunks from these docs are included in context
4. LLM generates response with platform knowledge

Adding New Documents

1. Create markdown file in appropriate subdirectory
2. Use clear section headers (##, ###) for automatic chunking
3. Include practical examples and code samples
4. Add tags in frontmatter if using complex categorization
5. Restart gateway or reload knowledge

Best Practices

• Keep chunks focused: Each section should cover one topic
• Use examples: Code samples and practical examples help
• Link concepts: Reference other docs for deeper dives
• Update regularly: Keep knowledge current with platform changes
• Test queries: Verify RAG retrieves relevant chunks

Maintenance

The DocumentLoader tracks:

• Content hashes for change detection
• Number of chunks per document
• Last update timestamps

Check logs for load statistics:

Knowledge documents loaded: { loaded: 5, updated: 2, skipped: 3 }

Monitor Qdrant collection stats:

GET /health
{
  "qdrantVectors": 1234,
  "qdrantIndexed": 1234
}