backend redesign

test/README.md

@@ -0,0 +1,109 @@
+# Test Clients
+
+Test clients for the DexOrder trading system.
+
+## History Client
+
+Tests the historical OHLC data request/response pattern between clients, Flink, and ingestors.
+
+### Quick Start
+
+```bash
+cd history_client
+./run-test.sh
+```
+
+This will:
+1. Start all required services (Kafka, Flink, Ingestor)
+2. Wait for services to initialize
+3. Run the test client to query historical data
+4. Display the results
+
+### What it tests
+
+- **ZMQ Communication**: Client → Flink REQ/REP pattern (port 5559)
+- **Work Distribution**: Flink → Ingestor PUB/SUB with exchange prefix filtering (port 5555)
+- **Response Channel**: Ingestor → Flink DEALER/ROUTER pattern (port 5556)
+- **Data Flow**: Request → Ingestor fetches data → Response back to Flink → Response to client
+
+### Expected Flow
+
+1. **Client** sends OHLCRequest to Flink (REQ/REP)
+   - Ticker: `BINANCE:BTC/USDT`
+   - Period: 3600s (1 hour)
+   - Range: Jan 1-7, 2026
+
+2. **Flink** publishes DataRequest to ingestor work queue (PUB/SUB)
+   - Topic prefix: `BINANCE:`
+   - Any ingestor subscribed to BINANCE can respond
+
+3. **Ingestor** receives request, fetches data, sends back response
+   - Uses CCXT to fetch from exchange
+   - Sends DataResponse via DEALER socket
+   - Also writes to Kafka for Flink processing
+
+4. **Flink** receives response, sends back to client
+   - Matches response by request_id
+   - Returns data or error to waiting client
+
+### Manual Testing
+
+Run the Python client directly:
+
+```bash
+cd history_client
+pip install pyzmq
+python client.py
+```
+
+Edit `client.py` to customize:
+- Flink hostname and port
+- Ticker symbol
+- Time range
+- Period (e.g., 3600 for 1h, 86400 for 1d)
+
+## Docker Compose Profiles
+
+The test client uses a Docker Compose profile to avoid starting automatically:
+
+```bash
+# Start all services
+docker-compose up -d
+
+# Run test client
+docker-compose --profile test up history-test-client
+
+# Or start everything including test
+docker-compose --profile test up
+```
+
+## Troubleshooting
+
+### Service logs
+
+```bash
+docker-compose logs -f ingestor
+docker-compose logs -f flink-jobmanager
+```
+
+### Check ZMQ ports
+
+```bash
+# From inside Flink container
+netstat -tlnp | grep 555
+```
+
+### Verify ingestor subscriptions
+
+Check ingestor logs for:
+```
+Subscribed to exchange prefix: BINANCE:
+Subscribed to exchange prefix: COINBASE:
+```
+
+### Test without Docker
+
+1. Start Kafka: `docker-compose up -d kafka`
+2. Build and run Flink app locally
+3. Run ingestor: `cd ingestor && npm start`
+4. Run test: `cd test/history_client && python client.py`