backend redesign
This commit is contained in:
287
deploy/k8s/README.md
Normal file
287
deploy/k8s/README.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# Kubernetes Deployment
|
||||
|
||||
This directory contains Kubernetes manifests using [Kustomize](https://kustomize.io/) for managing dev and production environments.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
deploy/k8s/
|
||||
├── base/ # Base manifests (shared)
|
||||
│ ├── backend.yaml
|
||||
│ ├── web.yaml
|
||||
│ ├── ingress.yaml
|
||||
│ ├── init.yaml
|
||||
│ └── kustomization.yaml
|
||||
├── dev/ # Dev overlay (minikube)
|
||||
│ ├── infrastructure.yaml # Kafka, Postgres, MinIO, Flink, Relay, Ingestor
|
||||
│ ├── ingress-dev.yaml # Dev ingress (dexorder.local)
|
||||
│ ├── patches.yaml # Dev-specific patches
|
||||
│ ├── kustomization.yaml
|
||||
│ └── secrets/
|
||||
│ ├── *.yaml # Actual secrets (gitignored)
|
||||
│ └── *.yaml.example # Templates
|
||||
├── prod/ # Production overlay
|
||||
│ ├── patches.yaml # Prod patches (replicas, resources, gVisor)
|
||||
│ ├── kustomization.yaml
|
||||
│ └── secrets/
|
||||
│ ├── *.yaml # Actual secrets (gitignored)
|
||||
│ └── *.yaml.example # Templates
|
||||
└── configmaps/ # Shared ConfigMaps
|
||||
├── relay-config.yaml
|
||||
├── ingestor-config.yaml
|
||||
└── flink-config.yaml
|
||||
```
|
||||
|
||||
## Dev Environment (Minikube)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- [minikube](https://minikube.sigs.k8s.io/docs/start/)
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
||||
- Docker
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Start everything
|
||||
bin/dev start
|
||||
|
||||
# Access the application
|
||||
# Web UI: http://dexorder.local/cryptochimp/
|
||||
# Backend: ws://dexorder.local/ws
|
||||
|
||||
# In another terminal, start tunnel for ingress
|
||||
bin/dev tunnel
|
||||
```
|
||||
|
||||
### Managing Dev Environment
|
||||
|
||||
```bash
|
||||
# Rebuild images after code changes
|
||||
bin/dev rebuild
|
||||
|
||||
# Redeploy services
|
||||
bin/dev deploy
|
||||
|
||||
# Full restart (rebuild + redeploy)
|
||||
bin/dev restart
|
||||
|
||||
# View status
|
||||
bin/dev status
|
||||
|
||||
# View logs
|
||||
bin/dev logs relay
|
||||
bin/dev logs ingestor
|
||||
bin/dev logs flink-jobmanager
|
||||
|
||||
# Open shell in pod
|
||||
bin/dev shell relay
|
||||
|
||||
# Clean everything
|
||||
bin/dev clean
|
||||
|
||||
# Stop minikube
|
||||
bin/dev stop
|
||||
```
|
||||
|
||||
### Setting Up Secrets (Dev)
|
||||
|
||||
```bash
|
||||
# Copy example secrets
|
||||
cd deploy/k8s/dev/secrets/
|
||||
cp ai-secrets.yaml.example ai-secrets.yaml
|
||||
cp postgres-secret.yaml.example postgres-secret.yaml
|
||||
cp minio-secret.yaml.example minio-secret.yaml
|
||||
cp ingestor-secrets.yaml.example ingestor-secrets.yaml
|
||||
|
||||
# Edit with actual values
|
||||
vim ai-secrets.yaml # Add your Anthropic API key
|
||||
|
||||
# Apply to cluster
|
||||
bin/secret-update dev
|
||||
|
||||
# Or update a specific secret
|
||||
bin/secret-update dev ai-secrets
|
||||
```
|
||||
|
||||
### Updating Configs (Dev)
|
||||
|
||||
```bash
|
||||
# Edit config files
|
||||
vim deploy/configmaps/relay-config.yaml
|
||||
|
||||
# Apply changes
|
||||
bin/config-update dev
|
||||
|
||||
# Or update specific config
|
||||
bin/config-update dev relay-config
|
||||
```
|
||||
|
||||
### Dev vs Docker Compose
|
||||
|
||||
The minikube dev environment mirrors production more closely than docker-compose:
|
||||
|
||||
| Feature | docker-compose | minikube |
|
||||
|---------|---------------|----------|
|
||||
| Environment parity | ❌ Different from prod | ✅ Same as prod |
|
||||
| Secrets management | `.env` files | K8s Secrets |
|
||||
| Configuration | Volume mounts | ConfigMaps |
|
||||
| Service discovery | DNS by service name | K8s Services |
|
||||
| Ingress/routing | Port mapping | nginx-ingress |
|
||||
| Resource limits | Limited support | Full K8s resources |
|
||||
| Init containers | No | Yes |
|
||||
| Readiness probes | No | Yes |
|
||||
|
||||
## Production Environment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Access to production Kubernetes cluster
|
||||
- `kubectl` configured with production context
|
||||
- Production secrets prepared
|
||||
|
||||
### Setting Up Secrets (Prod)
|
||||
|
||||
```bash
|
||||
# Copy example secrets
|
||||
cd deploy/k8s/prod/secrets/
|
||||
cp ai-secrets.yaml.example ai-secrets.yaml
|
||||
cp postgres-secret.yaml.example postgres-secret.yaml
|
||||
# ... etc
|
||||
|
||||
# Edit with production values
|
||||
vim ai-secrets.yaml
|
||||
|
||||
# Apply to cluster (will prompt for confirmation)
|
||||
bin/secret-update prod
|
||||
|
||||
# Or update specific secret
|
||||
bin/secret-update prod ai-secrets
|
||||
```
|
||||
|
||||
### Updating Configs (Prod)
|
||||
|
||||
```bash
|
||||
# Edit production configs if needed
|
||||
vim deploy/configmaps/relay-config.yaml
|
||||
|
||||
# Apply changes (will prompt for confirmation)
|
||||
bin/config-update prod
|
||||
```
|
||||
|
||||
### Deploying to Production
|
||||
|
||||
```bash
|
||||
# Verify kubectl context
|
||||
kubectl config current-context
|
||||
|
||||
# Apply manifests
|
||||
kubectl apply -k deploy/k8s/prod/
|
||||
|
||||
# Check rollout status
|
||||
kubectl rollout status statefulset/ai-backend
|
||||
kubectl rollout status deployment/ai-web
|
||||
|
||||
# View status
|
||||
kubectl get pods,svc,ingress
|
||||
```
|
||||
|
||||
## Kustomize Overlays
|
||||
|
||||
### Dev Overlay
|
||||
|
||||
- **imagePullPolicy: Never** - Uses locally built images
|
||||
- **Infrastructure services** - Kafka, Postgres, MinIO, Flink, Relay, Ingestor
|
||||
- **Local ingress** - `dexorder.local` (requires `/etc/hosts` entry)
|
||||
- **No gVisor** - RuntimeClass removed (not available in minikube)
|
||||
- **Single replicas** - Minimal resource usage
|
||||
|
||||
### Prod Overlay
|
||||
|
||||
- **imagePullPolicy: Always** - Pulls from registry
|
||||
- **Multiple replicas** - HA configuration
|
||||
- **Resource limits** - CPU/memory constraints
|
||||
- **gVisor** - Security sandbox via RuntimeClass
|
||||
- **Production ingress** - `dexorder.ai` with TLS
|
||||
|
||||
## Infrastructure Services (Dev Only)
|
||||
|
||||
These services are included in the dev environment but are expected to be managed separately in production:
|
||||
|
||||
- **Kafka** - KRaft mode (no Zookeeper), single broker
|
||||
- **PostgreSQL** - Iceberg catalog metadata
|
||||
- **MinIO** - S3-compatible object storage
|
||||
- **Iceberg REST Catalog** - Table metadata
|
||||
- **Flink** - JobManager + TaskManager
|
||||
- **Relay** - ZMQ message router
|
||||
- **Ingestor** - CCXT data fetcher
|
||||
|
||||
In production, you would typically use:
|
||||
- Managed Kafka (Confluent Cloud, MSK, etc.)
|
||||
- Managed PostgreSQL (RDS, Cloud SQL, etc.)
|
||||
- Object storage (S3, GCS, Azure Blob)
|
||||
- Flink Kubernetes Operator or managed Flink
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Minikube not starting
|
||||
|
||||
```bash
|
||||
minikube delete
|
||||
minikube start --cpus=6 --memory=12g --driver=docker
|
||||
```
|
||||
|
||||
### Images not found
|
||||
|
||||
Make sure you're using minikube's docker daemon:
|
||||
|
||||
```bash
|
||||
eval $(minikube docker-env)
|
||||
bin/dev rebuild
|
||||
```
|
||||
|
||||
### Ingress not working
|
||||
|
||||
Start minikube tunnel in another terminal:
|
||||
|
||||
```bash
|
||||
bin/dev tunnel
|
||||
```
|
||||
|
||||
### Secrets not found
|
||||
|
||||
Create secrets from examples:
|
||||
|
||||
```bash
|
||||
cd deploy/k8s/dev/secrets/
|
||||
cp *.example *.yaml
|
||||
vim ai-secrets.yaml # Edit with actual values
|
||||
bin/secret-update dev
|
||||
```
|
||||
|
||||
### Pods not starting
|
||||
|
||||
Check events and logs:
|
||||
|
||||
```bash
|
||||
kubectl get events --sort-by=.metadata.creationTimestamp
|
||||
kubectl describe pod <pod-name>
|
||||
kubectl logs <pod-name>
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
For automated deployments, you can use:
|
||||
|
||||
```bash
|
||||
# Build and push images
|
||||
docker build -t registry.example.com/dexorder/ai-web:$TAG .
|
||||
docker push registry.example.com/dexorder/ai-web:$TAG
|
||||
|
||||
# Update kustomization with new tag
|
||||
cd deploy/k8s/prod
|
||||
kustomize edit set image dexorder/ai-web=registry.example.com/dexorder/ai-web:$TAG
|
||||
|
||||
# Deploy
|
||||
kubectl apply -k deploy/k8s/prod/
|
||||
```
|
||||
Reference in New Issue
Block a user