288 lines
6.7 KiB
Markdown
288 lines
6.7 KiB
Markdown
# Kubernetes Deployment
|
|
|
|
This directory contains Kubernetes manifests using [Kustomize](https://kustomize.io/) for managing dev and production environments.
|
|
|
|
## Structure
|
|
|
|
```
|
|
deploy/k8s/
|
|
├── base/ # Base manifests (shared)
|
|
│ ├── backend.yaml
|
|
│ ├── web.yaml
|
|
│ ├── ingress.yaml
|
|
│ ├── init.yaml
|
|
│ └── kustomization.yaml
|
|
├── dev/ # Dev overlay (minikube)
|
|
│ ├── infrastructure.yaml # Kafka, Postgres, MinIO, Flink, Relay, Ingestor
|
|
│ ├── ingress-dev.yaml # Dev ingress (dexorder.local)
|
|
│ ├── patches.yaml # Dev-specific patches
|
|
│ ├── kustomization.yaml
|
|
│ └── secrets/
|
|
│ ├── *.yaml # Actual secrets (gitignored)
|
|
│ └── *.yaml.example # Templates
|
|
├── prod/ # Production overlay
|
|
│ ├── patches.yaml # Prod patches (replicas, resources, gVisor)
|
|
│ ├── kustomization.yaml
|
|
│ └── secrets/
|
|
│ ├── *.yaml # Actual secrets (gitignored)
|
|
│ └── *.yaml.example # Templates
|
|
└── configmaps/ # Shared ConfigMaps
|
|
├── relay-config.yaml
|
|
├── ingestor-config.yaml
|
|
└── flink-config.yaml
|
|
```
|
|
|
|
## Dev Environment (Minikube)
|
|
|
|
### Prerequisites
|
|
|
|
- [minikube](https://minikube.sigs.k8s.io/docs/start/)
|
|
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
|
- Docker
|
|
|
|
### Quick Start
|
|
|
|
```bash
|
|
# Start everything
|
|
bin/dev start
|
|
|
|
# Access the application
|
|
# Web UI: http://dexorder.local/cryptochimp/
|
|
# Backend: ws://dexorder.local/ws
|
|
|
|
# In another terminal, start tunnel for ingress
|
|
bin/dev tunnel
|
|
```
|
|
|
|
### Managing Dev Environment
|
|
|
|
```bash
|
|
# Rebuild images after code changes
|
|
bin/dev rebuild
|
|
|
|
# Redeploy services
|
|
bin/dev deploy
|
|
|
|
# Full restart (rebuild + redeploy)
|
|
bin/dev restart
|
|
|
|
# View status
|
|
bin/dev status
|
|
|
|
# View logs
|
|
bin/dev logs relay
|
|
bin/dev logs ingestor
|
|
bin/dev logs flink-jobmanager
|
|
|
|
# Open shell in pod
|
|
bin/dev shell relay
|
|
|
|
# Clean everything
|
|
bin/dev clean
|
|
|
|
# Stop minikube
|
|
bin/dev stop
|
|
```
|
|
|
|
### Setting Up Secrets (Dev)
|
|
|
|
```bash
|
|
# Copy example secrets
|
|
cd deploy/k8s/dev/secrets/
|
|
cp ai-secrets.yaml.example ai-secrets.yaml
|
|
cp postgres-secret.yaml.example postgres-secret.yaml
|
|
cp minio-secret.yaml.example minio-secret.yaml
|
|
cp ingestor-secrets.yaml.example ingestor-secrets.yaml
|
|
|
|
# Edit with actual values
|
|
vim ai-secrets.yaml # Add your Anthropic API key
|
|
|
|
# Apply to cluster
|
|
bin/secret-update dev
|
|
|
|
# Or update a specific secret
|
|
bin/secret-update dev ai-secrets
|
|
```
|
|
|
|
### Updating Configs (Dev)
|
|
|
|
```bash
|
|
# Edit config files
|
|
vim deploy/configmaps/relay-config.yaml
|
|
|
|
# Apply changes
|
|
bin/config-update dev
|
|
|
|
# Or update specific config
|
|
bin/config-update dev relay-config
|
|
```
|
|
|
|
### Dev vs Docker Compose
|
|
|
|
The minikube dev environment mirrors production more closely than docker-compose:
|
|
|
|
| Feature | docker-compose | minikube |
|
|
|---------|---------------|----------|
|
|
| Environment parity | ❌ Different from prod | ✅ Same as prod |
|
|
| Secrets management | `.env` files | K8s Secrets |
|
|
| Configuration | Volume mounts | ConfigMaps |
|
|
| Service discovery | DNS by service name | K8s Services |
|
|
| Ingress/routing | Port mapping | nginx-ingress |
|
|
| Resource limits | Limited support | Full K8s resources |
|
|
| Init containers | No | Yes |
|
|
| Readiness probes | No | Yes |
|
|
|
|
## Production Environment
|
|
|
|
### Prerequisites
|
|
|
|
- Access to production Kubernetes cluster
|
|
- `kubectl` configured with production context
|
|
- Production secrets prepared
|
|
|
|
### Setting Up Secrets (Prod)
|
|
|
|
```bash
|
|
# Copy example secrets
|
|
cd deploy/k8s/prod/secrets/
|
|
cp ai-secrets.yaml.example ai-secrets.yaml
|
|
cp postgres-secret.yaml.example postgres-secret.yaml
|
|
# ... etc
|
|
|
|
# Edit with production values
|
|
vim ai-secrets.yaml
|
|
|
|
# Apply to cluster (will prompt for confirmation)
|
|
bin/secret-update prod
|
|
|
|
# Or update specific secret
|
|
bin/secret-update prod ai-secrets
|
|
```
|
|
|
|
### Updating Configs (Prod)
|
|
|
|
```bash
|
|
# Edit production configs if needed
|
|
vim deploy/configmaps/relay-config.yaml
|
|
|
|
# Apply changes (will prompt for confirmation)
|
|
bin/config-update prod
|
|
```
|
|
|
|
### Deploying to Production
|
|
|
|
```bash
|
|
# Verify kubectl context
|
|
kubectl config current-context
|
|
|
|
# Apply manifests
|
|
kubectl apply -k deploy/k8s/prod/
|
|
|
|
# Check rollout status
|
|
kubectl rollout status statefulset/ai-backend
|
|
kubectl rollout status deployment/ai-web
|
|
|
|
# View status
|
|
kubectl get pods,svc,ingress
|
|
```
|
|
|
|
## Kustomize Overlays
|
|
|
|
### Dev Overlay
|
|
|
|
- **imagePullPolicy: Never** - Uses locally built images
|
|
- **Infrastructure services** - Kafka, Postgres, MinIO, Flink, Relay, Ingestor
|
|
- **Local ingress** - `dexorder.local` (requires `/etc/hosts` entry)
|
|
- **No gVisor** - RuntimeClass removed (not available in minikube)
|
|
- **Single replicas** - Minimal resource usage
|
|
|
|
### Prod Overlay
|
|
|
|
- **imagePullPolicy: Always** - Pulls from registry
|
|
- **Multiple replicas** - HA configuration
|
|
- **Resource limits** - CPU/memory constraints
|
|
- **gVisor** - Security sandbox via RuntimeClass
|
|
- **Production ingress** - `dexorder.ai` with TLS
|
|
|
|
## Infrastructure Services (Dev Only)
|
|
|
|
These services are included in the dev environment but are expected to be managed separately in production:
|
|
|
|
- **Kafka** - KRaft mode (no Zookeeper), single broker
|
|
- **PostgreSQL** - Iceberg catalog metadata
|
|
- **MinIO** - S3-compatible object storage
|
|
- **Iceberg REST Catalog** - Table metadata
|
|
- **Flink** - JobManager + TaskManager
|
|
- **Relay** - ZMQ message router
|
|
- **Ingestor** - CCXT data fetcher
|
|
|
|
In production, you would typically use:
|
|
- Managed Kafka (Confluent Cloud, MSK, etc.)
|
|
- Managed PostgreSQL (RDS, Cloud SQL, etc.)
|
|
- Object storage (S3, GCS, Azure Blob)
|
|
- Flink Kubernetes Operator or managed Flink
|
|
|
|
## Troubleshooting
|
|
|
|
### Minikube not starting
|
|
|
|
```bash
|
|
minikube delete
|
|
minikube start --cpus=6 --memory=12g --driver=docker
|
|
```
|
|
|
|
### Images not found
|
|
|
|
Make sure you're using minikube's docker daemon:
|
|
|
|
```bash
|
|
eval $(minikube docker-env)
|
|
bin/dev rebuild
|
|
```
|
|
|
|
### Ingress not working
|
|
|
|
Start minikube tunnel in another terminal:
|
|
|
|
```bash
|
|
bin/dev tunnel
|
|
```
|
|
|
|
### Secrets not found
|
|
|
|
Create secrets from examples:
|
|
|
|
```bash
|
|
cd deploy/k8s/dev/secrets/
|
|
cp *.example *.yaml
|
|
vim ai-secrets.yaml # Edit with actual values
|
|
bin/secret-update dev
|
|
```
|
|
|
|
### Pods not starting
|
|
|
|
Check events and logs:
|
|
|
|
```bash
|
|
kubectl get events --sort-by=.metadata.creationTimestamp
|
|
kubectl describe pod <pod-name>
|
|
kubectl logs <pod-name>
|
|
```
|
|
|
|
## CI/CD Integration
|
|
|
|
For automated deployments, you can use:
|
|
|
|
```bash
|
|
# Build and push images
|
|
docker build -t registry.example.com/dexorder/ai-web:$TAG .
|
|
docker push registry.example.com/dexorder/ai-web:$TAG
|
|
|
|
# Update kustomization with new tag
|
|
cd deploy/k8s/prod
|
|
kustomize edit set image dexorder/ai-web=registry.example.com/dexorder/ai-web:$TAG
|
|
|
|
# Deploy
|
|
kubectl apply -k deploy/k8s/prod/
|
|
```
|