ai/deploy/k8s/README.md

Kubernetes Deployment

This directory contains Kubernetes manifests using Kustomize for managing dev and production environments.

Structure

deploy/k8s/
├── base/                    # Base manifests (shared)
│   ├── backend.yaml
│   ├── web.yaml
│   ├── ingress.yaml
│   ├── init.yaml
│   └── kustomization.yaml
├── dev/                     # Dev overlay (minikube)
│   ├── infrastructure.yaml  # Kafka, Postgres, MinIO, Flink, Relay, Ingestor
│   ├── ingress-dev.yaml     # Dev ingress (dexorder.local)
│   ├── patches.yaml         # Dev-specific patches
│   ├── kustomization.yaml
│   └── secrets/
│       ├── *.yaml           # Actual secrets (gitignored)
│       └── *.yaml.example   # Templates
├── prod/                    # Production overlay
│   ├── patches.yaml         # Prod patches (replicas, resources, gVisor)
│   ├── kustomization.yaml
│   └── secrets/
│       ├── *.yaml           # Actual secrets (gitignored)
│       └── *.yaml.example   # Templates
└── configmaps/              # Shared ConfigMaps
    ├── relay-config.yaml
    ├── ingestor-config.yaml
    └── flink-config.yaml

Dev Environment (Minikube)

Prerequisites

• minikube
• kubectl
• Docker

Quick Start

# Start everything
bin/dev start

# Access the application
# Web UI: http://dexorder.local/cryptochimp/
# Backend: ws://dexorder.local/ws

# In another terminal, start tunnel for ingress
bin/dev tunnel

Managing Dev Environment

# Rebuild images after code changes
bin/dev rebuild

# Redeploy services
bin/dev deploy

# Full restart (rebuild + redeploy)
bin/dev restart

# View status
bin/dev status

# View logs
bin/dev logs relay
bin/dev logs ingestor
bin/dev logs flink-jobmanager

# Open shell in pod
bin/dev shell relay

# Clean everything
bin/dev clean

# Stop minikube
bin/dev stop

Setting Up Secrets (Dev)

# Copy example secrets
cd deploy/k8s/dev/secrets/
cp ai-secrets.yaml.example ai-secrets.yaml
cp postgres-secret.yaml.example postgres-secret.yaml
cp minio-secret.yaml.example minio-secret.yaml
cp ingestor-secrets.yaml.example ingestor-secrets.yaml

# Edit with actual values
vim ai-secrets.yaml  # Add your Anthropic API key

# Apply to cluster
bin/secret-update dev

# Or update a specific secret
bin/secret-update dev ai-secrets

Updating Configs (Dev)

# Edit config files
vim deploy/configmaps/relay-config.yaml

# Apply changes
bin/config-update dev

# Or update specific config
bin/config-update dev relay-config

Dev vs Docker Compose

The minikube dev environment mirrors production more closely than docker-compose:

Feature	docker-compose	minikube
Environment parity	❌ Different from prod	✅ Same as prod
Secrets management	.env files	K8s Secrets
Configuration	Volume mounts	ConfigMaps
Service discovery	DNS by service name	K8s Services
Ingress/routing	Port mapping	nginx-ingress
Resource limits	Limited support	Full K8s resources
Init containers	No	Yes
Readiness probes	No	Yes

Production Environment

Prerequisites

• Access to production Kubernetes cluster
• kubectl configured with production context
• Production secrets prepared

Setting Up Secrets (Prod)

# Copy example secrets
cd deploy/k8s/prod/secrets/
cp ai-secrets.yaml.example ai-secrets.yaml
cp postgres-secret.yaml.example postgres-secret.yaml
# ... etc

# Edit with production values
vim ai-secrets.yaml

# Apply to cluster (will prompt for confirmation)
bin/secret-update prod

# Or update specific secret
bin/secret-update prod ai-secrets

Updating Configs (Prod)

# Edit production configs if needed
vim deploy/configmaps/relay-config.yaml

# Apply changes (will prompt for confirmation)
bin/config-update prod

Deploying to Production

# Verify kubectl context
kubectl config current-context

# Apply manifests
kubectl apply -k deploy/k8s/prod/

# Check rollout status
kubectl rollout status statefulset/ai-backend
kubectl rollout status deployment/ai-web

# View status
kubectl get pods,svc,ingress

Kustomize Overlays

Dev Overlay

• imagePullPolicy: Never - Uses locally built images
• Infrastructure services - Kafka, Postgres, MinIO, Flink, Relay, Ingestor
• Local ingress - dexorder.local (requires /etc/hosts entry)
• No gVisor - RuntimeClass removed (not available in minikube)
• Single replicas - Minimal resource usage

Prod Overlay

• imagePullPolicy: Always - Pulls from registry
• Multiple replicas - HA configuration
• Resource limits - CPU/memory constraints
• gVisor - Security sandbox via RuntimeClass
• Production ingress - dexorder.ai with TLS

Infrastructure Services (Dev Only)

These services are included in the dev environment but are expected to be managed separately in production:

• Kafka - KRaft mode (no Zookeeper), single broker
• PostgreSQL - Iceberg catalog metadata
• MinIO - S3-compatible object storage
• Iceberg REST Catalog - Table metadata
• Flink - JobManager + TaskManager
• Relay - ZMQ message router
• Ingestor - CCXT data fetcher

In production, you would typically use:

• Managed Kafka (Confluent Cloud, MSK, etc.)
• Managed PostgreSQL (RDS, Cloud SQL, etc.)
• Object storage (S3, GCS, Azure Blob)
• Flink Kubernetes Operator or managed Flink

Troubleshooting

Minikube not starting

minikube delete
minikube start --cpus=6 --memory=12g --driver=docker

Images not found

Make sure you're using minikube's docker daemon:

eval $(minikube docker-env)
bin/dev rebuild

Ingress not working

Start minikube tunnel in another terminal:

bin/dev tunnel

Secrets not found

Create secrets from examples:

cd deploy/k8s/dev/secrets/
cp *.example *.yaml
vim ai-secrets.yaml  # Edit with actual values
bin/secret-update dev

Pods not starting

Check events and logs:

kubectl get events --sort-by=.metadata.creationTimestamp
kubectl describe pod <pod-name>
kubectl logs <pod-name>

CI/CD Integration

For automated deployments, you can use:

# Build and push images
docker build -t registry.example.com/dexorder/ai-web:$TAG .
docker push registry.example.com/dexorder/ai-web:$TAG

# Update kustomization with new tag
cd deploy/k8s/prod
kustomize edit set image dexorder/ai-web=registry.example.com/dexorder/ai-web:$TAG

# Deploy
kubectl apply -k deploy/k8s/prod/