deploy/k8s/
├── base/                    # Base manifests (shared)
│   ├── backend.yaml
│   ├── web.yaml
│   ├── ingress.yaml
│   ├── init.yaml
│   └── kustomization.yaml
├── dev/                     # Dev overlay (minikube)
│   ├── infrastructure.yaml  # Kafka, Postgres, MinIO, Flink, Relay, Ingestor
│   ├── ingress-dev.yaml     # Dev ingress (dexorder.local)
│   ├── patches.yaml         # Dev-specific patches
│   ├── kustomization.yaml
│   └── secrets/
│       ├── *.yaml           # Actual secrets (gitignored)
│       └── *.yaml.example   # Templates
├── prod/                    # Production overlay
│   ├── patches.yaml         # Prod patches (replicas, resources, gVisor)
│   ├── kustomization.yaml
│   └── secrets/
│       ├── *.yaml           # Actual secrets (gitignored)
│       └── *.yaml.example   # Templates
└── configmaps/              # Shared ConfigMaps
    ├── relay-config.yaml
    ├── ingestor-config.yaml
    └── flink-config.yaml

Dev Environment (Minikube)

Prerequisites

minikube
kubectl
Docker

Quick Start

# Start everything
bin/dev start

# Access the application
# Web UI: http://dexorder.local/cryptochimp/
# Backend: ws://dexorder.local/ws

# In another terminal, start tunnel for ingress
bin/dev tunnel

Managing Dev Environment

# Rebuild images after code changes
bin/dev rebuild

# Redeploy services
bin/dev deploy

# Full restart (rebuild + redeploy)
bin/dev restart

# View status
bin/dev status

# View logs
bin/dev logs relay
bin/dev logs ingestor
bin/dev logs flink-jobmanager

# Open shell in pod
bin/dev shell relay

# Clean everything
bin/dev clean

# Stop minikube
bin/dev stop

Setting Up Secrets (Dev)

# Copy example secrets
cd deploy/k8s/dev/secrets/
cp ai-secrets.yaml.example ai-secrets.yaml
cp postgres-secret.yaml.example postgres-secret.yaml
cp minio-secret.yaml.example minio-secret.yaml
cp ingestor-secrets.yaml.example ingestor-secrets.yaml

# Edit with actual values
vim ai-secrets.yaml  # Add your Anthropic API key

# Apply to cluster
bin/secret-update dev

# Or update a specific secret
bin/secret-update dev ai-secrets

Updating Configs (Dev)

# Edit config files
vim deploy/configmaps/relay-config.yaml

# Apply changes
bin/config-update dev

# Or update specific config
bin/config-update dev relay-config

Dev vs Docker Compose

The minikube dev environment mirrors production more closely than docker-compose:

Feature	docker-compose	minikube
Environment parity	❌ Different from prod	✅ Same as prod
Secrets management	`.env` files	K8s Secrets
Configuration	Volume mounts	ConfigMaps
Service discovery	DNS by service name	K8s Services
Ingress/routing	Port mapping	nginx-ingress
Resource limits	Limited support	Full K8s resources
Init containers	No	Yes
Readiness probes	No	Yes

Production Environment

Prerequisites

Access to production Kubernetes cluster
kubectl configured with production context
Production secrets prepared

Setting Up Secrets (Prod)

# Copy example secrets
cd deploy/k8s/prod/secrets/
cp ai-secrets.yaml.example ai-secrets.yaml
cp postgres-secret.yaml.example postgres-secret.yaml
# ... etc

# Edit with production values
vim ai-secrets.yaml

# Apply to cluster (will prompt for confirmation)
bin/secret-update prod

# Or update specific secret
bin/secret-update prod ai-secrets

Updating Configs (Prod)

# Edit production configs if needed
vim deploy/configmaps/relay-config.yaml

# Apply changes (will prompt for confirmation)
bin/config-update prod

Deploying to Production

# Verify kubectl context
kubectl config current-context

# Apply manifests
kubectl apply -k deploy/k8s/prod/

# Check rollout status
kubectl rollout status statefulset/ai-backend
kubectl rollout status deployment/ai-web

# View status
kubectl get pods,svc,ingress

Kustomize Overlays

Dev Overlay

imagePullPolicy: Never - Uses locally built images
Infrastructure services - Kafka, Postgres, MinIO, Flink, Relay, Ingestor
Local ingress - dexorder.local (requires /etc/hosts entry)
No gVisor - RuntimeClass removed (not available in minikube)
Single replicas - Minimal resource usage

Prod Overlay

imagePullPolicy: Always - Pulls from registry
Multiple replicas - HA configuration
Resource limits - CPU/memory constraints
gVisor - Security sandbox via RuntimeClass
Production ingress - dexorder.ai with TLS

Infrastructure Services (Dev Only)

These services are included in the dev environment but are expected to be managed separately in production:

Kafka - KRaft mode (no Zookeeper), single broker
PostgreSQL - Iceberg catalog metadata
MinIO - S3-compatible object storage
Iceberg REST Catalog - Table metadata
Flink - JobManager + TaskManager
Relay - ZMQ message router
Ingestor - CCXT data fetcher

In production, you would typically use:

Managed Kafka (Confluent Cloud, MSK, etc.)
Managed PostgreSQL (RDS, Cloud SQL, etc.)
Object storage (S3, GCS, Azure Blob)
Flink Kubernetes Operator or managed Flink

Troubleshooting

Minikube not starting

minikube delete
minikube start --cpus=6 --memory=12g --driver=docker

Images not found

Make sure you're using minikube's docker daemon:

eval $(minikube docker-env)
bin/dev rebuild

Ingress not working

Start minikube tunnel in another terminal:

bin/dev tunnel

Secrets not found

Create secrets from examples:

cd deploy/k8s/dev/secrets/
cp *.example *.yaml
vim ai-secrets.yaml  # Edit with actual values
bin/secret-update dev

Pods not starting

Check events and logs:

kubectl get events --sort-by=.metadata.creationTimestamp
kubectl describe pod <pod-name>
kubectl logs <pod-name>

CI/CD Integration

For automated deployments, you can use:

# Build and push images
docker build -t registry.example.com/dexorder/ai-web:$TAG .
docker push registry.example.com/dexorder/ai-web:$TAG

# Update kustomization with new tag
cd deploy/k8s/prod
kustomize edit set image dexorder/ai-web=registry.example.com/dexorder/ai-web:$TAG

# Deploy
kubectl apply -k deploy/k8s/prod/