backend redesign

kafka/README.md

@@ -0,0 +1,74 @@
+# Kafka Deployment for K8s Cluster
+
+## Install Strimzi Operator
+
+```bash
+# Add the Strimzi Helm repository (if using Helm 3.6+)
+helm install strimzi-kafka-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator \
+  -f values.yaml \
+  --namespace kafka --create-namespace
+
+# Wait for operator to be ready
+kubectl wait --for=condition=ready pod -l name=strimzi-cluster-operator -n kafka --timeout=300s
+```
+
+## Deploy Kafka Cluster
+
+```bash
+# Apply the metrics ConfigMap first
+kubectl apply -f kafka-metrics-config.yaml
+
+# Apply the Kafka cluster manifest
+kubectl apply -f kafka-cluster.yaml
+
+# Wait for Kafka to be ready (this may take a few minutes)
+kubectl wait kafka/trading-cluster --for=condition=Ready --timeout=300s -n default
+```
+
+## Verify Installation
+
+```bash
+# Check Kafka cluster status
+kubectl get kafka -n default
+
+# Check all pods
+kubectl get pods -n default | grep trading-cluster
+
+# Check Kafka cluster details
+kubectl describe kafka trading-cluster -n default
+```
+
+## Connect to Kafka
+
+Internal connection string (from within cluster):
+- **Plaintext**: `trading-cluster-kafka-bootstrap.default.svc:9092`
+- **TLS**: `trading-cluster-kafka-bootstrap.default.svc:9093`
+
+## Prometheus Metrics
+
+Kafka and ZooKeeper expose metrics on port 9404. Prometheus will automatically discover and scrape these metrics via pod annotations:
+- `prometheus.io/scrape: "true"`
+- `prometheus.io/port: "9404"`
+- `prometheus.io/path: "/metrics"`
+
+To verify metrics are being exported:
+```bash
+kubectl exec -it trading-cluster-kafka-0 -n default -- curl localhost:9404/metrics
+```
+
+## Create a Test Topic
+
+```bash
+kubectl run kafka-producer -ti --image=quay.io/strimzi/kafka:0.43.0-kafka-3.7.0 --rm=true --restart=Never -- \
+  bin/kafka-topics.sh --create --topic test-topic --bootstrap-server trading-cluster-kafka-bootstrap:9092 --partitions 3 --replication-factor 3
+```
+
+## Uninstall
+
+```bash
+# Delete Kafka cluster
+kubectl delete kafka trading-cluster -n default
+
+# Delete operator
+helm uninstall strimzi-kafka-operator -n kafka
+```