Kafka Deployment for K8s Cluster

Install Strimzi Operator

# Add the Strimzi Helm repository (if using Helm 3.6+)
helm install strimzi-kafka-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator \
  -f values.yaml \
  --namespace kafka --create-namespace

# Wait for operator to be ready
kubectl wait --for=condition=ready pod -l name=strimzi-cluster-operator -n kafka --timeout=300s

Deploy Kafka Cluster

# Apply the metrics ConfigMap first
kubectl apply -f kafka-metrics-config.yaml

# Apply the Kafka cluster manifest
kubectl apply -f kafka-cluster.yaml

# Wait for Kafka to be ready (this may take a few minutes)
kubectl wait kafka/trading-cluster --for=condition=Ready --timeout=300s -n default

Verify Installation

# Check Kafka cluster status
kubectl get kafka -n default

# Check all pods
kubectl get pods -n default | grep trading-cluster

# Check Kafka cluster details
kubectl describe kafka trading-cluster -n default

Connect to Kafka

Internal connection string (from within cluster):

Plaintext: trading-cluster-kafka-bootstrap.default.svc:9092
TLS: trading-cluster-kafka-bootstrap.default.svc:9093

Prometheus Metrics

Kafka and ZooKeeper expose metrics on port 9404. Prometheus will automatically discover and scrape these metrics via pod annotations:

prometheus.io/scrape: "true"
prometheus.io/port: "9404"
prometheus.io/path: "/metrics"

To verify metrics are being exported:

kubectl exec -it trading-cluster-kafka-0 -n default -- curl localhost:9404/metrics

Create a Test Topic

kubectl run kafka-producer -ti --image=quay.io/strimzi/kafka:0.43.0-kafka-3.7.0 --rm=true --restart=Never -- \
  bin/kafka-topics.sh --create --topic test-topic --bootstrap-server trading-cluster-kafka-bootstrap:9092 --partitions 3 --replication-factor 3

Uninstall

# Delete Kafka cluster
kubectl delete kafka trading-cluster -n default

# Delete operator
helm uninstall strimzi-kafka-operator -n kafka

2.0 KiB Raw Permalink Blame History