75 lines
2.0 KiB
Markdown
75 lines
2.0 KiB
Markdown
# Kafka Deployment for K8s Cluster
|
|
|
|
## Install Strimzi Operator
|
|
|
|
```bash
|
|
# Add the Strimzi Helm repository (if using Helm 3.6+)
|
|
helm install strimzi-kafka-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator \
|
|
-f values.yaml \
|
|
--namespace kafka --create-namespace
|
|
|
|
# Wait for operator to be ready
|
|
kubectl wait --for=condition=ready pod -l name=strimzi-cluster-operator -n kafka --timeout=300s
|
|
```
|
|
|
|
## Deploy Kafka Cluster
|
|
|
|
```bash
|
|
# Apply the metrics ConfigMap first
|
|
kubectl apply -f kafka-metrics-config.yaml
|
|
|
|
# Apply the Kafka cluster manifest
|
|
kubectl apply -f kafka-cluster.yaml
|
|
|
|
# Wait for Kafka to be ready (this may take a few minutes)
|
|
kubectl wait kafka/trading-cluster --for=condition=Ready --timeout=300s -n default
|
|
```
|
|
|
|
## Verify Installation
|
|
|
|
```bash
|
|
# Check Kafka cluster status
|
|
kubectl get kafka -n default
|
|
|
|
# Check all pods
|
|
kubectl get pods -n default | grep trading-cluster
|
|
|
|
# Check Kafka cluster details
|
|
kubectl describe kafka trading-cluster -n default
|
|
```
|
|
|
|
## Connect to Kafka
|
|
|
|
Internal connection string (from within cluster):
|
|
- **Plaintext**: `trading-cluster-kafka-bootstrap.default.svc:9092`
|
|
- **TLS**: `trading-cluster-kafka-bootstrap.default.svc:9093`
|
|
|
|
## Prometheus Metrics
|
|
|
|
Kafka and ZooKeeper expose metrics on port 9404. Prometheus will automatically discover and scrape these metrics via pod annotations:
|
|
- `prometheus.io/scrape: "true"`
|
|
- `prometheus.io/port: "9404"`
|
|
- `prometheus.io/path: "/metrics"`
|
|
|
|
To verify metrics are being exported:
|
|
```bash
|
|
kubectl exec -it trading-cluster-kafka-0 -n default -- curl localhost:9404/metrics
|
|
```
|
|
|
|
## Create a Test Topic
|
|
|
|
```bash
|
|
kubectl run kafka-producer -ti --image=quay.io/strimzi/kafka:0.43.0-kafka-3.7.0 --rm=true --restart=Never -- \
|
|
bin/kafka-topics.sh --create --topic test-topic --bootstrap-server trading-cluster-kafka-bootstrap:9092 --partitions 3 --replication-factor 3
|
|
```
|
|
|
|
## Uninstall
|
|
|
|
```bash
|
|
# Delete Kafka cluster
|
|
kubectl delete kafka trading-cluster -n default
|
|
|
|
# Delete operator
|
|
helm uninstall strimzi-kafka-operator -n kafka
|
|
```
|