Setting Up Kafka on Minikube K8s Using Strimzi
Before starting this demo, if you want to set up Kafka on AWS EC2, please check my previous article.
So I know what you are thinking right now, and it might be this why you are here! Setting up Kafka on Kubernetes might seem like an overwhelming task, but fear not! In this step-by-step guide, I’ll walk you through the process of setting up the necessary prerequisites to deploying Kafka on your Kubernetes cluster.
Prerequisites
Before diving into the Kafka setup, ensure you have the following prerequisites installed:
- Docker Desktop: If you’re running Kubernetes locally, Docker Desktop provides a convenient way to set up and manage Kubernetes clusters on your machine. Or, if you want you can use another VM type that minikube supports like Hyber-V or VirtualBox.
- Minikube: Minikube is a tool that allows you to run Kubernetes locally. It’s ideal for development and testing purposes. — NOT FOR PROD purpose 🤝
- kubectl: its CLI tool to interact with k8s.
Setting up a Kafka cluster on K8s can be approached in various ways, each with its own set of advantages and considerations such as Helm charts or using some cloud providers (Kafka-as-a-service Platform) with managed Kubernetes services such as AWS ECS. However, one of the easiest ways to use custom operators like Strimzi. read more about Strimzi
Step 1: Deploy Strimzi Operator
Before deploying the Strimzi cluster operator, create a namespace called kafka
> kubectl create namespace kafka
# default namespace is defualt
# you can check list of namespace on your cluster using below comand
> kubectl get ns
# you can switch the ns to default as kafka
> kubectl config set-context --current --namespace=kafka
Next is to apply the strimzi operator on minikube
> kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
#here we pass the ns name kafka
This command will get the yaml file for the operator and create the strimzi operator deployment.
To verify the pod status
> kubectl get pod -n kafka
In case you face some challenges, you can check the deployment logs
> kubectl logs deployment/strimzi-cluster-operator -n kafka -f
Step 2: Deploy Kafka Cluster using Strimzi
Now that Strimzi is installed, you can deploy a Kafka cluster. Strimzi simplifies this process by providing Custom Resource Definitions (CRDs) for Kafka resources.
Create a Kafka cluster YAML file (e.g., kafka-cluster.yaml
) with the desired configuration. Below is a minimal example:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.7.0
replicas: 1 # you can increase it for more broker replica
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: "3.7"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 1 # you can increase it for more ZK replica
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
Apply the configuration to your Kubernetes cluster:
> kubectl apply -f kafka-cluster.yaml -n kafka
This will create a Kafka cluster named my-cluster
with one Kafka broker replica and one Zookeeper replica.
To verify that the Kafka cluster is successfully deployed, you can check the status of the Kafka resources:
> kubectl get kafka -n kafka
Congratulations folks! you have successfully set up Kafka on your k8s cluster using Strimzi and Minikube.
The next crucial step is to start utilizing Kafka’s capabilities to create topics, configure users with access control lists (ACLs), and enable authentication mechanisms for external clients ( PLAINTEXT, SASL_SSL, or SSL ).
I’ll cover this in the next article ✌️
Remember, behind every great software engineer is a cup of coffee ☕️