Setting Up Kafka on Minikube K8s Using Strimzi

Before starting this demo, if you want to set up Kafka on AWS EC2, please check my previous article.

Setup a Kafka Cluster on AWS EC2

So I know what you are thinking right now, and it might be this why you are here! Setting up Kafka on Kubernetes might seem like an overwhelming task, but fear not! In this step-by-step guide, I’ll walk you through the process of setting up the necessary prerequisites to deploying Kafka on your Kubernetes cluster.

Prerequisites

Before diving into the Kafka setup, ensure you have the following prerequisites installed:

1. Docker Desktop: If you’re running Kubernetes locally, Docker Desktop provides a convenient way to set up and manage Kubernetes clusters on your machine. Or, if you want you can use another VM type that minikube supports like Hyber-V or VirtualBox.
2. Minikube: Minikube is a tool that allows you to run Kubernetes locally. It’s ideal for development and testing purposes. — NOT FOR PROD purpose 🤝
3. kubectl: its CLI tool to interact with k8s.

minikube start and check status

Setting up a Kafka cluster on K8s can be approached in various ways, each with its own set of advantages and considerations such as Helm charts or using some cloud providers (Kafka-as-a-service Platform) with managed Kubernetes services such as AWS ECS. However, one of the easiest ways to use custom operators like Strimzi. read more about Strimzi

Step 1: Deploy Strimzi Operator

Before deploying the Strimzi cluster operator, create a namespace called kafka

> kubectl create namespace kafka

# default namespace is defualt 
# you can check list of namespace on your cluster using below comand 

> kubectl get ns

# you can switch the ns to default as kafka 
> kubectl config set-context --current --namespace=kafka

Next is to apply the strimzi operator on minikube

> kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

#here we pass the ns name kafka 

This command will get the yaml file for the operator and create the strimzi operator deployment.

To verify the pod status

> kubectl get pod -n kafka

deployment and pod status

In case you face some challenges, you can check the deployment logs

> kubectl logs deployment/strimzi-cluster-operator -n kafka -f

Step 2: Deploy Kafka Cluster using Strimzi

Now that Strimzi is installed, you can deploy a Kafka cluster. Strimzi simplifies this process by providing Custom Resource Definitions (CRDs) for Kafka resources.

Create a Kafka cluster YAML file (e.g., kafka-cluster.yaml) with the desired configuration. Below is a minimal example:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.7.0
    replicas: 1 # you can increase it for more broker replica 
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.7"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 1 # you can increase it for more ZK replica
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Apply the configuration to your Kubernetes cluster:

> kubectl apply -f kafka-cluster.yaml -n kafka

apply Kafka-cluster

This will create a Kafka cluster named my-cluster with one Kafka broker replica and one Zookeeper replica.

broker and zk pods status

To verify that the Kafka cluster is successfully deployed, you can check the status of the Kafka resources:

> kubectl get kafka -n kafka

kafka cluster status

Congratulations folks! you have successfully set up Kafka on your k8s cluster using Strimzi and Minikube.

The next crucial step is to start utilizing Kafka’s capabilities to create topics, configure users with access control lists (ACLs), and enable authentication mechanisms for external clients ( PLAINTEXT, SASL_SSL, or SSL ).

I’ll cover this in the next article ✌️

Remember, behind every great software engineer is a cup of coffee ☕️

Automating Node.js Deployment with GitLab CI/CD, Docker, and OpenShift

Docker Crash Course Series — #2- Dockerfile & Container Creating?

Docker Crash Course Series — Basics to Intermediate #1- What is Docker?