Apache Kafka is a distributed streaming platform that allows you to build real-time data pipelines and streaming applications. In this blog, we will walk through the step-by-step process of setting up a Kafka cluster with three Zookeepers and three brokers, all deployed on EC2 instances. This setup ensures fault tolerance, scalability, and high availability for your Kafka infrastructure. You can deploy the cluster on your local computer or other cloud provider, I chose AWS since it’s easy to setup.
Before we get started, what is expected from you is to have a basic knowledge of AWS EC2 and AWS in general, in addition to having a basic knowledge of Apache Kafka concepts.
For best practices and for production deployment, it’s always recommended to locate each zookeeper and broker in a separate server for high availability and fault tolerance. However, in this demo, I deploy the zookeeper and broker on one EC2 for cost-saving purposes.
So, without further ado, let’s dig in!
Step 1: Setup EC2 Instances
1- To start with the setup, please make sure to have an AWS account- visit-this-link.
After login to the console, make sure to select the nearest region to you ( in this demo, I select Bargain region), then make sure you have already three subnets created and write down in note the IPs for later use.
2- Lunch EC2 instances with Ubuntu image t3.medium ( 4 GiB of memory).
We are targeting to lunch three EC2 instances for Quorum setup, however, to save time, let’s create the first server, prepare the environment, install Zookeeper and other dependencies, then take an AMI image from the EC2 and lunch the other two servers using the created AMI ( in step 2).
3- Open the ports (22, 2181, 2888, 3888 ) for the zookeepers and Kafka in the security group.