Apache Kafka Cluster Installation Guide

This document will provide you instruction Apache Kafka Cluster and how to set up Zookeeper.

Download the latest version of Apache Kafka and Extract Apache Kafka from their website here.

At the time of writing the current stable version is 1.0.8. Which I am using on most of the installations.

Prerequisite

  1. Download JRE from the site. http://www.oracle.com/technetwork/java/javase/downloads/index.html
  2. Install JRE. Install it into the server using an alternatives command and also set the environment variables.

2.1 To set path and JAVA_HOME variables, add the following commands to ~/.bashrc file.

              export JAVA_HOME =/usr/jdk/jdk-1.8.0_60
              export PATH=$PATH:$JAVA_HOME/bin

2.2 Now apply all the changes into current running system.

            $ source ~/.bashrc

3. Use the following command to change Java Alternatives.

update-alternatives --install /usr/bin/java java /opt/jdk/jdk1.8.0_60/bin/java 100

/opt/jdk/jdk1.8.0_60/bin/java this is the location where you extracted the java files orignally.

Now Boom Done with the java. Next step is Kafka setup. 
  1. Install and Configure Apache Zookeeper.

Setting up Apache Kafka

  1. Now Go to the config folder in Apache Kafka which extracted before and edit “server.properties” using any text editor.
  2. Update the following line.

broker.id=1

If you have multiple servers in kafka cluster then setup broker.id value accordingly.

listeners=PLAINTEXT://x.x.x.x:9092

Mention listener IP, it will help you to access kafka from other servers and consumer and publishers can connect with your cluster.

Find log.dirs and repelace after “=/tmp/kafka-logs” to “=/Home/username/kafka_2.10–0.10.1.1/kafka-logs” (change your version number).

Leave other setting as is. If your Apache Zookeeper on different server then change the “zookeeper.connect” property.

You can use this for zookeeper.connect = 192.168.10.10:2181,192.168.10.11:2181, 192.168.10.12:2181

This is showing the zookeeper cluster of three servers.

By default Apache Kafka will run on port 9092 and Apache Zookeeper will run on port 2181.

Setup Zookeeper cluster for Apache Kafka cluster

If you are using zookeeper came from the kafka package then go to zookeeper.properties file in config folder and add below lines to all servers in the kafka and zookeeper cluster.

dataDir is the location which you want to make it as default data directory of zookeeper.

dataDir=/home/username/kafka_store/zookeeper

initLimit=5

initLimit=5

syncLimit=2

server.1=192.168.10.10:2888:4888

server.2=192.168.10.11:2888:4888

server.3=192.168.10.12:2888:4888

tickTime=2000

Running Apache Kafka

After doing the above steps go to your Apache Kafka directory and run following command.

./bin/kafka-server-start.sh config/server.properties

Also start the zookeeper on all three servers which you have setup.

./bin/zookeeper-server-start.sh config/zookeeper.properties

You are good to go, your Apache Kafka and zookeeper is up and running on port 9092 and 2181.

Congratulations now your Kafka is setup.

If you want to change the RAM/memory allocation to APache Kafka then go to the start scripts of zookeeper and kafka in bin/ directory and update the Xms -Xmx values.

Now you have completed kafka apache installation (Kafka).

Leave a Reply.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: