How to Deploy a Zookeeper and Kafka cluster in Google Cloud Platform

Apache Kafka

One of the great advantages of Google Cloud Platform is how easy and fast it is to run experiments. For example, you can easily spin up a Zookeper and Kafka cluster in a matter of minutes with very little configuration.

For reference, you can find the sample properties files for each server in this repository: Cluster Config Examples

Note: The clusters below are not suitable for massive production use but good enough to run some tests

Zookeeper

Apache Zookeeper

Lets spin up 3 machines that will form our Zookeeper cluster. The command below can be run from the Google Cloud Shell or from your PC using the Google Cloud Platform CLI. All that you need to substitute are the project and instance name (e.g. zook-1)

Zookeeper instance variables

gcloud compute --project "[YOUR-PROJECT]" instances create "[INSTANCE-NAME]" --zone "europe-west1-c" --machine-type "n1-standard-1" --subnet "default" --maintenance-policy "MIGRATE" --service-account "[email protected]" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --tags "http-server" --image "centos-7-v20170523" --image-project "centos-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "[INSTANCE-NAME]"

Once the VM is running, we can ssh to it and run the following command to install some dependencies and download Kafka.

# Installing dependencies
sudo su
yum install -y wget nano java

# Automatically creating the Zookeeper myid file
mkdir /tmp/zookeeper
touch /tmp/zookeeper/myid
echo "ZOOKEEPER-ID" > zookeeper/myid

# Downloading and extracting Kafka
wget http://mirror.ox.ac.uk/sites/rsync.apache.org/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz
tar -xvzf kafka_2.12-0.10.2.1.tgz
cd kafka_2.12-0.10.2.1

Modify the zookeeper properties to include all the instance details.

nano config/zookeeper.properties
# In milliseconds
tickTime=2000

# In ticks
initLimit=10
syncLimit=5
maxClientCnxns=30

# All Zookeeper servers need to be aware of other Zookeepers part of the cluster
server.1=zook-1:2888:3888
server.2=zook-2:2888:3888
server.3=zook-3:2888:3888

Once we’ve update the settings of all Zookeeper server properties, we can start them

bin/zookeeper-server-start.sh config/zookeeper.properties

Kafka

Lets spin up 3 machines that will form our Kafka cluster. All that you need to substitute are the project name, instance name (e.g. kafka-1), host name and broker Id.

Kafka instance variables

gcloud compute --project "[YOUR-PROJECT]" instances create "[INSTANCE-NAME]" --zone "europe-west1-c" --machine-type "n1-standard-1" --subnet "default" --maintenance-policy "MIGRATE" --service-account "[email protected]" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --tags "http-server" --image "centos-7-v20170523" --image-project "centos-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "[INSTANCE-NAME]"

Lets ssh to the VM and run the following command to install some dependencies and download Kafka.

# Installing dependencies
sudo su
yum install -y wget nano java

# Downloading and extracting Kafka
wget http://mirror.ox.ac.uk/sites/rsync.apache.org/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz
tar -xvzf kafka_2.12-0.10.2.1.tgz
cd kafka_2.12-0.10.2.1

Now, update the Kafka server properties with the property values below:

nano config/server.properties
# Broker id needs to be unique for each Kafka server
broker.id=[BROKER-ID]

# This will be the Zookeeper servers created before
zookeeper.connect=zook-1:2181,zook-2:2181,zook-3:2181

# Examples of hostnames - kafka-1, kafka-2, kafka-3
host.name=[HOSTNAME]

Once we’ve update the settings of all Kafka server properties, we can start them

bin/kafka-server-start.sh config/server.properties
Written on June 6, 2017