KAFKA Cluster Sizing Tool (By Topics)
Originally published: 25/07/2020 12:47
Publication number: ELQ-91340-1
View all versions & Certificate
certified

KAFKA Cluster Sizing Tool (By Topics)

Free downloadable version of KAFKA Cluster sizing tool (by topics) in Excel format

Description
Apache Kafka is a distributed event streaming / messaging platform which is capable of handling trillions of events a day. The basic architecture is of Apache Kafka is organised around topics, producers, consumers, and brokers.
All the streaming data is organised into topics. Topics are the main functional building blocks of a Kafka Cluster.

This FREE & EASY TO USE Kafka Cluster Sizing calculator helps you with estimation of your Kafka Cluster size based on key information filled by user for each of the topics planned in an Apache Kafka Cluster.

FILLING OUT THE TOOL:
====================
1) Input number of planned topics in your cluster
2) Input Default Retention Days
3) Input Default Replication Factor
4) Default # of Partitions
5) Num of Brokers Planned
6) Production Throughput / Partition (Mb/sec)
7) For each planned topic fill out following details:
Topic Group:
===========
This could be a project / department or functional unit. This field can be used for reporting purpose.

Topic Name:
===========
Name of the topic

Message Have Identifiers?:
=====================
Whether the message will have a message ID. This information is not used currently, it can be used for estimating number of partitions and message size in future versions of the tool

Maximum Message Size (bytes):
===========================
Maximum Size of the message sent to this topic in bytes

Messages Per Second (During Peak Hrs.):
==================================
Number of messages sent during peak hours to this topic

Avg. Num. of Message Per Second:
=============================
Number of messages sent during non peak hours

Peak Hrs Per Day:
===============
Estimate number of peak hours per days

Retention Days:
=============
Number of days messages for this topic will be retained in the cluster, If no provided default retention days will be used

Replication Factor:
================
Replication factor the the topic, if not provided default Replication factor will be used

# Of Partitions (Planned):
====================
Number of partition planned for this topic, if not provided default # of partition will be used

CALCULATIONS:
==============
This calculator will calculate the following metrics:
Total Storage
Incoming byte rate
Network Throughput
Total Number of partition

This is FREE TOOL, Future version of this tool will have many more metrics for a KAFKA Cluster.

Please rate this tool and donate small amount to support future development on such useful tools.

This Best Practice includes
1 MS Excel file

Piyush V. Agarwal offers you this Best Practice for free!

download for free

Add to bookmarks

Discuss


4.8 / 5 (6 votes)

please wait...