Originally published: 25/07/2020 12:47
Publication number: ELQ-91340-1
View all versions & Certificate
Publication number: ELQ-91340-1
View all versions & Certificate
KAFKA Cluster Sizing Tool (By Topics)
Free downloadable version of KAFKA Cluster sizing tool (by topics) in Excel format
by Piyush V. Agarwal
Solution Architect - BI & DW | Media & Entertainment | Manufacturing | Finance | KAFKA | DRUID | Industry 4.0Follow
Solution Architect - BI & DW | Media & Entertainment | Manufacturing | Finance | KAFKA | DRUID | Industry 4.0Follow
Description
Apache Kafka is a distributed event streaming / messaging platform which is capable of handling trillions of events a day. The basic architecture is of Apache Kafka is organised around topics, producers, consumers, and brokers.
All the streaming data is organised into topics. Topics are the main functional building blocks of a Kafka Cluster.
This FREE & EASY TO USE Kafka Cluster Sizing calculator helps you with estimation of your Kafka Cluster size based on key information filled by user for each of the topics planned in an Apache Kafka Cluster.
FILLING OUT THE TOOL:
====================
1) Input number of planned topics in your cluster
2) Input Default Retention Days
3) Input Default Replication Factor
4) Default # of Partitions
5) Num of Brokers Planned
6) Production Throughput / Partition (Mb/sec)
7) For each planned topic fill out following details:
Topic Group:
===========
This could be a project / department or functional unit. This field can be used for reporting purpose.
Topic Name:
===========
Name of the topic
Message Have Identifiers?:
=====================
Whether the message will have a message ID. This information is not used currently, it can be used for estimating number of partitions and message size in future versions of the tool
Maximum Message Size (bytes):
===========================
Maximum Size of the message sent to this topic in bytes
Messages Per Second (During Peak Hrs.):
==================================
Number of messages sent during peak hours to this topic
Avg. Num. of Message Per Second:
=============================
Number of messages sent during non peak hours
Peak Hrs Per Day:
===============
Estimate number of peak hours per days
Retention Days:
=============
Number of days messages for this topic will be retained in the cluster, If no provided default retention days will be used
Replication Factor:
================
Replication factor the the topic, if not provided default Replication factor will be used
# Of Partitions (Planned):
====================
Number of partition planned for this topic, if not provided default # of partition will be used
CALCULATIONS:
==============
This calculator will calculate the following metrics:
Total Storage
Incoming byte rate
Network Throughput
Total Number of partition
This is FREE TOOL, Future version of this tool will have many more metrics for a KAFKA Cluster.
Please rate this tool and donate small amount to support future development on such useful tools.
Apache Kafka is a distributed event streaming / messaging platform which is capable of handling trillions of events a day. The basic architecture is of Apache Kafka is organised around topics, producers, consumers, and brokers.
All the streaming data is organised into topics. Topics are the main functional building blocks of a Kafka Cluster.
This FREE & EASY TO USE Kafka Cluster Sizing calculator helps you with estimation of your Kafka Cluster size based on key information filled by user for each of the topics planned in an Apache Kafka Cluster.
FILLING OUT THE TOOL:
====================
1) Input number of planned topics in your cluster
2) Input Default Retention Days
3) Input Default Replication Factor
4) Default # of Partitions
5) Num of Brokers Planned
6) Production Throughput / Partition (Mb/sec)
7) For each planned topic fill out following details:
Topic Group:
===========
This could be a project / department or functional unit. This field can be used for reporting purpose.
Topic Name:
===========
Name of the topic
Message Have Identifiers?:
=====================
Whether the message will have a message ID. This information is not used currently, it can be used for estimating number of partitions and message size in future versions of the tool
Maximum Message Size (bytes):
===========================
Maximum Size of the message sent to this topic in bytes
Messages Per Second (During Peak Hrs.):
==================================
Number of messages sent during peak hours to this topic
Avg. Num. of Message Per Second:
=============================
Number of messages sent during non peak hours
Peak Hrs Per Day:
===============
Estimate number of peak hours per days
Retention Days:
=============
Number of days messages for this topic will be retained in the cluster, If no provided default retention days will be used
Replication Factor:
================
Replication factor the the topic, if not provided default Replication factor will be used
# Of Partitions (Planned):
====================
Number of partition planned for this topic, if not provided default # of partition will be used
CALCULATIONS:
==============
This calculator will calculate the following metrics:
Total Storage
Incoming byte rate
Network Throughput
Total Number of partition
This is FREE TOOL, Future version of this tool will have many more metrics for a KAFKA Cluster.
Please rate this tool and donate small amount to support future development on such useful tools.
This Best Practice includes
1 MS Excel file