Manasvi Gupta bio photo

Manasvi Gupta

Apache Kafka, Spark, Scala

Twitter LinkedIn Github

In this post, we are going to look at some key differences between Apache Kafka and Traditional message brokers (e.g JMS, ActiveMQ).

1. Design

Apache Kafka : client-centric, with the client taking over many of the functions of a traditional broker, such as fair distribution of related messages to consumers, in return for an extremely fast and scalable broker.

Traditional message brokers : broker-centric, where the broker is responsible for the distribution of messages, and clients only have to worry about sending and receiving messages.

2. Scalability

Apache Kafka : horizontally scalable by adding more partitions
Traditional message brokers : not possible to scale horizontally

3. Performance

Apache Kafka : does not slow down with addition of new consumers
Traditional message brokers : both queue and topic performance degrades as the number of consumers rise on a destination

4. Publish options

Apache Kafka : unifies both publish-subscribe and point-to-point messaging under a single destination type— topic

Traditonal message brokers : provides both publish-subscribe and point-to-point messaging for publishing messages

5. Journal

Apache Kafka : each topic has a journal and there can be multiple topics.
Traditional message brokers : messages from all queues are stored in the same journal

6. Message delivery failures

Apache Kafka : client is responsible for replaying failed messages
Traditional message brokers : broker is responsible for re-delivery of messages

7. Message structure

Apache Kafka : message is a key-value pair. Payload of the message is sent as the value. Key, on the other hand, is used primarily for partitioning purposes and should contain a business-specific key in order to place related messages on the same partition.

Traditional message brokers : message consists of metadata (headers and properties) and body (payload)

8. Message integrity

Apache Kafka : includes checksums to detect corruption of messages in storage and has a comprehensive set of security features

Traditonal message brokers : no such option provided out-of-the box

9. Message retention

Apache Kafka : messages are not deleted on the broker once consumed.

Traditional message brokers : broker would either delete a successfully processed message or re-deliver an unprocessed or failed one (assuming transactions were in play)

10. Replay of messages

Apache Kafka : since Kafka retains messages, it is possible for clients to retrieve any message, providing option for re-processing of all messages.

Traditional message brokers : not possible, since, broker does not re-deliver messages that have been processed successfully by client