Message Routing
Kafka provides routing capabilities through Kafka Connect and Kafka Streams, including content-based routing, message transformation, and message enrichment.
Memphis message routing is similar to the implementation of RabbitMQ using routing keys, wildcards, content-based routing, and more. Similar to RabbitMQ, it is also embedded within the broker and does not require external libraries or tools.
Log Compaction
Compaction has been created to support a long-term, potentially infinite record store based on specific keys.
Kafka supports native topic compaction, which runs on all brokers. This runs automatically for compacted topics, condensing the log down to the latest version of messages sharing the same key.
At the moment, Memphis does not support compaction, but it will in the future.
Message Replay
The ability to re-consume committed messages.
Kafka does support replay by seeking specific offsets as the consumers have control over resetting the offset.
Memphis does not support replay yet but will in the near future (2023).
Stream Enrichment
Kafka, with its Kafka Streams library, allows developers to implement elastic and scalable client applications that can leverage essential stream processing features such as tables, joins, and aggregations of several topics and export to multiple sources via Kafka connect.
Memphis provides a similar behavior and more. Embedded inside the broker, Memphis users can create serverless-type functions or complete containerized applications that aggregate several stations and streams, decorate and enrich messages from different sources, write complex functions that cannot be achieved via SQL, and manipulate the schema. In addition, Memphis embedded connectors frameworks will help to push the results directly to a defined sink.
Pull Retry Mechanism
In case of a failure or lack of ability to acknowledge consumed messages, there should be a retry mechanism that will retry to pull the same offset or batch of messages.
In Kafka, it is the client’s responsibility to implement one. Some key factors must be considered to implement such a mechanism, like blocking vs. non-blocking, offset tracking, idempotency, and more.
In Memphis, the retry mechanism is built-in and turned on by default within the SDK and broker. During consumer creation, the parameter maxMsgDeliveries will determine the number of retries the station will deliver a message if an acknowledgment does not arrive till maxAckTimeMs . The broker itself records the offsets given and will expose only the unacknowledged ones to the retry request.
