kafka消息投递语义
所属分类 kafka
浏览量 1391
At most once
Messages may be lost but are never redelivered
消息可能丢失但不会重复投递
At least once
Messages are never lost but may be redelivered
消息不会丢失但可能重复投递
Exactly once
this is what people actually want, each message is delivered once and only once
消息只投递一次
从0.11.0.0版本以后,Kafka生产者支持幂等投递选项,以保证即使消息被重新发送,日志中也不会有重复的条目。
为了实现这一点,broker给没用个生成者指定一个ID并且每条消息指定一个序列号。
站在消费者的角度,先保存位置后处理消息就是“最多一次”;先处理消息后保存位置就是“最少一次”;
至于“精确一次”,可以使用事务生产者来实现,即在同一个事务中接收并处理消息,将位置(offset)保存到另一个topic中。
The consumer's position is stored as a message in a topic,
so we can write the offset to Kafka in the same transaction as the output topics receiving the processed data.
If the transaction is aborted, the consumer's position will revert to its old value
and the produced data on the output topics will not be visible to other consumers, depending on their "isolation level."
In the default "read_uncommitted" isolation level, all messages are visible to consumers even if they were part of an aborted transaction,
but in "read_committed," the consumer will only return messages from transactions which were committed
(and any messages which were not part of a transaction).
上一篇
下一篇
CountDownLatch 与 CyclicBarrier
让自己更优秀的16条法则
kafka高可用机制简介
redis使用注意点
集群session处理
分布式数据库主键生成方案