kafka1.1.0
本机部署三个实例
一个实例需要2个端口  jmx端口  服务端口

jmx端口 在 kafka-server-start.sh 里配置





kafka1
kafka2
kafka3



id
0
1
2



端口
9092
9093
9094



JMX端口
9995
9996
9997






zookeeper.connect=localhost:2182,localhost:2183/kafka
zkroot  kafka


启动zk集群
请参考  zookeeper集群搭建 

依次启动kafka broker 
nohup ./bin/kafka-server-start.sh  config/server.properties &

发送消息
https://gitee.com/dyyx/kafkademo/blob/master/src/main/java/dyyx/ProducerTest.java
自动创建topic 

消费消息
https://gitee.com/dyyx/kafkademo/blob/master/src/main/java/dyyx/ConsumerTest.java

查看 offset
./bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group kafkademo_test_group
 

停止 broker
kafka-server-stop.sh

可以命令行工具 发送消费消息
 kafka命令行发送接收消息 



数据存储
kafka0/kafka-logs/kafka_demo_test_topic-1

00000000000000000000.index
00000000000000000000.log
00000000000000000000.timeindex
leader-epoch-checkpoint



../../bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log

Starting offset: 0
baseOffset: 0 lastOffset: 0 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 3 isTransactional: false position: 0 CreateTime: 1597727377253 isvalid: true size: 128 magic: 2 compresscodec: NONE crc: 2324548880
baseOffset: 1 lastOffset: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 3 isTransactional: false position: 128 CreateTime: 1597727377275 isvalid: true size: 128 magic: 2 compresscodec: NONE crc: 3632234908

size: 128  compresscodec: NONE 


查看主题offset
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --time -1 --topic kafka_demo_test_topic
--time   timestamp of the offsets before that   (earliest)  (default: -1)  




查看数据文件数据
./kafka-run-class.sh kafka.tools.DumpLogSegments  --files /Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs/kafka_demo_test_topic-1/00000000000000000023.log   | head -n 10
./kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration  --files /Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs/kafka_demo_test_topic-1/00000000000000000023.log   | head -n 10
./kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs/kafka_demo_test_topic-1/00000000000000000023.log   | head -n 10

--print-data-log  显示消息数据

offset: 23 position: 0 CreateTime: 1598148413924 isvalid: true keysize: 16 valuesize: 45 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 1598148413924_14 payload: 1598148413924_14,Sun Aug 23 10:06:53 CST 2020
offset: 24 position: 130 CreateTime: 1598148413961 isvalid: true keysize: 16 valuesize: 45 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 1598148413960_18 payload: 1598148413960_18,Sun Aug 23 10:06:53 CST 2020
offset: 25 position: 260 CreateTime: 1598148414002 isvalid: true keysize: 16 valuesize: 45 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 1598148414001_23 payload: 1598148414001_23,Sun Aug 23 10:06:54 CST 2020


性能压测

./kafka-producer-perf-test.sh --topic kafka_demo_test_topic --num-records 1000000 --throughput -1 --record-size 128 --producer-props bootstrap.servers=localhost:9092 acks=-1 linger.ms=2000 compression.type=lz4

./kafka-consumer-perf-test.sh --broker-list localhost:9092 --messages 1000000 --topic producer






配置文件实例
kafka-cluster/kafka0/config/server.properties 

broker.id=0
listeners=PLAINTEXT://localhost:9092
num.network.threads=5
num.io.threads=10
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs
num.partitions=6
default.replication.factor=2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=3
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2182,localhost:2183/kafka
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0


常见异常
Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

UnknownTopicOrPartitionException
创建 topic  往 zk写入元数据，需要一定的时间 ， 可稍后重试

This topic/partition doesn't exist. 
This exception is used in contexts where a topic doesn't seem to exist based on possibly stale metadata. 
This exception is retriable because the topic or partition might subsequently be created.

producer 可捕获此异常，并重试



完整例子代码
https://gitee.com/dyyx/kafkademo


 kafka命令行发送接收消息 
 zookeeper集群搭建 
 kafka在zookeeper中的元数据 
 kafka1.1.0无法消费问题