首页  

kafka集群搭建     所属分类 kafka 浏览量 315
kafka1.1.0
本机部署三个实例
一个实例需要2个端口  jmx端口  服务端口

jmx端口 在 kafka-server-start.sh 里配置


kafka1 kafka2 kafka3
id 0 1 2
端口 9092 9093 9094
JMX端口 9995 9996 9997
zookeeper.connect=localhost:2182,localhost:2183/kafka zkroot kafka 启动zk集群 请参考 zookeeper集群搭建 依次启动kafka broker nohup ./bin/kafka-server-start.sh config/server.properties & 发送消息 https://gitee.com/dyyx/kafkademo/blob/master/src/main/java/dyyx/ProducerTest.java 自动创建topic 消费消息 https://gitee.com/dyyx/kafkademo/blob/master/src/main/java/dyyx/ConsumerTest.java 查看 offset ./bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group kafkademo_test_group 停止 broker kafka-server-stop.sh 可以命令行工具 发送消费消息 kafka命令行发送接收消息
数据存储 kafka0/kafka-logs/kafka_demo_test_topic-1 00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex leader-epoch-checkpoint ../../bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log Starting offset: 0 baseOffset: 0 lastOffset: 0 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 3 isTransactional: false position: 0 CreateTime: 1597727377253 isvalid: true size: 128 magic: 2 compresscodec: NONE crc: 2324548880 baseOffset: 1 lastOffset: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 3 isTransactional: false position: 128 CreateTime: 1597727377275 isvalid: true size: 128 magic: 2 compresscodec: NONE crc: 3632234908 size: 128 compresscodec: NONE 查看主题offset ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --time -1 --topic kafka_demo_test_topic --time timestamp of the offsets before that (earliest) (default: -1)
查看数据文件数据 ./kafka-run-class.sh kafka.tools.DumpLogSegments --files /Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs/kafka_demo_test_topic-1/00000000000000000023.log | head -n 10 ./kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --files /Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs/kafka_demo_test_topic-1/00000000000000000023.log | head -n 10 ./kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs/kafka_demo_test_topic-1/00000000000000000023.log | head -n 10 --print-data-log 显示消息数据 offset: 23 position: 0 CreateTime: 1598148413924 isvalid: true keysize: 16 valuesize: 45 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 1598148413924_14 payload: 1598148413924_14,Sun Aug 23 10:06:53 CST 2020 offset: 24 position: 130 CreateTime: 1598148413961 isvalid: true keysize: 16 valuesize: 45 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 1598148413960_18 payload: 1598148413960_18,Sun Aug 23 10:06:53 CST 2020 offset: 25 position: 260 CreateTime: 1598148414002 isvalid: true keysize: 16 valuesize: 45 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: 1598148414001_23 payload: 1598148414001_23,Sun Aug 23 10:06:54 CST 2020
性能压测 ./kafka-producer-perf-test.sh --topic kafka_demo_test_topic --num-records 1000000 --throughput -1 --record-size 128 --producer-props bootstrap.servers=localhost:9092 acks=-1 linger.ms=2000 compression.type=lz4 ./kafka-consumer-perf-test.sh --broker-list localhost:9092 --messages 1000000 --topic producer
配置文件实例 kafka-cluster/kafka0/config/server.properties broker.id=0 listeners=PLAINTEXT://localhost:9092 num.network.threads=5 num.io.threads=10 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/Users/dugang/soft/elk/kafka-cluster/kafka0/kafka-logs num.partitions=6 default.replication.factor=2 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=3 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=localhost:2182,localhost:2183/kafka zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0
常见异常 Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. UnknownTopicOrPartitionException 创建 topic 往 zk写入元数据,需要一定的时间 , 可稍后重试 This topic/partition doesn't exist. This exception is used in contexts where a topic doesn't seem to exist based on possibly stale metadata. This exception is retriable because the topic or partition might subsequently be created. producer 可捕获此异常,并重试
完整例子代码 https://gitee.com/dyyx/kafkademo
kafka命令行发送接收消息 zookeeper集群搭建 kafka在zookeeper中的元数据 kafka1.1.0无法消费问题

上一篇     下一篇
降级 限流 熔断 简单理解

kafka在zookeeper中的元数据

Kafka配置参数

kafka命令行发送接收消息

zookeeper命令行zkCli使用

spring构造注入优缺点