Kafka性能測試之produce

在kafka的bin目錄下，有兩個腳本kafka-producer-perf-test.sh?和kafka-consumer-perf-test.sh，這兩個腳本的作用是用來測試生產者和消費者的。

[root@hostname bin]# ./kafka-producer-perf-test.sh --help

usage: producer-performance [-h] --topic TOPIC --num-records NUM-RECORDS [--payload-delimiter PAYLOAD-DELIMITER] --throughput THROUGHPUT

? ? ? ? ? ? ? ? ? ? ? ? ? ? [--producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]] [--producer.config CONFIG-FILE] (--record-size RECORD-SIZE |

? ? ? ? ? ? ? ? ? ? ? ? ? ? --payload-file PAYLOAD-FILE)

This tool is used to verify the producer performance.

optional arguments:

? -h, --help? ? ? ? ? ? show this help message and exit

? --topic TOPIC? ? ? ? ? produce messages to this topic

? --num-records NUM-RECORDS

? ? ? ? ? ? ? ? ? ? ? ? number of messages to produce

? --payload-delimiter PAYLOAD-DELIMITER

? ? ? ? ? ? ? ? ? ? ? ? provides delimiter to be used when --payload-file is? provided.? Defaults? to? new? line.? Note? that this parameter will be ignored if --

? ? ? ? ? ? ? ? ? ? ? ? payload-file is not provided. (default: \n)

? --throughput THROUGHPUT

? ? ? ? ? ? ? ? ? ? ? ? throttle maximum message throughput to *approximately* THROUGHPUT messages/sec

? --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]

? ? ? ? ? ? ? ? ? ? ? ? kafka producer related configuration properties like bootstrap.servers,client.id etc.? These configs take precedence over those passed via

? ? ? ? ? ? ? ? ? ? ? ? --producer.config.

? --producer.config CONFIG-FILE

? ? ? ? ? ? ? ? ? ? ? ? producer config properties file.

? either --record-size or --payload-file must be specified but not both.

? --record-size RECORD-SIZE

? ? ? ? ? ? ? ? ? ? ? ? message size in bytes. Note that you must provide exactly one of --record-size or --payload-file.

? --payload-file PAYLOAD-FILE

? ? ? ? ? ? ? ? ? ? ? ? file to read the message payloads from. This works only for UTF-8? encoded? text files. Payloads will be read from this file and a payload

? ? ? ? ? ? ? ? ? ? ? ? will be randomly selected when sending messages. Note that you must provide exactly one of --record-size or --payload-file.

根據(jù)help命令可以得知相關參數(shù)以及參數(shù)的含義。

[root@hadoop-sh1-core1 bin]# ./kafka-topics.sh --zookeeper hadoop-sh1-master2:2181 --topic "test003" --describe? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Topic:test003? PartitionCount:3? ? ? ? ReplicationFactor:1? ? Configs:

? ? ? ? Topic: test003? Partition: 0? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test003? Partition: 1? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

? ? ? ? Topic: test003? Partition: 2? ? Leader: 2? ? ? Replicas: 2? ? Isr: 2


[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test003 --num-records 1000000 --record-size 1024? --throughput -1 --producer.config ../config/producer.properties

[2018-07-02 14:29:03,086] WARN Error while fetching metadata with correlation id 1 : {test003=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

250992 records sent, 50198.4 records/sec (49.02 MB/sec), 425.2 ms avg latency, 1167.0 max latency.

526995 records sent, 103616.8 records/sec (101.19 MB/sec), 266.2 ms avg latency, 1080.0 max latency.

1000000 records sent, 83998.320034 records/sec (82.03 MB/sec), 317.24 ms avg latency, 1314.00 ms max latency, 259 ms 50th, 868 ms 95th, 1212 ms 99th, 1228 ms 99.9th.

不限制吞吐量，3個partition的topic，3個備份，ack=all的情況下，最高達到了101.19 MB/sec的速度，平均延遲在200多到400多左右，最高延遲是1228 ms。

調節(jié)topic參數(shù)(還是3臺機器)，將topic的partition數(shù)量調高到10，備份數(shù)降到1，還是不限制吞吐量

Created topic "test004".

[root@hadoop-sh1-core1 bin]# ./kafka-topics.sh --zookeeper hadoop-sh1-master2:2181 --topic "test004" --describe? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Topic:test004? PartitionCount:10? ? ? ReplicationFactor:1? ? Configs:

? ? ? ? Topic: test004? Partition: 0? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test004? Partition: 1? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

? ? ? ? Topic: test004? Partition: 2? ? Leader: 2? ? ? Replicas: 2? ? Isr: 2

? ? ? ? Topic: test004? Partition: 3? ? Leader: 3? ? ? Replicas: 3? ? Isr: 3

? ? ? ? Topic: test004? Partition: 4? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test004? Partition: 5? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

? ? ? ? Topic: test004? Partition: 6? ? Leader: 2? ? ? Replicas: 2? ? Isr: 2

? ? ? ? Topic: test004? Partition: 7? ? Leader: 3? ? ? Replicas: 3? ? Isr: 3

? ? ? ? Topic: test004? Partition: 8? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test004? Partition: 9? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1


[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 1000000 --record-size 1024? --throughput -1 --producer.config ../config/producer.properties

525946 records sent, 105189.2 records/sec (102.72 MB/sec), 209.1 ms avg latency, 1107.0 max latency.

1000000 records sent, 114155.251142 records/sec (111.48 MB/sec), 226.50 ms avg latency, 1203.00 ms max latency, 152 ms 50th, 1007 ms 95th, 1191 ms 99th, 1196 ms 99.9th.? ? ? ?

從上面可以看見，平均延遲和最大延遲以及吞吐量都上升了許多，因為還是只有三臺機器，吞吐量可能只能到100M多每秒，不知道再增加是否還會提升。

還是3個partiton，3份備份，將吞吐降到了1w，再看看測試情況

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test003 --num-records 1000000 --record-size 1024? --throughput 10000 --producer.config ../config/producer.properties

49979 records sent, 9995.8 records/sec (9.76 MB/sec), 1.4 ms avg latency, 203.0 max latency.

50043 records sent, 10008.6 records/sec (9.77 MB/sec), 0.6 ms avg latency, 18.0 max latency.

49973 records sent, 9994.6 records/sec (9.76 MB/sec), 0.7 ms avg latency, 41.0 max latency.

50057 records sent, 10009.4 records/sec (9.77 MB/sec), 0.5 ms avg latency, 23.0 max latency.

50002 records sent, 10000.4 records/sec (9.77 MB/sec), 0.6 ms avg latency, 46.0 max latency.

50015 records sent, 10003.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 24.0 max latency.

49993 records sent, 9996.6 records/sec (9.76 MB/sec), 0.6 ms avg latency, 47.0 max latency.

50010 records sent, 10000.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 26.0 max latency.

50015 records sent, 10003.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 26.0 max latency.

50005 records sent, 10001.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 19.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 24.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.5 ms avg latency, 18.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.5 ms avg latency, 23.0 max latency.

50000 records sent, 10000.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 23.0 max latency.

50016 records sent, 10003.2 records/sec (9.77 MB/sec), 0.6 ms avg latency, 31.0 max latency.

49994 records sent, 9998.8 records/sec (9.76 MB/sec), 0.6 ms avg latency, 29.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 21.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 21.0 max latency.

50019 records sent, 10001.8 records/sec (9.77 MB/sec), 0.6 ms avg latency, 32.0 max latency.

1000000 records sent, 9999.200064 records/sec (9.76 MB/sec), 0.61 ms avg latency, 203.00 ms max latency, 1 ms 50th, 1 ms 95th, 2 ms 99th, 19 ms 99.9th.

平均延時降到了0.6ms，最大延遲在幾十ms以內。

接下來再調節(jié)record size大小，看是否造成影響。

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 1000000 --record-size 258? --throughput -1 --producer.config ../config/producer.properties

1000000 records sent, 323519.896474 records/sec (79.60 MB/sec), 4.77 ms avg latency, 210.00 ms max latency, 3 ms 50th, 18 ms 95th, 39 ms 99th, 46 ms 99.9th.

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 1000000 --record-size 1024? --throughput -1 --producer.config ../config/producer.properties

525946 records sent, 105189.2 records/sec (102.72 MB/sec), 209.1 ms avg latency, 1107.0 max latency.

1000000 records sent, 114155.251142 records/sec (111.48 MB/sec), 226.50 ms avg latency, 1203.00 ms max latency, 152 ms 50th, 1007 ms 95th, 1191 ms 99th, 1196 ms 99.9th.

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 10000000 --record-size 258? --throughput -1 --producer.config ../config/producer.properties

1949374 records sent, 389874.8 records/sec (95.93 MB/sec), 3.6 ms avg latency, 212.0 max latency.

2392851 records sent, 478570.2 records/sec (117.75 MB/sec), 2.3 ms avg latency, 21.0 max latency.

2372680 records sent, 474536.0 records/sec (116.76 MB/sec), 2.4 ms avg latency, 26.0 max latency.

2362539 records sent, 472507.8 records/sec (116.26 MB/sec), 2.3 ms avg latency, 24.0 max latency.

10000000 records sent, 455435.624175 records/sec (112.06 MB/sec), 2.54 ms avg latency, 212.00 ms max latency, 2 ms 50th, 4 ms 95th, 10 ms 99th, 20 ms 99.9th.

兩者對比可以發(fā)現(xiàn)，size從1024降到258，性能有了顯著 提升，延遲就降了不少，第三個，再將records數(shù)量上升到1000w條，發(fā)現(xiàn)，延遲并沒有上升對少，吞吐穩(wěn)定在了110M多，可能三臺機器的瓶頸就是這么多了。

由以上的測試結論可以得出，kafka的produce效率，和record size大小，partition數(shù)量，備份數(shù)，ack的值等有關。（當然，和單臺機器的性能也有一些關系。_）

?著作權歸作者所有,轉載或內容合作請聯(lián)系作者
【社區(qū)內容提示】社區(qū)部分內容疑似由AI輔助生成，瀏覽時請結合常識與多方信息審慎甄別。
平臺聲明：文章內容（如有圖片或視頻亦包括在內）由作者上傳并發(fā)布，文章內容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務。

相關閱讀更多精彩內容

  • 本文轉載自http://dataunion.org/?p=9307 背景介紹Kafka簡介Kafka是一種分布式的...
    Bottle丶Fish閱讀 5,583評論 0贊 34
  • Kafka簡介 Kafka是一種分布式的，基于發(fā)布/訂閱的消息系統(tǒng)。主要設計目標如下： 以時間復雜度為O(1)的方...
    Alukar閱讀 3,148評論 0贊 43
  • 背景介紹 Kafka簡介 Kafka是一種分布式的，基于發(fā)布/訂閱的消息系統(tǒng)。主要設計目標如下： 以時間復雜度為O...
    高廣超閱讀 13,043評論 8贊 167
  • “蟲穴？”沈紅英疑惑道。 “看來沈掌門并不知道?！?沈紅英不可否認地點了點頭。 “癸巳年十月初一，白馬紅刀肖邕；癸...
    葛弗閱讀 417評論 0贊 1
  • 劉葉葉～企劃文案～利他組 【日精進打卡第28天】 【知～學習】 《六項精進》2遍 共50遍 《大學》2遍 共61遍...
    小火紙兒閱讀 108評論 0贊 0

友情鏈接更多精彩內容