Apache kafka 開源分布式事件流平臺，用于數(shù)據(jù)管道、流分析、數(shù)據(jù)集成等。
kafka由servers和clients組成，通過TCP網(wǎng)絡(luò)協(xié)議交流。
event (record or message)
Event key: "Alice"
Event value: "Made a payment of $200 to Bob"
Event timestamp: "Jun. 25, 2020 at 2:06 p.m."
optional metadata headers
Producer vs Consumer
Producers are those client applications that publish (write) events to Kafka, and consumers are those that subscribe to (read and process) these events.
Topic
事件存儲在topic中
Topics in Kafka are always multi-producer and multi-subscriber: a topic can have zero, one, or many producers that write events to it, as well as zero, one, or many consumers that subscribe to these events.
每個topic都支持多生產(chǎn)者&多消費(fèi)者。
partition 分區(qū)
Topics are partitioned, meaning a topic is spread over a number of "buckets" located on different Kafka brokers.
topic是分區(qū)的，這意味著一個topic分布在位于不同Kafka代理上的多個“桶”上。
Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.
消費(fèi)的順序跟生產(chǎn)的順序嚴(yán)格一致。
數(shù)據(jù)安全
A common production setting is a replication factor of 3, i.e., there will always be three copies of your data.
數(shù)據(jù)都是備份的，默認(rèn)備3份。
APIs
admin API 管理
producer API 生產(chǎn)
consumer API 消費(fèi)
streams API 事件流處理高級功能
connect API 負(fù)責(zé)導(dǎo)入導(dǎo)出數(shù)據(jù)
安裝
- tar包安裝，像ES
- docker鏡像
- Zookeeper
python SDK kafka-python(kafka-python-ng)
查看官方文檔https://kafka-python.readthedocs.io/en/master/index.html
