apache druid(時(shí)序數(shù)據(jù)庫)集群搭建

準(zhǔn)備環(huán)境

mysql、zookeeper、hadoop(HDFS)環(huán)境
jdk1.8(每臺(tái)druid服務(wù)器都需要安裝)

druid的服務(wù)器列表

服務(wù)器地址 服務(wù)器角色 備注
10.32.10.240 druid-query Broker、Router
10.32.10.241 druid-query Broker、Router
10.32.10.242 druid-master coordinator、overlord
10.32.10.243 druid-master coordinator、overlord
10.32.10.244 druid-data Historical、MiddleManager
10.32.10.245 druid-data Historical、MiddleManager

開始搭建druid集群

下載apache-druid-0.18.0-bin.tar.gz并上傳到服務(wù)器(/mnt/druid)

所有服務(wù)器都需要操作

進(jìn)入/mnt/druid，解壓tar包

??tar -zxvf apache-druid-0.18.0-bin.tar.gz

進(jìn)行druid的集群配置共公目錄/mnt/druid/apache-druid-0.18.0/conf/druid/cluster/_common

??cd apache-druid-0.18.0/conf/druid/cluster/_common/

配置common.runtime.properties

??vim common.runtime.properties

修改common.runtime.properties配置文件
druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "mysql->metadata-storage", "druid-hdfs-storage", "druid-kafka-indexing-service"]
druid.host=10.32.10.240  #本機(jī)服務(wù)器地址
### zookeeper配置
druid.zk.service.host=192.168.11.111:2181,192.168.11.112:2181,192.168.11.113:2181
druid.zk.paths.base=/druid018
# Metadata storage
# For MySQL (make sure to include the MySQL JDBC driver on the classpath):
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://127.0.0.1:3306/druid018?characterEncoding=UTF8
druid.metadata.storage.connector.user=root
druid.metadata.storage.connector.password=root
# For HDFS:
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://10.32.10.16:8020/druid018/segments
#
# Indexing service logs
#
# For HDFS:
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=hdfs://10.32.10.16:8020/druid018/indexing-logs
修改完成后，創(chuàng)建druid日志目錄

??mkdir -p /mnt/druid/apache-druid-0.18.0/var/sv/

引入以下mysql包到apache-druid-0.18.0/extensions/mysql-metadata-storage目錄

mysql-connector-java-5.1.48.jar、password-connector-java-5.1.48.jar

配置master coordinator-overlord（242 243 節(jié)點(diǎn)同步操作)

進(jìn)行配置文件目錄
??cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/master/coordinator-overlord
jvm.config配置

-server
-Xms15g
-Xmx15g
-XX:+ExitOnOutOfMemoryError
-XX:+UseG1GC
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Dderby.stream.error.file=var/druid/derby.log

runtime.properties配置

druid.service=druid/coordinator
druid.plaintextPort=8081
druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S
# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT5S
druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

啟動(dòng)druid的master節(jié)點(diǎn)
/apache-druid-0.18.0/bin/start-cluster-master-no-zk-server &

配置 data節(jié)點(diǎn) MiddleManager Historical （244 245節(jié)點(diǎn)同時(shí)操作）

MiddleManager配置：
進(jìn)行配置文件目錄
??cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/data/middleManager
jvm.config配置

-server
-Xms1g
-Xmx1g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

runtime.properties配置

druid.service=druid/middleManager
druid.plaintextPort=8091
# Number of tasks per middleManager
druid.worker.capacity=4
# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g ->Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError ->Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task
# HTTP server threads
druid.server.http.numThreads=60
# Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000
druid.indexer.fork.property.druid.processing.numThreads=1
# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

Historical配置：
進(jìn)行配置文件目錄
??cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/data/historical
jvm.config配置

-server
-Xms8g
-Xmx8g
-XX:MaxDirectMemorySize=13g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
runtime.properties配置
druid.service=druid/historical
druid.plaintextPort=8083
# HTTP server threads
druid.server.http.numThreads=60
# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=4     #1:4 cpu核數(shù)
druid.processing.numThreads=15      # cpu核數(shù)-1
druid.processing.tmpDir=var/druid/processing
# Segment storage
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":300000000000}]
druid.server.maxSize=300000000000
# Query cache
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=256000000

啟動(dòng)命令：
/apache-druid-0.18.0/bin/start-cluster-data-server &

配置 query節(jié)點(diǎn) broker & router(240 241節(jié)點(diǎn)同時(shí)操作）

borker配置：
進(jìn)行配置文件目錄
??cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/query/broker
jvm.config配置

-server
-Xms12g
-Xmx12g
-XX:MaxDirectMemorySize=6g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
runtime.properties配置
druid.service=druid/broker
druid.plaintextPort=8082
# HTTP server settings
druid.server.http.numThreads=60
# HTTP client settings
druid.broker.http.numConnections=50
druid.broker.http.maxQueuedBytes=10000000
# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=6
druid.processing.numThreads=1
druid.processing.tmpDir=var/druid/processing
# Query cache disabled -- push down caching and merging instead
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false

borker配置：
進(jìn)行配置文件目錄
??cd /mnt/druid/apache-druid-0.18.0/conf/druid/cluster/query/router
jvm.config配置

-server
-Xms1g
-Xmx1g
-XX:+UseG1GC
-XX:MaxDirectMemorySize=128m
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
runtime.properties配置
druid.service=druid/router
druid.plaintextPort=8888
# HTTP proxy
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.server.http.numThreads=100
# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
# Management proxy to coordinator / overlord: required for unified web console.
druid.router.managementProxy.enabled=true

啟動(dòng)命令：
/apache-druid-0.18.0/bin/start-cluster-query-server &

配置完成后，由druid的query節(jié)點(diǎn)的8888端口提供可視化頁面，可以通過nginx來反向代理10.32.10.240:8888和10.32.10.241:8888

注意：?jiǎn)?dòng)完成后要通過exit退出服務(wù)器連接，不能直接關(guān)閉窗口或斷開連接，不然druid的后臺(tái)進(jìn)程也會(huì)被關(guān)閉

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

友情鏈接更多精彩內(nèi)容