一、 Flume安裝
1） 準(zhǔn)備
???????jdk:openjdk8
???????hadoop:hadoop-2.6.0-cdh5.14.0
???????在cloudera官網(wǎng)下載cdh5版本的flume安裝包
$ tar -zxvf flume-ng-1.6.0-cdh5.14.0.tar.gz -C /opt/cloudera/ # 解壓到/opt/cloudera目錄下
$ cd /opt/cloudera
$ sudo mv flume-ng-1.6.0-cdh5.14.0 flume # 將文件夾名改flume
$ sudo chown -R hadoop ./flume # 修改文件權(quán)限
二、 修改配置文件
$ cp conf/flume-env.sh.template conf/flume-env.sh
$ sudo vim conf/flume-env.sh
在配置文件flume-env.sh中，修改JAVA_HOME路徑：
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 #換成你的java路徑
三、 驗(yàn)證安裝是否成功
$ cd /opt/cloudera/flume
$ ./bin/flume-ng version
結(jié)果顯示如下，則證明安裝成功：
Flume 1.6.0-cdh5.14.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 939f6066f6056e7e8042292647eb5032628ac5a0
Compiled by jenkins on Sat Jan 6 13:37:47 PST 2018
From source with checksum c350591306508e3cdd4942fb1debefc9
四、 Flume小例子(Netcat )
???????任務(wù)介紹：NetCat Source：監(jiān)聽一個(gè)指定的網(wǎng)絡(luò)端口，即只要應(yīng)用程序向這個(gè)端口里面寫數(shù)據(jù)，這個(gè)source組件就可以獲取到信息。
①.創(chuàng)建你自己的屬性文件，基于現(xiàn)有的模板。
$ cp conf/flume-conf.properties.template conf/flume.conf
②.修改conf/flume.conf文件
文件內(nèi)容如下：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 通過channel將source與sink連接起來
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
五、 ubuntu安裝telnet
1. 首先更新一下
$ sudo apt-get update
2. 安裝
$ sudo apt-get install xinetd telnetd
3.修改/etc/xinetd.conf，并加入以下內(nèi)容：
# Simple configuration file for xinetd
# Some defaults, and include /etc/xinetd.d/
defaults
{
# Please note that you need a log_type line to be able to use log_on_success
# and log_on_failure. The default is the following :
# log_type = SYSLOG daemon info
instances = 60
log_type = SYSLOG authpriv
log_on_success = HOST PID
log_on_failure = HOST
cps = 25 30
}
includedir /etc/xinetd.d
4.修改/etc/xinetd.d/telnet并加入以下內(nèi)容：
# default: on
# description: The telnet server serves telnet sessions;it uses
# unencrypted username/password pairs for authentication.
service telnet
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/sbin/in.telnetd
log_on_failure += USERID
}
5.重啟機(jī)器或重啟網(wǎng)絡(luò)服務(wù)
### 1. 重啟系統(tǒng)
$ sudo reboot
### 2. 重啟網(wǎng)絡(luò)服務(wù)
$ sudo /etc/init.d/xinetd restart
6.測(cè)試telnet是否安裝成功
### 1. win10下使用cmd窗口執(zhí)行如下命令：
$ telnet ip地址
六、啟動(dòng)agent的shell操作：
$ cd /opt/cloudera/flume
$ ./bin/flume-ng agent -n a1 -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console
結(jié)果顯示如下,則證明運(yùn)行成功：
2018-08-13 19:35:41,475 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: []
2018-08-13 19:35:41,476 (conf-file-poller-0) [WARN - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:135)] No configuration found for this host:a1
2018-08-13 19:35:41,492 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
七、使用telnet發(fā)送消息
### 查看當(dāng)前開啟的端口號(hào)
$ netstat -tnlp
結(jié)果顯示如下：
tcp6 0 0 127.0.0.1:44444 :::* LISTEN 2124/java
### 監(jiān)聽端口號(hào)本機(jī)44444
$ telnet localhost 44444
結(jié)果顯示如下：
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
### 使用telnet進(jìn)行輸入
$ hellow flume
flume 結(jié)果顯示如下：
2018-08-13 20:35:23,610 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 14
2018-08-13 20:35:23,610 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1
2018-08-13 20:35:23,610 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 77 20 66 6C 75 6D 65 0D hellow flume. }
至此結(jié)束。
