實(shí)驗(yàn)拓?fù)鋱D

lab

1、jdk環(huán)境部署好（java-1.8.0-openjdk）
2、ELK軟件版
redis 2.8 epel
logstash 1.5 rpm
es 1.7 rpm
kibana 4.1 rpm
3、安裝部署
elk-node3:(logstash,nginx 192.168.9.120)
# ~]# yum install /data/pkg/logstash-1.5.4-1.noarch.rpm nginx -y
# systemctl start nginx.service
# systemctl enable nginx.service
# vim /etc/logstash/conf.d/nginx-redis.conf
# input {
# file {
# path => ["/var/log/nginx/access.log"]
# type => "nginxlog"
# }
# }
# output {
# redis {
# host => "192.168.9.119"
# port => "6379"
# data_type => "list"
# key => "logstash-nginxlog"
# }
# }
# ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-redis.conf --configtest
# ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-redis.conf

elk-node2:(redis 192.168.9.119)
# vim /etc/redis.conf
# bind 0.0.0.0
# systemctl start redis.service
# systemctl enable redis.service

elk-node1:(logstash-server 192.168.9.118)
# ~]# yum install /data/pkg/logstash-1.5.4-1.noarch.rpm -y
# vi /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/nginx
# NGUSERNAME [a-zA-Z.@-+_%]+
# NGUSER %{NGUSERNAME}
# NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}
# ========
# vim /etc/logstash/conf.d/redis-grok-es.conf
# input {
# redis {
# host => "192.168.9.119"
# port => "6379"
# data_type => "list"
# key => "logstash-nginxlog"
# }
# }
# filter {
# grok {
# match => {"message" => "%{NGINXACCESS}"}
# }
# }
# output {
# elasticsearch {
# cluster => "loges"
# index => "logstash-%{+YYYY.MM.dd}"
# }
# }
elk:(elasticsearch,kibana 192.168.9.77)
# yum install -y elasticsearch-1.7.2.noarch.rpm
# vi /etc/elasticsearch/elasticsearch.yml
# cluster.name: loges
# node.name: "elk"
# 安裝head插件，上傳到plugins目錄解壓就可以用了
# cd /usr/share/elasticsearch/plugins/
# unzip elasticsearch-head-latest.zip
# mv elasticsearch-head-master/ head
# systemctl enable elasticsearch.service
# systemctl start elasticsearch.service
注意點(diǎn)：(簡單部署測試)
1、elasticsearch是最終的數(shù)據(jù)分布式存儲(chǔ)。
2、logstash-server實(shí)時(shí)去redis拉取數(shù)據(jù)
3、logstash-agent實(shí)時(shí)向redis推送數(shù)據(jù)
4、當(dāng)運(yùn)氣起來后，redis中是看不到數(shù)據(jù)了，查看keys是顯示沒有一個(gè)keys，其實(shí)數(shù)據(jù)已經(jīng)被推送到了ES。
5、web日志數(shù)據(jù)結(jié)構(gòu)化(grok)是在logstash-server上實(shí)施(當(dāng)然也可以在agent，在大規(guī)模環(huán)境下，數(shù)據(jù)統(tǒng)一由logstash-server處理會(huì)比較好，減少前段web負(fù)載，日志時(shí)間)
ES查看數(shù)據(jù)信息：

head