ELK架構圖：

logstash

官方網(wǎng)站：https://www.elastic.co/
logstash工作模式：Agent---Server
logstash工作流程：input---(filter,codec)---output
Agent與Server并無區(qū)別。
常用插件：
input plugins: stdin,file,redis,
filter plugins:grok,
output plugins:stdout,redis,elasticsearch,
logstash是屬于重量級數(shù)據(jù)收集工具，需要有JDK環(huán)境。

部署JDK

# yum install -y java-1.8.0-openjdk-headles java-1.8.0-openjdk-devel java-1.8.0-openjdk
# echo "export JAVA_HOME=/usr" > /etc/profile.d/java.sh
# source /etc/profile.d/java.sh

安裝Logstash

Logstash版本有：1.X,2.X,5.X
(vm做實驗可以設置CPU雙核四線程，2G內(nèi)存，重量級)
# yum install -y logstash-1.5.4-1.noarch.rpm
# echo "export PATH=/opt/logstash/bin:$PATH" > /etc/profile.d/logstash.sh //logstash命令路徑
# /etc/sysconfig/logstash 啟動參數(shù)
# /etc/logstash/conf.d/ 此目錄下的所有文件
# logstash --help //需要好一會兒才出現(xiàn)幫助，啟動比較慢
編輯測試文件：
# vim /etc/logstash/conf.d/simple.conf
# input { //設置數(shù)據(jù)輸入方式
# stdin {} //標準輸入，鍵盤
# }
# output { //設置數(shù)據(jù)輸出方式
# stdout { //標準輸出，屏幕
# codec => rubydebug //采用輸出格式
# }
# }
運行：
# logstash -f /etc/logstash/conf.d/simple.conf --configtest //測試配置文件編寫是否正確
# Configuration OK
# logstash -f /etc/logstash/conf.d/simple.conf //運行
# Logstash startup completed //信息提示啟動完成
# hello,logstash //此時等待我們從標準輸入數(shù)據(jù)(鍵盤)，接著會在標準輸出(屏幕)打印如下數(shù)據(jù)
# {
# "message" => "hello,logstash",
# "@version" => "1",
# "@timestamp" => "2017-03-02T09:35:12.773Z",
# "host" => "elk"
# }
Logstash基本工作流程完成，接下來就是研究各類插件。

input plugins:file, udp

file作為數(shù)據(jù)輸入，參考說明https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-file.html#_file_rotation //官方說明
# vim /etc/logstash/conf.d/file-simple.conf
# input {
# file {
# path => ["/var/log/httpd/access_log"] //數(shù)組，可以輸入多個日志文件
# type => "system" //歸類，可以在filter插件中調(diào)用
# start_position => "beginning" //文件內(nèi)容的監(jiān)控位置從最先開始，(日志滾動是從新的日志文件第一行開始監(jiān)控)
# }
# }
# output {
# stdout {
# codec => rubydebug
# }
# }
# logstash -f /etc/logstash/conf.d/file-simple.conf --configtest
# logstash -f /etc/logstash/conf.d/file-simple.conf
采用udp方式來輸入數(shù)據(jù)到logstash，官方說明：https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-udp.html
數(shù)據(jù)生產(chǎn)者將數(shù)據(jù)以udp協(xié)議的方式通過網(wǎng)絡發(fā)送至logstash指定的udp端口
數(shù)據(jù)生產(chǎn)者采用collectd性能監(jiān)控工具實現(xiàn)，epel源中安裝。
# 另外一臺主機
# yum install collectd -y
# [root@elknode1 ~]# grep -Ev "(^#|$)" /etc/collectd.conf
# Hostname "elk-node1"
# LoadPlugin syslog
# LoadPlugin cpu
# LoadPlugin df
# LoadPlugin disk
# LoadPlugin interface
# LoadPlugin load
# LoadPlugin memory
# LoadPlugin network
# <Plugin network>
# <Server "192.168.9.77" "25826"> //將監(jiān)控數(shù)據(jù)發(fā)送至此Server
# </Server>
# </Plugin>
# Include "/etc/collectd.d"
# systemctl start collectd.service
配置logstash文件：
# vim /etc/logstash/conf.d/udp-simple.conf
# input {
# udp {
# port => "25826"
# codec => collectd {}
# type => "collectd"
# }
# }
# output {
# stdout {
# codec => rubydebug
# }
# }
# logstash -f /etc/logstash/conf.d/udp-simple.conf --configtest
# logstash -f /etc/logstash/conf.d/udp-simple.conf
啟動完成就會有日志信息送過來
# {
# "host" => "elk-node1",
# "@timestamp" => "2017-02-28T23:46:14.354Z",
# "plugin" => "disk",
# "plugin_instance" => "dm-1",
# "collectd_type" => "disk_ops",
# "read" => 322,
# "write" => 358,
# "@version" => "1",
# "type" => "collectd"
# }

filter plugins:grok(web日志核心插件)

grok用于分析并結構化文本數(shù)據(jù)
格式化模版：
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns 此目錄下放置很多默認模版：
aws bro firewalls grok-patterns haproxy java junos linux-syslog mcollective mongodb nagios postgresql rails redis ruby
# COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
# 以上就是apache的common日志格式模版，
# 以IPORHOST為例：模版文件中定義了如何去匹配IP或者HOST
# IPORHOST (?:%{HOSTNAME}|%{IP})
# HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(.?|\b)
# HOST %{HOSTNAME}
# IPV4 (?<

output plugins：redis, elasticsearch

(在ELK圖中l(wèi)ogstash server的數(shù)據(jù)來源很有可能是redis，也就是在input plugins上采用redis插件)
redis作為數(shù)據(jù)輸出對象,https://www.elastic.co/guide/en/logstash/1.5/plugins-outputs-redis.html
部署redis:
# yum install -y redis
# vim /etc/redis.conf
# bind 0.0.0.0
# systemctl start redis.service
測試redis正常工作:
# [root@elk patterns]# redis-cli
# 127.0.0.1:6379> SET name neo
# OK
# 127.0.0.1:6379> get name
# "neo"
# 127.0.0.1:6379>
將nginx日志數(shù)據(jù)結構化之后輸出到redis數(shù)據(jù)庫中：
# vim /etc/logstash/conf.d/grok-nginx-redis.conf
# input {
# file {
# path => ["/var/log/nginx/access.log"]
# type => "nginxlog"
# }
# }
# filter {
# grok {
# match => { "message" => "%{NGINXACCESS}" }
# }
# }
# output {
# redis {
# port => "redis"
# host => "192.168.9.77"
# data_type => "list" //list,channel兩種模式
# key => "logstash-nginxlog" //The name of a Redis list or channel.
# }
# }
# logstash -f grok-nginx-redis.conf.conf --configtest
# systemctl -f /etc/logstash/conf.d/grok-nginx-redis.conf
在redis數(shù)據(jù)庫中檢查數(shù)據(jù)：
# [root@elk patterns]# redis-cli -h 192.168.9.77
# 192.168.9.77:6379> llen logstash-nginxlog
# (integer) 66
# 192.168.9.77:6379> lindex logstash-nginxlog 65
# "{"message":"192.168.0.215 - - [02/Mar/2017:21:38:07 +0800] \"GET /poweredby.png HTTP/1.1\" 200 2811 \"http://192.168.9.77/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0\" \"-\"","@version":"1","@timestamp":"2017-03-02T13:38:07.870Z","host":"elk","path":"/var/log/nginx/access.log","type":"nginxlog","clientip":"192.168.0.215","remote_user":"-","timestamp":"02/Mar/2017:21:38:07 +0800","verb":"GET","request":"/poweredby.png","httpversion":"1.1","response":"200","bytes":"2811","referrer":"\"http://192.168.9.77/\"","agent":"\"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0\"","http_x_forwarded_for":"\"-\""}"
# 192.168.9.77:6379>
#已經(jīng)可以查看到redis收到了logstash發(fā)送過來的數(shù)據(jù)
elasticsearch作為logstash的數(shù)據(jù)輸出對象：
# vim /etc/elasticsearch/elasticsearch.yml
# cluster.name: myes
# node.name: "elk"
# network.bind_host: 192.168.9.77
# transport.tcp.port: 9300
# http.port: 9200
配置elasticsearch輸出的logstash配置文件：以訪問日志文件作為輸入，經(jīng)過grok數(shù)據(jù)結構化處理輸出至es
# vim /etc/logstash/conf.d/grok-nginx-es.conf
# input {
# file {
# path => ["/var/log/nginx/access.log"]
# type => "nginxlog"
# }
# }
# filter {
# grok {
# match => { "message" => "%{NGINXACCESS}" }
# }
# }
# output {
# elasticsearch {
# cluster => "myes"
# index => "logstash-%{+YYYY.MM.dd}"
# }
# }
# logstash -f /etc/logstash/conf.d/grok-nginx-es.conf --configtest
# logstash -f /etc/logstash/conf.d/grok-nginx-es.conf
測試查看es上的存儲數(shù)據(jù)：
# [root@elk patterns]# curl -XGET 'http://192.168.9.77:9200/_cat/indices'
# yellow open .marvel-2017.03.02 1 1 15208 0 19.9mb 19.9mb
# yellow open .marvel-2017.03.01 1 1 2054 0 4.4mb 4.4mb
# yellow open logstash-2017.03.02 5 1 45 0 141.2kb 141.2kb
# logstash-2017.03.02 索引數(shù)據(jù)就是日志數(shù)據(jù)的索引
# curl -XGET 'http://192.168.9.77:9200/logstash-2017.03.02/_search?pretty' //查看所有數(shù)據(jù)
通過在es安裝head插件，來web展示搜索數(shù)據(jù)：