使用elk收集應用日志

使用redis作為broker，logstash作為indexer，收集日志進入es。

相關軟件版本號

• redis 3.0.6
• elasticsearch 5.3.0
• logstash 5.6.1
• kibana 5.3.0

logstash配置

創(chuàng)建文件：~/test/logstash/redis2es.conf

    input{
        redis { 
            batch_count => 1
            data_type => "list"
            key => "logstash-test"
            host => "127.0.0.1"
            port => 6379
            threads => 5
        }
    }
    output{
        stdout{
            codec => json_lines
        }
        elasticsearch{
            hosts => ["127.0.0.1:9200"]
            index => "logstash-%{type}-%{+YYYY.MM.dd}"
            document_type => "%{type}"
            sniffing => true
            template_overwrite => true
        }
    }

測試

1. 使用上述文件啟動logstash，ying:logstash-5.6.1 ying$ bin/logstash -f ~/test/logstash/redis2es.conf
2. 使用redis-cli發(fā)送數(shù)據(jù)到redis。

RPUSH logstash-test '{"name":"johnson.zheng","type":"test","say":"hello"}'

1. logstash看到輸出數(shù)據(jù)

{"name":"johnson.zheng","@version":"1","say":"hello","@timestamp":"2017-09-30T06:53:49.110Z","type":"test"}

1. kibana中也能看到數(shù)據(jù)

WX20170930-150244.png

后續(xù)，只需要在應用中只需要將日志寫入（使用RPUSH）到redis的logstash-test中，即可將日志收集進es，使用kibana分析。