ELK日志分析平臺(tái)

ELK架構(gòu)圖例

flowchart LR
subgraph Z1[web cluster]
  subgraph web1
    H1([apache]) --> F1([filebeat])
  end
  subgraph web2
    H2([apache]) --> F2([filebeat])
  end
  subgraph web3
    H3([apache]) --> F3([filebeat])
  end
end
subgraph Z2[Logstash]
  F1 & F2 & F3 --> A1{input} --> A2{filter} --> A3{output}
end
subgraph Z3[ES Cluster]
  ES1([Elasticsearch]);ES2([Elasticsearch]);ES3([Elasticsearch]);ES4([Elasticsearch]);ES5([Elasticsearch])
end
A3 --> ES1 & ES2 & ES3 & ES4 & ES5
ES1 & ES2 & ES3 & ES4 & ES5 --> K((kibana))
classDef WEB color:#ff0000,fill:#99ff99
class web1,web2,web3 WEB
classDef ZONE fill:#ffffc0,color:#ff00ff
class Z1,Z2,Z3 ZONE

logstash安裝

購(gòu)買云主機(jī)

logstash云主機(jī)安裝

[root@logstash ~]# vim /etc/hosts
192.168.1.41    es-0001
192.168.1.42    es-0002
192.168.1.43    es-0003
192.168.1.44    es-0004
192.168.1.45    es-0005
192.168.1.47    logstash
[root@logstash ~]# yum install -y java-1.8.0-openjdk logstash

基礎(chǔ)配置樣例

[root@logstash ~]# ln -s /etc/logstash /usr/share/logstash/config
[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input { 
  stdin {}
}

filter{ }

output{ 
  stdout{}
}
[root@logstash ~]# /usr/share/logstash/bin/logstash

插件與調(diào)試格式

使用json格式字符串測(cè)試 {"a":"1", "b":"2", "c":"3"}

[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input { 
  stdin { codec => "json" }
}

filter{ }

output{ 
  stdout{ codec => "rubydebug" }
}
[root@logstash ~]# /usr/share/logstash/bin/logstash

官方手冊(cè)地址

https://www.elastic.co/guide/en/logstash/current/index.html

input file插件

[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input { 
  file {
    path => ["/tmp/c.log"]
    type => "test"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sincedb"
  }
}
filter{ }
output{ 
  stdout{ codec => "rubydebug" }
}
[root@logstash ~]# rm -rf /var/lib/logstash/plugins/inputs/file/.sincedb_*
[root@logstash ~]# /usr/share/logstash/bin/logstash

filter grok插件

正則表達(dá)式分組匹配格式: (?<名字>正則表達(dá)式)

正則表達(dá)式宏調(diào)用格式: %{宏名稱:名字}

宏文件路徑

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns

[root@logstash ~]# echo '192.168.1.252 - - [29/Jul/2020:14:06:57 +0800] "GET /info.html HTTP/1.1" 200 119 "-" "curl/7.29.0"' >/tmp/c.log
[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input { 
  file {
    path => ["/tmp/c.log"]
    type => "test"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter{ 
  grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  }
}
output{ 
  stdout{ codec => "rubydebug" }
}
[root@logstash ~]# /usr/share/logstash/bin/logstash

output elasticsearch插件

[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input { 
  file {
    path => ["/tmp/c.log"]
    type => "test"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter{ 
  grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  }
}
output{ 
  stdout{ codec => "rubydebug" }
  elasticsearch {
    hosts => ["es-0004:9200", "es-0005:9200"]
    index => "weblog-%{+YYYY.MM.dd}"
  }
}
[root@logstash ~]# /usr/share/logstash/bin/logstash

瀏覽器打開(kāi) head 插件，通過(guò) web 頁(yè)面瀏覽驗(yàn)證
http://公網(wǎng)IP:9200/_plugin/head/

filebeat配置

logstash beats插件

[root@logstash ~]# vim /etc/logstash/conf.d/my.conf
input { 
  stdin { codec => "json" }
  file{
    path => ["/tmp/c.log"]
    type => "test"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sincedb"
  }
  beats {
    port => 5044
  }
} 

filter{ 
  grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
  }
} 

output{ 
  stdout{ codec => "rubydebug" }
  elasticsearch {
    hosts => ["es-0004:9200", "es-0005:9200"]
    index => "weblog-%{+YYYY.MM.dd}"
  }
}
[root@logstash ~]# /usr/share/logstash/bin/logstash

web服務(wù)安裝filebeat

[root@web ~]# yum install -y filebeat
[root@web ~]# vim /etc/filebeat/filebeat.yml
24:  enabled: true
28:  - /var/log/httpd/access_log
45:    fields: 
46:       my_type: apache
148, 150 注釋掉
161: output.logstash:
163:   hosts: ["192.168.1.47:5044"]
180, 181, 182 注釋掉
[root@web ~]# grep -Pv "^\s*(#|$)" /etc/filebeat/filebeat.yml
[root@web ~]# systemctl enable --now filebeat

網(wǎng)站日志分析實(shí)戰(zhàn)

1、停止 kibana 服務(wù)

[root@kibana ~]# systemctl stop kibana

2、清空 elasticsearch 中所有數(shù)據(jù)

[root@kibana ~]# curl -XDELETE http://es-0001:9200/*

訪問(wèn) web 頁(yè)面，瀏覽器打開(kāi) head 插件，通過(guò) web 頁(yè)面瀏覽驗(yàn)證

3、配置 web 日志，獲取用戶真實(shí)IP
通過(guò) ELB 把 web 服務(wù)發(fā)布公網(wǎng)
https://support.huaweicloud.com/elb_faq/elb_faq_0090.html

4、配置 filebeat
詳見(jiàn)配置文件 filebeat.yml
重啟服務(wù)

[root@web ~]# systemctl restart filebeat

5、配置 logstash
詳見(jiàn)配置文件 logstash.conf
啟動(dòng)服務(wù)

[root@logstash ~]# /usr/share/logstash/bin/logstash

6、配置 kibana
啟動(dòng)服務(wù)，通過(guò)web頁(yè)面配置 kibana

[root@kibana ~]# systemctl start kibana

常見(jiàn)錯(cuò)誤

使用通配符刪除報(bào)錯(cuò)

[root@es-0001 ~]# curl -XDELETE http://localhost:9200/*
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Wildcard expressions or all indices are not allowed"}],"type":"illegal_argument_exception","reason":"Wildcard expressions or all indices are not allowed"},"status":400}
# 由于設(shè)置了destructive_requires_name 參數(shù)，不允許使用通配符
# 查看及解決方式
[root@es-0001 ~]# curl -XGET http://es-0001:9200/_cluster/settings?pretty
{
  "persistent" : {
    "action" : {
      "destructive_requires_name" : "true"
    }
  },
  "transient" : { }
}
[root@es-0001 ~]# curl -XPUT http://localhost:9200/_cluster/settings -d '
{
  "persistent": {
      "action": {
        "destructive_requires_name": "false"
      }
   }
}'
[root@es-0001 ~]# curl -XDELETE http://localhost:9200/*
{"acknowledged":true}