filebeat+logstash收集錯(cuò)誤日志發(fā)送郵件提醒

20181109110339174.png

因?yàn)橹皇占e(cuò)誤日志并且數(shù)據(jù)量并不是非常大所以簡化流程

使用filebeat+logstash發(fā)送異常日志

一、filebeat配置

1、將安裝包解壓到指定目錄

drwxr-xr-x  3 root root     4096 Oct 28 09:46 ./
drwx------ 13 root root     4096 Nov  1 13:30 ../
drwxr-xr-x  7 root root     4096 Oct 31 16:30 filebeat-8.4.3-linux-x86_64/
-rw-r--r--  1 root root 60723429 Oct 28 09:46 filebeat-8.4.3-linux-x86_64.tar.gz

2、找到filebeat.yml文件

root@e:/# cd filebeat-8.4.3-linux-x86_64/
root@e:/# filebeat-8.4.3-linux-x86_64# ls
@  data  fields.yml  filebeat  filebeat.reference.yml  filebeat.yml  kibana  LICENSE.txt  logs  module  modules.d  nohup.out  NOTICE.txt  README.md

3、配置采集

參考文檔：https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

# ============================== Filebeat inputs ===============================
max_procs: 1 # 配置cpu核數(shù) 減少資源占用
queue.mem.events: 2048                  # 存儲于內(nèi)存隊(duì)列的事件數(shù)，排隊(duì)發(fā)送 (默認(rèn)4096)
queue.mem.flush.min_events: 1536        # 小于 queue.mem.events ,增加此值可提高吞吐量 (默認(rèn)值2048)

filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream  # 指定數(shù)據(jù)的輸入類型
  ignore_older: 48h # 忽略這個(gè)時(shí)間之前的文件(根據(jù)文件改變時(shí)間)
  max_bytes: 20480  # *單條日志的大小限制,將其從默認(rèn)10M降低到20k，按照公式計(jì)算 20k * 4096 ～= 80M
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  # Change to true to enable this input configuration.
  enabled: true # 啟用

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/logs/aaa/debug.log           # 日志路徑
    #- c:\programdata\elasticsearch\logs\*
  tail_files: true   # 從日志最后一行開始讀取
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  exclude_lines: ['DEBUG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  include_lines: ['ERROR','Exception']  #只處理包含ERROR 和 Exception日志
  parsers:
   - multiline:              #合并多行日志 
       pattern: '^\d{4}-\d{2}-\d{2}'       # 匹配以 YYYY-MM-DD HH:mm:ss 開頭的行 
       negate: true                        # 是否匹配pattern的情況
       match: after                        # 將其追加到上一行之后 pattern + negate + match 組合成一條語意為: 如果匹配 YYYY-MM-DD HH:mm:ss 開頭的行，則將其合并到當(dāng)前行的上一行
       max_lines: 20                      # 最多匹配多少行，如果超出最大行數(shù)，則丟棄多余的行(默認(rèn)500)
       timeout: 2s                         # 超時(shí)時(shí)間后,即使還未匹配到下一個(gè)行日志(下一個(gè)多行事件)，也將此次匹配的事件刷出 (默認(rèn)5s)
  
# ------------------------------ Logstash Output -------------------------------
#指定輸出的logstash地址
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.1.12:5044"]

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info  #filebeat日志級別 調(diào)試時(shí)可以設(shè)為debug

4、啟動

/filebeat-8.4.3-linux-x86_64# ./filebeat -e -c filebeat.yml    前臺啟動
/filebeat-8.4.3-linux-x86_64# nohup ./filebeat -e -c filebeat.yml &  后臺啟動

注：正常運(yùn)行一段時(shí)間后filebeat自動關(guān)閉
解決：
1、使用service方式啟動
2、在原來的命令之前加上disown參數(shù)，這個(gè)參數(shù)將會使啟動的nohup進(jìn)程從當(dāng)前shell的作業(yè)列表中清除，從而避免nohup進(jìn)程在關(guān)閉這個(gè)shell時(shí)接收到SIGHUP信號

nohup ./filebeat -e -c filebeat.yml & disown

二、logstash配置

1、將安裝包解壓到指定目錄

drwxr-xr-x  4 root root      4096 Oct 31 14:13 ./
drwxr-xr-x 25 root root      4096 Oct 28 11:12 ../
drwxr-xr-x 14 root root      4096 Oct 31 16:29 logstash-8.5.3/
-rw-r--r--  1 root root 330116325 Oct 28 09:25 logstash-8.5.3-linux-x86_64.tar.gz

2、找到logstash-sample.conf 文件

root@e:/logstash-8.5.3/config# ls
jvm.options  log4j2.properties  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options

3、配置

參考文檔：https://www.elastic.co/guide/en/logstash/current/output-plugins.html

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
  beats {
    host => '192.168.1.12'  #本機(jī)ip
    port => 5044
  }
}

filter {
   # 匹配message中的時(shí)間格式化到logdate
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logdate}"]
    }  
   # 排除 IdempotentException 異常（不處理該異常）
    if [message] =~ "IdempotentException" {
      drop { }
    }
}

output {
  # 輸出到文件
  #file{
  # path => "/test/test.log"
  # codec => line {format => "%{message}"}
  # }
  
  # 接收參數(shù)輸出到控制臺
  #  stdout { codec => rubydebug }
  
  #輸出到郵件 騰訊企業(yè)郵箱為例
    email {
        port => 587
        address => 'smtp.exmail.qq.com'
        username => '***@***.com'
        password => '******'      #授權(quán)碼
        authentication => 'plain'
        from => '***@***.com'
        subject => '錯(cuò)誤告警：IP:%{[host][ip][0]}, 時(shí)間:%{logdate}'
        to => '***@***.com'
        use_tls => true
        domain => 'smtp.exmail.qq.com'
        body => '[服務(wù)器信息]IP：%{[host][ip][0]}\n[錯(cuò)誤信息]：%{[log][file][path]}\n%{message}'
    }
}

4、啟動

/logstash-8.5.3# ./bin/logstash -f ./config/logstash-sample.conf     前臺啟動
/logstash-8.5.3# nohup ./bin/logstash -f ./config/logstash-sample.conf  后臺啟動

5、 logstash配置調(diào)優(yōu) (logstash.yml、jvm.options）

lostash.yml: 
pipeline.workers: 1 (不配置的情況下，默認(rèn)是系統(tǒng)核數(shù)，控制output或filter插件的工作線程數(shù)（只能設(shè)置正整數(shù)），當(dāng)發(fā)現(xiàn)事件正在備份或CPU沒有飽和，則可以增加工作線程，以提高性能。) 

#內(nèi)存大小 根據(jù)服務(wù)器的性能進(jìn)行配置
jvm.options: 
-Xms1g 
-Xmx1g

6、郵件內(nèi)容

d0aa7d04d1015dc48272f2828085bf73.png