本項(xiàng)目是通過(guò)采集Nginx的日志，將日志存儲(chǔ)到ES中，使用EA（ElasticAlert）調(diào)用ES的接口抓取數(shù)據(jù)過(guò)濾，對(duì)于觸發(fā)EA規(guī)則的調(diào)用釘釘?shù)慕涌诎l(fā)送告警
安裝過(guò)程；
1、Nginx安裝（略）
2、安裝ES（略）
3、安裝python 3
4、安裝elastalert
1、數(shù)據(jù)流：
（1）、用戶訪問(wèn)nginx，nginx產(chǎn)生訪問(wèn)日志（這里可以將Nginx與modsecurity整合做成WAF）；
（2）、filebeat讀取nginx的日志發(fā)給logstash；
（3）、logstash接收到日志，存儲(chǔ)到ES中；
（4）、在EA中定義規(guī)則，并抽取ES數(shù)據(jù)過(guò)濾，對(duì)于觸發(fā)規(guī)則的數(shù)據(jù)調(diào)用釘釘；
（5）、釘釘收到告警信息顯示給用戶。
2、安裝python 3.x
由于vim\yum\firewall-cmd命令都是依賴python 2.x，安裝python 3.x后會(huì)導(dǎo)致這些命令不可用
yum -y install wget openssl openssl-devel gcc gcc-c++
wget https://www.python.org/ftp/python/3.6.9/Python-3.6.9.tgz
tar xf Python-3.6.9.tgz
cd Python-3.6.9
./configure --prefix=/usr/local/python --with-openssl
make && make install
mv /usr/bin/python /usr/bin/python_old
ln -s /usr/local/python/bin/python3 /usr/bin/python
ln -s /usr/local/python/bin/pip3 /usr/bin/pip
pip install --upgrade pip
sed -i '1s/python/python2.7/g' /usr/bin/yum
sed -i '1s/python/python2.7/g' /usr/libexec/urlgrabber-ext-down
python -V
顯示為3.6.9
3、安裝elasticalert
可以直接安裝
pip install elastalert
也可以拉取源碼
git clone https://github.com/Yelp/elastalert.git? //建議使用源碼安裝
cd alastalert/
pip install setuptools-rust
python setup.py install
pip install -r requirements.txt
如有報(bào)錯(cuò)
elastalert 0.2.4 requires jira>=2.0.0, but you'll have jira 1.0.14 which is incompatible.
可執(zhí)行
pip install jira==2.0.0? //執(zhí)行后不用再執(zhí)行pip install -r requirements.txt
4、創(chuàng)建索引
# cd /usr/local/python3.6/bin/
# ./elastalert-create-index
root@ea bin]# ./elastalert-create-index
Enter Elasticsearch host: 192.168.25.160
Enter Elasticsearch port: 9200
Use SSL? t/f: f
Enter optional basic-auth username (or leave blank):
Enter optional basic-auth password (or leave blank):
Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):
New index name? (Default elastalert_status)
New alias name? (Default elastalert_alerts)
Name of existing index to copy? (Default None)
Elastic Version: 7.4.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
或
0. curl -XPUT http:///elastalert-create-index
1. curl -XPUT http:///elastalert_status_status
2. curl -XPUT http:///elastalert_status_error
3. curl -XPUT http:///elastalert_status_silence
4. curl -XPUT http:///elastalert_status_past
6、申請(qǐng)釘釘機(jī)器人
在釘釘中創(chuàng)建群聊，在群聊的群設(shè)置中的智能群助手中添加機(jī)器人
在本步驟中獲取到webhook地址，并將自己運(yùn)行EA的服務(wù)器的公網(wǎng)IP加入到機(jī)器人安全設(shè)置的IP地址段中。
7、安裝釘釘告警插件
mkdir /home/dingding
cd /home/dingding
wget https://github.com/xuyaoqiang/elastalert-dingtalk-plugin/archive/master.zip
unzip master.zip
cd elastalert-dingtalk-plugin-master/
pip install pyOpenSSL==16.2.0
pip install setuptools==46.1.3
設(shè)置config文件
#vim config.yaml? //修改es的IP和端口，如果ES設(shè)置了認(rèn)證需要設(shè)置用戶名和密碼
es_host: 192.168.25.160
es_port: 9200
設(shè)置rule
以下是查詢數(shù)據(jù)，發(fā)現(xiàn)用戶訪問(wèn)有404即告警
ame: API error (staus >= 400)
type: frequency
index: nginx-prod-*
num_events: 1
timeframe:
? ? minutes: 5
filter:
- term:
? response: "404"
#只需要的字段 https://elastalert.readthedocs.io/en/latest/ruletypes.html#include
include: ["method", "url_path", "url_args", "response", "request_time"]
alert:
- "elastalert_modules.dingtalk_alert.DingTalkAlerter"
dingtalk_webhook: "https://oapi.dingtalk.com/robot/send?access_token=****************"
dingtalk_msgtype: "text"
啟動(dòng)插件
cd /home/dingding/elastalert-dingtalk-plugin-master? ?
python -m elastalert.elastalert --config ./config.yaml --rule ./rules/api_error.yaml? --verbose
8、測(cè)試
訪問(wèn)nginx上一個(gè)不存在地址
以下是告警內(nèi)容
./rules/api_error
At least 1 events occurred between 2021-09-30 22:13 CST and 2021-09-30 22:18 CST
@timestamp: 2021-09-30T14:18:36Z
_id: 2VoSN3wB-MARRt7DZxkg
_index: nginx-prod-2021.09.30
_type: _doc
num_hits: 49
num_matches: 8
response: 404
#將告警中的時(shí)間戳修改成本地時(shí)間，在rule后添加，添加后由于日期格式不一致需要在ES中刪除elastalert_status索引，該索引會(huì)自動(dòng)重建
match_enhancements:
? - "elastalert.enhancements.TimeEnhancement"
elastalert重啟
#cd /root/media/dingding/elastalert-dingtalk-plugin-master
# python3 -m elastalert.elastalert --config ./config.yaml --rule ./rules/api_error.yaml
#nohup python3 -m elastalert.elastalert --verbose > elastalert.log 2>&1 &
釘釘接口測(cè)試python腳本
# filename: dingtalk.py
import requests
import json
import sys
def gaojing(data):
? ? # 將消息提交給釘釘機(jī)器人
? ? headers = {'Content-Type': 'application/json;charset=utf-8'}
? ? # 注意替換釘釘群的機(jī)器人webhook
? ? webhook = "https://oapi.dingtalk.com/robot/send?access_token=fc716b8d501a66345970599e7833c943cb2f66beea96a404d630dcc6945b3930&sign=SEC5b5fc4ddc31ded61bbd07b18e831b960f7be3562ca07e56f6b97e3b36b8ae8d8"
? ? requests.post(url=webhook,data=json.dumps(data),headers=headers)
def get_data(text_content):
? ? # 返回釘釘機(jī)器人所需的文本格式
? ? text = {
? ? ? ? "msgtype": "text",
? ? ? ? "text": {
? ? ? ? ? ? "content": text_content
? ? ? ? },
? ? }
? ? # print(json.dumps(text))
? ? return text
if __name__ == "__main__":
? ? # 命令行第一個(gè)參數(shù)為告警內(nèi)容
? ? text_content = sys.argv[1]
? ? data = get_data(text_content)
? ? gaojing(data)
該腳本需要在命令行中運(yùn)行：python dingtalk hello
安裝參考：
? ? https://blog.csdn.net/hljczm/article/details/109510750
https://github.com/Yelp/elastalert/issues/1675
創(chuàng)建elastalert索引：
? ? https://elastalert.readthedocs.io/en/latest/running_elastalert.html#setting-up-elasticsearch
告警功能擴(kuò)展：
