多渠道報警

背景

統(tǒng)一監(jiān)控報警平臺
使用Grafana匯聚了,Prometheus,ELK,CloudWatch監(jiān)控信息.
簡單理解 ELK,AWS均可以實現(xiàn)短信和電話報警

需求

1.Grafana支持短信渠道報警和電話渠道報警.
2.對不同微服務(wù)的開發(fā)團隊發(fā)送報警信息.

要求

1.阿里云短信&電話服務(wù)API
2.Jenkins-plugin:Generic Webhook Trigger Plugin

邏輯

• 阿里云

  • 下載阿里云短信服務(wù)SDK
  • 創(chuàng)建短信服務(wù)ACCESS_KEY_ID

• Grafana

  • 產(chǎn)生報警信息
  • 將報警信息發(fā)送給Jenkins的webhook接口

• Jenkins

  • 創(chuàng)建JOB,執(zhí)行python腳本
  • 設(shè)置JOB的webhook地址
  • 創(chuàng)建電話號碼變量

上圖說明

首先看下阿里云短信接口的python demo

# -*- coding: utf-8 -*-
import sys
from aliyunsdkdysmsapi.request.v20170525 import SendSmsRequest
from aliyunsdkdysmsapi.request.v20170525 import QuerySendDetailsRequest
from aliyunsdkcore.client import AcsClient
import uuid
from aliyunsdkcore.profile import region_provider
from aliyunsdkcore.http import method_type as MT
from aliyunsdkcore.http import format_type as FT
import const
import json

"""
短信業(yè)務(wù)調(diào)用接口示例，版本號：v20170525

Created on 2017-06-12

"""
try:
    reload(sys)
    sys.setdefaultencoding('utf8')
except NameError:
    pass
except Exception as err:
    raise err

# 注意：不要更改
REGION = "cn-hangzhou"
PRODUCT_NAME = "Dysmsapi"
DOMAIN = "dysmsapi.aliyuncs.com"

#Jenkins webhook 變量
PhoneNumbers= sys.argv[1].split(",")
Contents= sys.argv[2]
State= sys.argv[3]

print "PhoneNumbers:%s" % PhoneNumbers
print "Contents: %s" % Contents


acs_client = AcsClient(const.ACCESS_KEY_ID, const.ACCESS_KEY_SECRET, REGION)
region_provider.add_endpoint(PRODUCT_NAME, REGION, DOMAIN)

def send_sms(business_id, phone_numbers, sign_name, template_code, template_param=None):
    smsRequest = SendSmsRequest.SendSmsRequest()
    # 申請的短信模板編碼,必填
    smsRequest.set_TemplateCode(template_code)

    # 短信模板變量參數(shù)
    if template_param is not None:
        smsRequest.set_TemplateParam(template_param)

    # 設(shè)置業(yè)務(wù)請求流水號，必填。
    smsRequest.set_OutId(business_id)

    # 短信簽名
    smsRequest.set_SignName(sign_name)

    # 數(shù)據(jù)提交方式
    # smsRequest.set_method(MT.POST)

    # 數(shù)據(jù)提交格式
    # smsRequest.set_accept_format(FT.JSON)

    # 短信發(fā)送的號碼列表，必填。
    smsRequest.set_PhoneNumbers(phone_numbers)

    # 調(diào)用短信發(fā)送接口，返回json
    smsResponse = acs_client.do_action_with_exception(smsRequest)

    # TODO 業(yè)務(wù)處理

    return smsResponse



if __name__ == '__main__':
    __business_id = uuid.uuid1()
    #print(__business_id)
    params = {"name":State,"contents":Contents}
    params = json.dumps(params)
    print params
    for PhoneNumber in PhoneNumbers:
        print(send_sms(__business_id, PhoneNumber, "玉符", "SMS_142145090", params))

再看下阿里云電話接口的python demo

# -*- coding: utf-8 -*-
from aliyunsdkdyvmsapi.request.v20170525 import SingleCallByTtsRequest
from aliyunsdkdyvmsapi.request.v20170525 import SingleCallByVoiceRequest
from aliyunsdkcore.client import AcsClient
import uuid
import json
import sys
import re

"""
語音業(yè)務(wù)調(diào)用接口示例，版本號：v20170525
Created on 2017-06-12

安裝依賴庫
wget http://ytx-sdk.oss-cn-shanghai.aliyuncs.com/dyvms_python.zip
unzip dyvms_python.zip
sudo python dyvms_python/api_demo/aliyun-python-sdk-core/setup.py install
sudo python dyvms_python/api_demo/aliyun-python-sdk-dyvmsapi/setup.py install
sudo python dyvms_python/api_sdk/aliyun-python-sdk-core/setup.py install
sudo python dyvms_python/api_sdk/aliyun-python-sdk-dyvmsapi/setup.py install


""" 
#暫時不支持多region,默認配置杭州
REGION = "cn-hangzhou"
# ACCESS_KEY_ID/ACCESS_KEY_SECRET 根據(jù)實際申請的賬號信息進行替換
ACCESS_KEY_ID = "xxxxxxx"
ACCESS_KEY_SECRET = "xxxxxx"
#初始化AcsClient
acs_client = AcsClient(ACCESS_KEY_ID, ACCESS_KEY_SECRET, REGION)


def tts_call(business_id, called_number, called_show_number, tts_code, tts_param=None):
    ttsRequest = SingleCallByTtsRequest.SingleCallByTtsRequest()
    # 申請的語音通知tts模板編碼,必填
    ttsRequest.set_TtsCode(tts_code)
    # 設(shè)置業(yè)務(wù)請求流水號，必填。后端服務(wù)基于此標識區(qū)分是否重復請求的判斷
    ttsRequest.set_OutId(business_id)
    # 語音通知的被叫號碼，必填。
    ttsRequest.set_CalledNumber(called_number)
    # 語音通知顯示號碼，必填。
    ttsRequest.set_CalledShowNumber(called_show_number)
    # tts模板變量參數(shù)
    if tts_param is not None:
        ttsRequest.set_TtsParam(tts_param)
    ttsResponse = acs_client.do_action_with_exception(ttsRequest)
    return ttsResponse


__business_id = uuid.uuid1()
print __business_id
#模板中不存在變量的情況下為{}


PhoneNumbers = sys.argv[1].split(",")
Contents = re.sub('[\[\]]', '', sys.argv[2])
State = sys.argv[3]
Appname = sys.argv[4].split("-")[1]
Level = "高級別"
Value = "-"

params = {"level": Level, "Appname": Appname, "Contents": Contents}
params = json.dumps(params)

print Appname, Contents


print "State:%s" % State
if State == "alerting":
    for PhoneNumber in PhoneNumbers:
        print "正在呼叫%s" % PhoneNumber
        print tts_call(__business_id, PhoneNumber,
                       "073182705718", "TTS_148381535", params)

安裝Jenkins插件

訪問jenkins頁面 https://jenkins.$demo.com/pluginManager/available
搜索并安裝插件 Generic Webhook Trigger Plugin
創(chuàng)建Jenkins Job (下圖是幾處關(guān)鍵設(shè)置)

image.png

image.png

短信

image.png

電話

image.png

Jenkins Job創(chuàng)建完成

創(chuàng)建Grafana Notification channels

Url: https://jenkins.$dome.com/generic-webhook-trigger/invoke?token=sms (此處的sms是jenkins中的token)

image.png

完成

在Grafana觸發(fā)報警后會通過這個Channel 發(fā)送一個POST請求到Jenkins.
Jenkins會執(zhí)行對應(yīng)的JOB
job執(zhí)行結(jié)果

image.png