Alertmanager自定義郵件告警模板

1. Prometheus配置

  • 將告警規(guī)則引入：prometheus.yml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 20s
  external_labels:
    monitor: 172.17.14.5:19093
    replica: 1

rule_files:
  - ./alerts.yaml
  - /opt1/prometheus/alerts_rules.yaml #告警規(guī)則配置文件

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['172.17.14.48:9099']

scrape_configs:
- job_name: 'file_sd'
  file_sd_configs:
  - refresh_interval: 1m
    files: 
    - ./conf.d/*.json
  • 告警規(guī)則配置：/opt1/prometheus/alerts_rules.yaml
groups:
    - name: 主機(jī)狀態(tài)-監(jiān)控告警
      rules:
      - alert: 主機(jī)失聯(lián)
        expr: up == 0
        for: 5m
        labels:
          status: 非常嚴(yán)重
        annotations:
          summary: "{{$labels.instance}}:服務(wù)器宕機(jī)"
          description: "{{$labels.instance}}:服務(wù)器超過5分鐘無法連接"
      
      - alert: CPU使用率過高
        expr: 100-(avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by(instance,company,ownningsystem)* 100) > 90
        for: 30m
        labels:
          status: 一般告警
        annotations:
          summary: "{{$labels.mountpoint}} CPU使用率過高！"
          description: "{{$labels.mountpoint }} CPU使用大于90%(目前使用:{{humanize  $value}}%)"
  
      - alert: 內(nèi)存使用率過高
        expr: 100 -(node_memory_MemTotal_bytes -node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes ) / node_memory_MemTotal_bytes * 100> 85
        for: 30m
        labels:
          status: 嚴(yán)重告警
        annotations:
          summary: "{{$labels.mountpoint}} 內(nèi)存使用率過高！"
          description: "{{$labels.mountpoint }} 內(nèi)存使用大于85%(目前使用:{{humanize $value}}%)"
 
      - alert: 磁盤分區(qū)使用率過高
        expr: 100-(node_filesystem_free_bytes{fstype=~"ext4|xfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs"}*100) > 85
        for: 1m
        labels:
          status: 嚴(yán)重告警
        annotations:
          summary: "{{$labels.mountpoint}} 磁盤分區(qū)使用率過高！"
          description: "{{$labels.mountpoint }} 磁盤分區(qū)使用大于85%(目前使用:{{humanize $value}}%)"

      - alert: 主機(jī)重啟
        expr: sum(time() - node_boot_time_seconds)by(instance,company,ownningsystem)/60/60 <= 12
        for: 1m
        labels:
          status: 非常嚴(yán)重
        annotations:
          summary: "{{$labels.mountpoint}} 主機(jī)重啟"
          description: "{{$labels.mountpoint }} 主機(jī)在過去12h內(nèi)發(fā)生重啟，請確認(rèn)業(yè)務(wù)是否正常！(重啟后已運(yùn)行時(shí)間：{{humanize $value}}h)"
  • 重新加載Prometheus配置
curl -X POST http://localhost:9090/-/reload

2. Alertmanager配置

  • alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'IP：PORT'
  smtp_from: 'abc@qq.com'
  smtp_auth_username: 'abc@qq.com'
  smtp_auth_password: 'passwd'
  smtp_hello: 'qq.com'
  smtp_require_tls: false
  
route:

  # 這里的標(biāo)簽列表是接收到報(bào)警信息后的重新分組標(biāo)簽，例如，接收到的報(bào)警信息里面有許多具有 cluster=A 和 alertname=LatncyHigh 這樣的標(biāo)簽的報(bào)警信息將會批量被聚合到一個(gè)分組里面
  group_by: ['alertname']
  # 當(dāng)一個(gè)新的報(bào)警分組被創(chuàng)建后，需要等待至少group_wait時(shí)間來初始化通知，這種方式可以確保您能有足夠的時(shí)間為同一分組來獲取多個(gè)警報(bào)，然后一起觸發(fā)這個(gè)報(bào)警信息。
  group_wait: 5m
  # 當(dāng)?shù)谝粋€(gè)報(bào)警發(fā)送后，等待'group_interval'時(shí)間來發(fā)送新的一組報(bào)警信息。
  group_interval: 10m
  # 如果一個(gè)報(bào)警信息已經(jīng)發(fā)送成功了，等待'repeat_interval'時(shí)間來重新發(fā)送他們
  repeat_interval: 12h
  # 默認(rèn)的receiver：如果一個(gè)報(bào)警沒有被一個(gè)route匹配，則發(fā)送給默認(rèn)的接收器
  receiver: default 
  # 上面所有的屬性都由所有子路由繼承，并且可以在每個(gè)子路由上進(jìn)行覆蓋

  routes:
  - receiver: receiver1
    match_re:
      company: aaa
  - receiver: receiver1
    group_interval: 1h
    match_re:
      company: aaa
      alertname: 主機(jī)失聯(lián)
  - receiver: receiver2
    match_re:
      company: bbb
      alertname: 主機(jī)失聯(lián)

templates:
  - './templates/*.tmpl'

receivers:
- name: 'receiver1'
  email_configs:
  - to: '11@qq.com'
    send_resolved: true
    headers: { Subject: " 【監(jiān)控告警】 {{ .CommonLabels.alertname }} " } #標(biāo)題
    html: '{{ template "alertsList.html" . }}' #模板

- name: 'receiver2'
  email_configs:
  - to: '12@qq.com,22@qq.com'
    send_resolved: true
    headers: { Subject: " 【監(jiān)控告警】 {{ .CommonLabels.alertname }} " } #標(biāo)題
    html: '{{ template "alertsList.html" . }}' #模板

- name: 'default'
  email_configs:
  - to: 'default@qq.com'
    send_resolved: false
    headers: { Subject: " 【監(jiān)控告警】 {{ .CommonLabels.alertname }} " } #標(biāo)題
    html: '{{ template "alertsList.html" . }}' #模板
  • 郵件模板定義：./templates/emails.tmpl

官方語法介紹：https://prometheus.io/docs/alerting/latest/notifications/
官方模板鏈接：https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl

#########基于alertmanager官方模板修改，內(nèi)容可刪減#########

{{ define "__alertmanager" }}Alertmanager{{ end }}

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__description" }}{{ end }}

{{ define "alertsList.html" }}
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--
Style and HTML derived from https://github.com/mailgun/transactional-email-templates

The MIT License (MIT)

Copyright (c) 2014 Mailgun

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-->
<html xmlns="http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/xhtml" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<head style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
<meta name="viewport" content="width=device-width" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
<title style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">{{ template "__subject" . }}</title>

</head>

<body itemscope="" itemtype="http://schema.org/EmailMessage" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; -webkit-font-smoothing: antialiased; -webkit-text-size-adjust: none; height: 100%; line-height: 1.6em; width: 100% !important; background-color: #f6f6f6; margin: 0; padding: 0;" bgcolor="#f6f6f6">

<table style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; background-color: #f6f6f6; margin: 0;" bgcolor="#f6f6f6">
  <div style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; max-width: 600px; display: block; margin: 0 auto; padding: 0;">
    <table width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; border-radius: 3px; background-color: #fff; margin: 0; border: 1px solid #e9e9e9;" bgcolor="#fff">
      <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
        <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: top; color: #fff; font-weight: 500; text-align: center; border-radius: 3px 3px 0 0; background-color: #E6522C; margin: 0; padding: 20px;" align="center" bgcolor="#E6522C" valign="top">
          發(fā)生 {{ .Alerts | len }}  個(gè) {{ range .GroupLabels.SortedPairs }}
                {{ .Value }}
          {{ end }} 告警 ?。≌埍M快處理
        </td>
      </tr>
      <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
        <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 10px;" valign="top">
          <table border="1" cellpadding="2" cellspacing="0" width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            <tr border="1" cellpadding="2" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            </tr>
          <table border="1" cellpadding="2" cellspacing="0" width="100%" cellpadding="0" cellspacing="0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
            <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>告警名稱</strong> 
              </td>
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>告警級別</strong> 
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>實(shí)例</strong> 
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>所屬系統(tǒng)</strong> 
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>廠商</strong> 
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>觸發(fā)時(shí)間</strong>
              </td>
              <td width="50px" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 16px; vertical-align: middle; margin: 0; padding: 3 3 3px;" valign="top" >
                <strong>說明</strong> 
              </td>
            </tr>
            {{ range .Alerts.Firing }}
            <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
               <!-- {{ .Labels.alertname }} -->
                {{ .Labels.severity }}
              </td>
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
                {{ .Status }}
              </td>
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
                {{ .Labels.instance }}
              </td>
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
                {{ .Labels.ownningsystem }}
              </td>
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
                {{ .Labels.company }}
              </td>
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
                {{  .StartsAt.Format "2006-01-02 15:04:05"  }}
              </td>
              <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 3 3 3px;" valign="top">
                {{ .Annotations.description  }}
              </td>
            </tr>
            {{ end }}
          </table>
        </td>
      </tr>
      </table>
    </table>

    <div style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; width: 100%; clear: both; color: #999; margin: 0; padding: 20px;">
      <table width="100%" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
        <tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
          <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; vertical-align: top; text-align: center; color: #999; margin: 0; padding: 0 0 20px;" align="center" valign="top"><a href="{{ .ExternalURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; color: #999; text-decoration: underline; margin: 0;">Sent by {{ template "__alertmanager" . }}</a></td>
        </tr>
      </table>
    </div></div>
</table>

</body>
</html>
{{ end }}
  • 重新加載alertmanager配置
curl -lv -X POST http://localhost:9099/-/reload

3. 告警結(jié)果

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容