使用 Thanos 實現(xiàn) Prometheus 高可用和監(jiān)控數(shù)據(jù)持久化

thanos 官方主頁：https://thanos.io/

thanos 邏輯架構(gòu)圖

thanos 主要組件

  • sidecar: 和 prometheus 部署在同一節(jié)點，收集 prometheus 的數(shù)據(jù)，并持久化到對象存儲（比如阿里云OSS），同時提供查詢接口給 query 查詢數(shù)據(jù)
  • query: 查詢 sidecar 的數(shù)據(jù)（一般是兩小時內(nèi)的數(shù)據(jù)，可配置）和 store gateway 的數(shù)據(jù)（一般是超過兩小時的持久化數(shù)據(jù)，可配置）
  • store gateway: 提供查詢接口給 query 來查詢在對象存儲（比如阿里云OSS）的持久化數(shù)據(jù)

thanos 系統(tǒng)部署架構(gòu)圖

示例環(huán)境

  • 節(jié)點1（CentOS 7.x / 172.18.255.129 / 119.23.40.213）: promethues + thanos sidecar + thanos store gateway + thanos query
  • 節(jié)點2（CentOS 7.x / 172.18.255.130 / 120.79.71.229）: promethues + thanos sidecar + thanos store gateway + thanos query

部署過程

1、節(jié)點1（172.18.255.129/119.23.40.213）部署 prometheus

# cd /usr/local/src/
# wget https://github.com/prometheus/prometheus/releases/download/v2.28.0/prometheus-2.28.0.linux-amd64.tar.gz
# tar xvf prometheus-2.28.0.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s prometheus-2.28.0.linux-amd64/ prometheus
# cd prometheus

// 這里需要聲明 external_labels，給 prometheus 打上標(biāo)簽
// prometheus 高可用集群的第 1 個節(jié)點，region 設(shè)置為 cn-shenzhen，replica 設(shè)置為 1
# vim prometheus.yml 
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
  external_labels:
    region: cn-shenzhen
    monitor: infrastructure
    replica: 1

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']


// --web.external-url= 修改為對應(yīng)節(jié)點的對外訪問 IP
# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/usr/local/prometheus/data/ --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h --storage.tsdb.retention.time=2h --storage.tsdb.wal-compression --web.enable-lifecycle --log.level=info --web.listen-address=0.0.0.0:9090 --web.external-url=http://119.23.40.213:9090

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload && systemctl enable prometheus && systemctl start prometheus && systemctl status prometheus

2、節(jié)點2（172.18.255.130/120.79.71.229）部署 prometheus

# cd /usr/local/src/
# wget https://github.com/prometheus/prometheus/releases/download/v2.28.0/prometheus-2.28.0.linux-amd64.tar.gz
# tar xvf prometheus-2.28.0.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s prometheus-2.28.0.linux-amd64/ prometheus
# cd prometheus

// 這里需要聲明 external_labels，給 prometheus 打上標(biāo)簽
// prometheus 高可用集群的第 2 個節(jié)點，region 設(shè)置為 cn-shanghai，replica 設(shè)置為 2。
# vim prometheus.yml 
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
  external_labels:
    region: cn-shanghai
    monitor: infrastructure
    replica: 2

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

// --web.external-url= 修改為對應(yīng)節(jié)點的對外訪問 IP
# vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/usr/local/prometheus/data/ --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h --storage.tsdb.retention.time=2h --storage.tsdb.wal-compression --web.enable-lifecycle --log.level=info --web.listen-address=0.0.0.0:9090 --web.external-url=http://120.79.71.229:9090

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload && systemctl enable prometheus && systemctl start prometheus && systemctl status prometheus

3、節(jié)點1（172.18.255.129/119.23.40.213）部署 thanos，并配置組件 thanos sidecar 和 thanos store gateway

# cd /usr/local/src/
# wget https://github.com/thanos-io/thanos/releases/download/v0.21.1/thanos-0.21.1.linux-amd64.tar.gz
# tar xvf thanos-0.21.1.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s thanos-0.21.1.linux-amd64/ thanos
# cd thanos

// 配置 thanos sidecar，實現(xiàn)收集 promethues 高可用集群的第 1 個節(jié)點的監(jiān)控數(shù)據(jù)，并持久化到阿里云 oss
// 阿里云 oss bucket 請自行開通并配置好 access_key_id 和 access_key_secret，后面的步驟都是使用這個 bucket
# vim aliyun-oss.yaml
type: ALIYUNOSS
config:
  endpoint: "oss-cn-shenzhen-internal.aliyuncs.com"
  bucket: "prometheus-persistent-data"
  access_key_id: "XXX"
  access_key_secret: "XXX"

# vim /etc/systemd/system/thanos-sidecar.service
[Unit]
Description=Thanos Sidecar
After=network-online.target

[Service]
Restart=on-failure
# --prometheus.url=http://localhost:9090 --> 指定本節(jié)點的 prometheus 地址
# --tsdb.path=/usr/local/prometheus/data/ --> 指定本節(jié)點的 prometheus 數(shù)據(jù)目錄
ExecStart=/usr/local/thanos/thanos sidecar --prometheus.url=http://localhost:9090 --tsdb.path=/usr/local/prometheus/data/ --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --grpc-address=0.0.0.0:19090 --http-address=0.0.0.0:19091

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload && systemctl enable thanos-sidecar && systemctl start thanos-sidecar && systemctl status thanos-sidecar


// 配置 thanos store gateway，實現(xiàn)讀取對象存儲（比如阿里云OSS）的數(shù)據(jù)，并提供給 thanos query 查詢
# vim /etc/systemd/system/thanos-store-gateway.service
[Unit]
Description=Thanos Store Gateway
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/thanos/thanos store --data-dir=/usr/local/thanos/data --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --http-address=0.0.0.0:10902 --grpc-address=0.0.0.0:10901

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload && systemctl enable thanos-store-gateway && systemctl start thanos-store-gateway && systemctl status thanos-store-gateway

4、節(jié)點2（172.18.255.130/120.79.71.229）部署 thanos，并配置組件 thanos sidecar 和 thanos store gateway

// 安裝 thanos
# cd /usr/local/src/
# wget https://github.com/thanos-io/thanos/releases/download/v0.21.1/thanos-0.21.1.linux-amd64.tar.gz
# tar xvf thanos-0.21.1.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# ln -s thanos-0.21.1.linux-amd64/ thanos
# cd thanos


// 配置 thanos sidecar，實現(xiàn)收集 promethues 高可用集群的第 2 個節(jié)點的監(jiān)控數(shù)據(jù)，并持久化到阿里云 oss
# vim aliyun-oss.yaml
type: ALIYUNOSS
config:
  endpoint: "oss-cn-shenzhen-internal.aliyuncs.com"
  bucket: "prometheus-persistent-data"
  access_key_id: "XXX"
  access_key_secret: "XXX"

# vim /etc/systemd/system/thanos-sidecar.service
[Unit]
Description=Thanos Sidecar
After=network-online.target

[Service]
Restart=on-failure
# --prometheus.url=http://localhost:9090 --> 指定本節(jié)點的 prometheus 地址
# --tsdb.path=/usr/local/prometheus/data/ --> 指定本節(jié)點的 prometheus 數(shù)據(jù)目錄
ExecStart=/usr/local/thanos/thanos sidecar --prometheus.url=http://localhost:9090 --tsdb.path=/usr/local/prometheus/data/ --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --grpc-address=0.0.0.0:19090 --http-address=0.0.0.0:19091

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload && systemctl enable thanos-sidecar && systemctl start thanos-sidecar && systemctl status thanos-sidecar


// 配置 thanos store gateway，實現(xiàn)讀取對象存儲（比如阿里云OSS）的數(shù)據(jù)，并提供給 thanos query 查詢
# vim /etc/systemd/system/thanos-store-gateway.service
[Unit]
Description=Thanos Store Gateway
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/local/thanos/thanos store --data-dir=/usr/local/thanos/data --objstore.config-file=/usr/local/thanos/aliyun-oss.yaml --http-address=0.0.0.0:10902 --grpc-address=0.0.0.0:10901

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload && systemctl enable thanos-store-gateway && systemctl start thanos-store-gateway && systemctl status thanos-store-gateway

5、節(jié)點1（172.18.255.129/119.23.40.213）配置組件 thanos-query，查詢節(jié)點 1 和節(jié)點 2 的 sidecar、store gateway 數(shù)據(jù)

# vim /etc/systemd/system/thanos-query.service
[Unit]
Description=Thanos Query
After=network-online.target

[Service]
Restart=on-failure
# --store=172.18.255.129:19090 --> 請求當(dāng)前節(jié)點 thanos sidecar 數(shù)據(jù)
# --store=172.18.255.130:19090 --> 請求另外一個節(jié)點 thanos sidecar 數(shù)據(jù)

# --store=172.18.255.129:10901 --> 請求當(dāng)前節(jié)點 thanos store gateway 數(shù)據(jù)
# --store=172.18.255.130:10901 --> 請求另外一個節(jié)點 thanos store gateway 數(shù)據(jù)
# --query.replica-label "replica" --query.replica-label "region" --> 加上后，thanos query 查詢同一節(jié)點的數(shù)據(jù)時，會自動去重
ExecStart=/usr/local/thanos/thanos query --http-address=0.0.0.0:19192 --grpc-address 0.0.0.0:19092 --store=172.18.255.129:19090 --store=172.18.255.130:19090 --store=172.18.255.129:10901 --store=172.18.255.130:10901 --query.replica-label "replica" --query.replica-label "region"

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload && systemctl enable thanos-query && systemctl start thanos-query && systemctl status thanos-query

實現(xiàn)的效果：


6、節(jié)點2（172.18.255.130/120.79.71.229）配置組件 thanos-query，查詢節(jié)點 1 和節(jié)點 2 的 sidecar、store gateway 數(shù)據(jù)

# vim /etc/systemd/system/thanos-query.service
[Unit]
Description=Thanos Query
After=network-online.target

[Service]
Restart=on-failure
# --store=172.18.255.130:19090 --> 請求當(dāng)前節(jié)點 thanos sidecar 數(shù)據(jù)
# --store=172.18.255.129:19090 --> 請求另外一個節(jié)點 thanos sidecar 數(shù)據(jù)

# --store=172.18.255.130:10901 --> 請求當(dāng)前節(jié)點 thanos store gateway 數(shù)據(jù)
# --store=172.18.255.129:10901 --> 請求另外一個節(jié)點 thanos store gateway 數(shù)據(jù)
# --query.replica-label "replica" --query.replica-label "region" --> 加上后，thanos query 查詢同一節(jié)點的數(shù)據(jù)時，會自動去重
ExecStart=/usr/local/thanos/thanos query --http-address=0.0.0.0:19192 --grpc-address 0.0.0.0:19092 --store=172.18.255.130:19090 --store=172.18.255.129:19090 --store=172.18.255.130:10901 --store=172.18.255.129:10901 --query.replica-label "replica" --query.replica-label "region"

[Install]
WantedBy=multi-user.target


# systemctl daemon-reload && systemctl enable thanos-query && systemctl start thanos-query && systemctl status thanos-query

實現(xiàn)的效果：


7、在負(fù)載均衡服務(wù)（比如 HAProxy、阿里云 SLB 等）配置兩個節(jié)點的 thanos query ，就可以實現(xiàn) thanso + prometheus 的高可用了，后續(xù)的查詢操作都通過 thanos query 進(jìn)行

其他

1、prometheus 的配置參數(shù) external_labels 和 thanos query 啟動參數(shù) --query.replica-label "replica" --query.replica-label "region" 的作用

2、thanos 只會保留其中一個節(jié)點上傳的持久化監(jiān)控數(shù)據(jù)（根據(jù)配置，要 2 小時后才會上傳到阿里云 OSS）


3、除了實現(xiàn) prometheus 高可用場景，thanos 也可以用作多個 prometheus 的匯總查詢?nèi)肟?，只需部?thanos sidecar 和 thanos store gateway 到相應(yīng)的 prometheus 節(jié)點，然后 thanos query 通過參數(shù) --store=ip:port 連接 thanos sidecar 和 thanos store gateway 即可

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容