官方文檔

下載最新的版本

wget https://github.com/prometheus/prometheus/releases/download/v2.19.2/prometheus-2.19.2.linux-amd64.tar.gz
tar xzvf prometheus-2.19.2.linux-amd64.tar.gz
cd prometheus-2.19.2.linux-amd64

配置

[sysadmin@VM_201_13_centos prometheus-2.19.2.linux-amd64]$ ls 
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool  tsdb

Prometheus 從被監(jiān)控目標(biāo)設(shè)備通過檢索 metrics HTTP endpoints 收集metrics。因?yàn)镻rometheus自身也以同樣的方式發(fā)布數(shù)據(jù)，所以它也可以監(jiān)控自身的健康狀態(tài)。

雖然Prometheus server僅采集自身的數(shù)據(jù)沒什么意義，但是作為開始的范例很合適。

將下面的內(nèi)容保存為 Prometheus 配置文件prometheus.yml：
實(shí)際上安裝包內(nèi)已經(jīng)包含了該文件，無需修改

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

查看其他配置項(xiàng)，請參閱 configuration documentation。

啟動 Prometheus

[sysadmin@VM_201_13_centos prometheus-2.19.2.linux-amd64]$ ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool  tsdb

[sysadmin@VM_201_13_centos prometheus-2.19.2.linux-amd64]$ ./prometheus --config.file=prometheus.yml
level=info ts=2020-07-02T12:28:39.465Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2020-07-02T12:28:39.465Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.2, branch=HEAD, revision=c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)"
level=info ts=2020-07-02T12:28:39.465Z caller=main.go:338 build_context="(go=go1.14.4, user=root@dd72efe1549d, date=20200626-09:02:20)"
level=info ts=2020-07-02T12:28:39.465Z caller=main.go:339 host_details="(Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 VM_201_13_centos (none))"
level=info ts=2020-07-02T12:28:39.465Z caller=main.go:340 fd_limits="(soft=100001, hard=100002)"
level=info ts=2020-07-02T12:28:39.465Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-07-02T12:28:39.468Z caller=main.go:678 msg="Starting TSDB ..."
level=info ts=2020-07-02T12:28:39.471Z caller=head.go:645 component=tsdb msg="Replaying WAL and on-disk memory mappable chunks if any, this may take a while"
level=info ts=2020-07-02T12:28:39.471Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-07-02T12:28:39.472Z caller=head.go:706 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2020-07-02T12:28:39.472Z caller=head.go:709 component=tsdb msg="WAL replay completed" duration=975.63μs
level=info ts=2020-07-02T12:28:39.473Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-07-02T12:28:39.473Z caller=main.go:695 msg="TSDB started"
level=info ts=2020-07-02T12:28:39.473Z caller=main.go:799 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2020-07-02T12:28:39.474Z caller=main.go:827 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2020-07-02T12:28:39.474Z caller=main.go:646 msg="Server is ready to receive web requests."

新開一個bash shell，查看服務(wù)和端口

[sysadmin@VM_201_13_centos ~]$ netstat -anpl |grep prometheus
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:44570         127.0.0.1:9090          ESTABLISHED 8214/./prometheus   
tcp6       0      0 :::9090                 :::*                    LISTEN      8214/./prometheus   
tcp6       0      0 ::1:58018               ::1:9090                ESTABLISHED 8214/./prometheus   
tcp6       0      0 127.0.0.1:9090          127.0.0.1:44570         ESTABLISHED 8214/./prometheus   
tcp6       0      0 ::1:9090                ::1:58018               ESTABLISHED 8214/./prometheus   

訪問測試 http://ip:9090/

微信截圖_20200702203423.png

訪問測試 http://ip:9090/metrics/

微信截圖_20200702203558.png

Using the expression browser

我們試試查看Prometheus 收集的自身的數(shù)據(jù)。要使用 Prometheus 自帶的 expression browser，導(dǎo)航到 http://localhost:9090/graph ，選擇"Graph" tab內(nèi)的 "Console" view 。

既然你可以從localhost:9090/metrics查看數(shù)據(jù)，那么可以看到一個Prometheus自己發(fā)布的指標(biāo) prometheus_target_interval_length_seconds (目標(biāo)采集的實(shí)際間隔實(shí)際)。輸入到expression console，然后點(diǎn)擊 "Execute":

prometheus_target_interval_length_seconds

微信截圖_20200702204243.png

如上，返回了一些不同的時(shí)間序列 (along with the latest value recorded for each), 都是 prometheus_target_interval_length_seconds的metric name，但是擁有不同的標(biāo)簽。這些標(biāo)簽顯示了不同的時(shí)間段( latency percentiles)和 目標(biāo)組間隔(target group intervals)。

如果我們只對 99th percentile latencies有興趣，我們可以通過如下查詢：

prometheus_target_interval_length_seconds{quantile="0.99"}

微信截圖_20200702211919.png

要統(tǒng)計(jì)返回的時(shí)間序列數(shù)量，可以查詢?nèi)缦拢?/p>

count(prometheus_target_interval_length_seconds)

微信截圖_20200702212022.png

更多表達(dá)式語言參見 expression language documentation.

Using the graphing interface

要圖形表示，導(dǎo)航到 http://localhost:9090/graph ，點(diǎn)擊 "Graph" tab。

比如，使用如下查詢，圖形表示demo Prometheus的per-second rate of chunks ：

rate(prometheus_tsdb_head_chunks_created_total[1m])

微信截圖_20200702212257.png

Experiment with the graph range parameters and other settings.

Starting up some sample targets

開始接入幾個sample targets來演示。

Node Exporter用來作為范例，怎么使用參見 see these instructions.

tar -xzvf node_exporter-*.*.tar.gz
cd node_exporter-*.*

# Start 3 example targets in separate terminals:
./node_exporter --web.listen-address 0.0.0.0:8080
./node_exporter --web.listen-address 0.0.0.0:8081
./node_exporter --web.listen-address 0.0.0.0:8082

范例監(jiān)聽在 http://ip:8080/metrics, http://ip:8081/metrics, and http://ip:8082/metrics。

Configure Prometheus to monitor the sample targets

現(xiàn)在我們配置 Prometheus 來采集這些對象。 我們把三個endpoints組成一個group到j(luò)ob內(nèi)，稱為node。 假設(shè)，前兩個是生產(chǎn)，第三個是金絲雀實(shí)例。為了在Prometheus區(qū)分，我們增加了數(shù)個endpoint組，給每個組增加標(biāo)簽。在范例中，我們將增加 group="production" 標(biāo)簽給第一個組， group="canary" 給第二個。

要實(shí)現(xiàn)這些，在prometheus.yml文件添加如下job定義到scrape_configs section，然后重啟Prometheus 實(shí)例。

scrape_configs:
  - job_name:       'node'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']
        labels:
          group: 'production'

      - targets: ['localhost:8082']
        labels:
          group: 'canary'

回到 expression browser ，確認(rèn) Prometheus 現(xiàn)在已經(jīng)有了關(guān)于這三個范例的信息，比如查詢

node_cpu_seconds_total

QQ截圖20200703100002.png

Configure rules for aggregating scraped data into new time series

雖然在范例中沒有這個問題，但是當(dāng)computed ad-hoc ，跨成千上萬時(shí)間序列的查詢會變慢。為提升效率， Prometheus 允許通過配置recording rules 將預(yù)記錄的表達(dá)式插入到新時(shí)間序列中。 假設(shè)我們需要5分鐘時(shí)間窗口(job, instance and mode dimensions)的平均值 per-second rate of cpu time (node_cpu_seconds_total) 。我們可以查詢?nèi)缦拢?/p>

avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))

實(shí)施圖形展示。

要將這個表達(dá)式的值放到一個新metric job_instance_mode:node_cpu_seconds:avg_rate5m，使用下面的recording rule 保存為 prometheus.rules.yml:

groups:
- name: cpu-node
  rules:
  - record: job_instance_mode:node_cpu_seconds:avg_rate5m
    expr: avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))

要讓 Prometheus 使用這個 rule，在 prometheus.yml 添加 rule_files 聲明部分。 如下是范例：

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

  # Attach these extra labels to all timeseries collected by this Prometheus instance.
  external_labels:
    monitor: 'codelab-monitor'

rule_files:
  - 'prometheus.rules.yml'

scrape_configs:
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

  - job_name:       'node'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']
        labels:
          group: 'production'

      - targets: ['localhost:8082']
        labels:
          group: 'canary'

使用新配置文件重啟 Prometheus ，確認(rèn)一個新指標(biāo)job_instance_mode:node_cpu_seconds:avg_rate5m可以通過expression browser查詢了。

QQ截圖20200703110955.png