分布式時序計算框架vortex metrics使用介紹

vortex metrics是一款用Java寫的,輕量級的,高性能的分布式時序計算框架。
vortex metrics框架源自作者的另一個分布式流式計算框架vortex,可以看作是vortex關(guān)于時序計算場景的一個具體實現(xiàn)

框架特性:


  1. 去中心化模式管理應(yīng)用集群
  2. 支持應(yīng)用動態(tài)水平擴(kuò)展
  3. 并發(fā)性能極高且穩(wěn)定(網(wǎng)絡(luò)層可適配Netty4,Mina,Grizzly等框架)
  4. CP分布式架構(gòu),支持?jǐn)?shù)據(jù)的最終一致性
  5. 數(shù)據(jù)存儲在內(nèi)存中,訪問快捷。

數(shù)據(jù)存儲:


vortex metrics本身不提供外部存儲(但可以擴(kuò)展去實現(xiàn)),相比另一個時序數(shù)據(jù)庫OpenTSDB,vortex metrics更偏向?qū)崟r計算,將延遲盡可能保持在秒級,但由于是內(nèi)存計算,計算結(jié)果是存儲在內(nèi)存中的,有限的存儲空間讓它并不適用于存儲大量指標(biāo)的業(yè)務(wù)場景。但同時,對于一個指標(biāo)的窗口數(shù)據(jù),若需要保留的時間跨度較長時,vortex metrics提供了定制接口,可以保存到外部存儲,比如數(shù)據(jù)庫或緩存,但這需要開發(fā)人員做一些額外的開發(fā)工作。此外,vortex metrics提供了一個簡單的Web查詢界面來查看效果統(tǒng)計,并且這個Web項目也正在持續(xù)改進(jìn)中。

實際上,vortex metrics的功能是很單一的,因為它只做一件事,那就是根據(jù)客戶端源源不斷的上報數(shù)據(jù)實時地計算并輸出單位時間窗口內(nèi)的統(tǒng)計數(shù)據(jù),記住這點很重要。

安裝


Maven:

<dependency>
    <groupId>com.github.paganini2008.atlantis</groupId>
    <artifactId>vortex-spring-boot-starter</artifactId>
    <version>1.0-RC3</version>
</dependency>

兼容性:

  • Jdk 1.8 (or later)
  • SpringBoot 2.2.0 (or later)
  • Redis 3.0 (or later)
  • MySQL 5.0 (or later)

vortex metrics是一個基于SpringBoot的Web應(yīng)用程序,前面說過,它依賴vortex框架, 而vortex是通過另一個微服務(wù)分布式協(xié)調(diào)框架tridenter來實現(xiàn)集群模式的,并支持動態(tài)水平擴(kuò)展。

參考配置

# 集群配置
spring.application.name=vortex-metrics
spring.application.cluster.name=vortex-metrics-cluster

# vortex配置
atlantis.framework.vortex.bufferzone.collectionName=metric  # 緩沖區(qū)集合名稱
atlantis.framework.vortex.bufferzone.pullSize=1000  # 緩沖區(qū)每次拉取的消息數(shù)量
atlantis.framework.vortex.processor.threads=200  # 數(shù)據(jù)消費者線程數(shù)

服務(wù)的默認(rèn)端口為6150,輸入地址:http://localhost:6150/metric/ 可以看到這個界面

image.png

vortex metrics將server數(shù)據(jù)接收端和監(jiān)控界面做在同一個應(yīng)用中的,高并發(fā)場景下,可能會出現(xiàn)數(shù)據(jù)計算完成并已同步,但界面會滯后更新的情況。

HTTP API說明


vortex metrics目前只提供了2個接口:

  1. 上報數(shù)據(jù)接口:
    POST http://localhost:6150/metrics/sequence/{dataType}
    比如,要上報小汽車的時速,請求體示例:
{
"name": "car",
"metric": "speed",
"value": 120,
"timestamp": 1613200609281
}

參數(shù)說明:

參數(shù)名 類型 描述 舉例
dateType 字符串 數(shù)據(jù)類型,取值范圍:bigint,numeric,bool bigint
name 字符串 應(yīng)用名稱 car
metric 字符串 上報指標(biāo)名稱 speed
value 數(shù)值 上報數(shù)值 120
timestamp 長整型 上報時間戳 1613200609281
  1. 查看效果統(tǒng)計數(shù)據(jù)接口:
    GET http://localhost:6150/metrics/sequence/{dataType}/{name}/{metric}
    參數(shù)同上

這里以上面上報小汽車的時速為例:


image.png

對應(yīng)的響應(yīng)體:

{
    "dataType": "bigint",
    "name": "car",
    "metric": "speed",
    "data": {
        "07:13:00": {
            "speed": {
                "middleValue": 198,
                "count": 2918,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627686839790
            }
        },
        "07:14:00": {
            "speed": {
                "middleValue": 199,
                "count": 10890,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627686899924
            }
        },
        "07:15:00": {
            "speed": {
                "middleValue": 198,
                "count": 10917,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627686959928
            }
        },
        "07:16:00": {
            "speed": {
                "middleValue": 199,
                "count": 10903,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687019591
            }
        },
        "07:17:00": {
            "speed": {
                "middleValue": 199,
                "count": 10841,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687079709
            }
        },
        "07:18:00": {
            "speed": {
                "middleValue": 199,
                "count": 10901,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687139969
            }
        },
        "07:19:00": {
            "speed": {
                "middleValue": 199,
                "count": 10878,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687199267
            }
        },
        "07:20:00": {
            "speed": {
                "middleValue": 199,
                "count": 10887,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687259118
            }
        },
        "07:21:00": {
            "speed": {
                "middleValue": 199,
                "count": 10911,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687319918
            }
        },
        "07:22:00": {
            "speed": {
                "middleValue": 199,
                "count": 10866,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687379834
            }
        },
        "07:23:00": {
            "speed": {
                "middleValue": 199,
                "count": 10910,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687439577
            }
        },
        "07:24:00": {
            "speed": {
                "middleValue": 200,
                "count": 10951,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687499147
            }
        },
        "07:25:00": {
            "speed": {
                "middleValue": 200,
                "count": 10980,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687559897
            }
        },
        "07:26:00": {
            "speed": {
                "middleValue": 199,
                "count": 7811,
                "highestValue": 299,
                "lowestValue": 100,
                "timestamp": 1627687602546
            }
        },
        "07:27:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687620790
            }
        },
        "07:28:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687680790
            }
        },
        "07:29:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687740790
            }
        },
        "07:30:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687800790
            }
        },
        "07:31:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687860790
            }
        },
        "07:32:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687920790
            }
        },
        "07:33:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627687980790
            }
        },
        "07:34:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688040790
            }
        },
        "07:35:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688100790
            }
        },
        "07:36:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688160790
            }
        },
        "07:37:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688220790
            }
        },
        "07:38:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688280790
            }
        },
        "07:39:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688340790
            }
        },
        "07:40:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688400790
            }
        },
        "07:41:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688460790
            }
        },
        "07:42:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688520790
            }
        },
        "07:43:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688580790
            }
        },
        "07:44:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688640790
            }
        },
        "07:45:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688700790
            }
        },
        "07:46:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688760790
            }
        },
        "07:47:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688820790
            }
        },
        "07:48:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688880790
            }
        },
        "07:49:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627688940790
            }
        },
        "07:50:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689000790
            }
        },
        "07:51:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689060790
            }
        },
        "07:52:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689120790
            }
        },
        "07:53:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689180790
            }
        },
        "07:54:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689240790
            }
        },
        "07:55:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689300790
            }
        },
        "07:56:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689360790
            }
        },
        "07:57:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689420790
            }
        },
        "07:58:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689480790
            }
        },
        "07:59:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689540790
            }
        },
        "08:00:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689600790
            }
        },
        "08:01:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689660790
            }
        },
        "08:02:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689720790
            }
        },
        "08:03:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689780790
            }
        },
        "08:04:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689840790
            }
        },
        "08:05:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689900790
            }
        },
        "08:06:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627689960790
            }
        },
        "08:07:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627690020790
            }
        },
        "08:08:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627690080790
            }
        },
        "08:09:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627690140790
            }
        },
        "08:10:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627690200790
            }
        },
        "08:11:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627690260790
            }
        },
        "08:12:00": {
            "speed": {
                "middleValue": 0,
                "count": 0,
                "highestValue": 0,
                "lowestValue": 0,
                "timestamp": 1627690320790
            }
        }
    }
}

可以看到,響應(yīng)體json中的data為時序數(shù)據(jù),vortex metrics默認(rèn)的統(tǒng)計窗口為1分鐘,默認(rèn)滾動保留前60條記錄,也就是用戶可以通過界面看到前60分鐘的統(tǒng)計數(shù)據(jù)。其中,highestValue,lowestValue,middleValue分別代表最大值,最小值,平均值,count是 這1分鐘內(nèi)的數(shù)據(jù)條數(shù),timestamp是時間戳,準(zhǔn)確點講,是這1分鐘內(nèi)最后一條數(shù)據(jù)的時間戳。
說明一下,vortex metrics默認(rèn)的統(tǒng)計窗口,保留前多少條記錄,可以通過配置文件設(shè)置的,但系統(tǒng)需要重啟才能生效。

如何監(jiān)控指標(biāo)


vortex metrics框架為了方便用戶內(nèi)置了3個不同數(shù)據(jù)類型的測試接口:

  1. http://localhost:6150/metrics/test/bigint/{name}/{metric}
  2. http://localhost:6150/metrics/test/numeric/{name}/{metric}
  3. http://localhost:6150/metrics/test/bool/{name}/{metric}
    都是GET方式,設(shè)置name和metric參數(shù)即可

這里還是以統(tǒng)計小汽車的平均速度為例,數(shù)據(jù)上報地址為:
http://localhost:6150/metrics/test/bigint/car/speed

打開壓測工具,這里以JMeter為例

  • 設(shè)置線程組


    image.png
  • 設(shè)置HTTP請求


    image.png

    (關(guān)于Jmeter如何使用,大家可以自行百度,這里僅供測試)

  • 點擊綠色箭頭運行

然后在界面中,location輸入框里查詢接口地址:
http://localhost:6150/metrics/sequence/bigint/car/speed
點擊search, 可以看到:

image.png

vortex metrics集群如何保證數(shù)據(jù)一致性?


vortex metrics集群內(nèi)的應(yīng)用節(jié)點通過內(nèi)部網(wǎng)絡(luò)通訊(默認(rèn)Netty4實現(xiàn))實現(xiàn)相互復(fù)制,保證數(shù)據(jù)的最終一致性,即每個節(jié)點的數(shù)據(jù)最終都是一致的,全量的。
比如現(xiàn)在再起2臺服務(wù),端口分別為6151和6152
這樣的話,
http://localhost:6151/metrics/sequence/bigint/car/speed
http://localhost:6152/metrics/sequence/bigint/car/speed
http://localhost:6150/metrics/sequence/bigint/car/speed
這3個不同地址的接口返回的數(shù)據(jù)是應(yīng)該是最終一致的
localhost:6151:

image.png

localhost:6152
image.png

如圖,應(yīng)用啟動之后加入到vortex metrics集群,應(yīng)用間會互相同步數(shù)據(jù),達(dá)到最終一致的。

然后現(xiàn)在對任一一臺服務(wù)進(jìn)行壓測,比如端口為6152的服務(wù)
觀察localhost:6150地址的接口數(shù)據(jù)變化:


image.png

結(jié)果驗證了vortex metrics集群中應(yīng)用數(shù)據(jù)是雙向同步的。

前面說過,vortex metrics是內(nèi)存計算型的,即計算結(jié)果是駐留在內(nèi)存中的,同步之后,多個應(yīng)該同樣是占據(jù)大量的內(nèi)存,如果存在大量的指標(biāo)數(shù)據(jù),可能會有延遲,所以vortex metrics不適用于存儲大量指標(biāo)的運算場景,但是你可以部署多個vortex metrics集群來解決這個問題,目前經(jīng)壓測,單集群可以較好的支持1000個左右的指標(biāo)數(shù)據(jù)的業(yè)務(wù)場景,高并發(fā)下延遲保持在1~5分鐘內(nèi)。

如何保存歷史數(shù)據(jù)


前面說到,vortex metrics默認(rèn)的統(tǒng)計窗口為1分鐘,滾動保留前60條記錄,那如果你想保留之前的歷史記錄,需要另外做開發(fā)。
首先要實現(xiàn)接口:

public interface MetricEvictionHandler<I, T extends Metric<T>> {

    void onEldestMetricRemoval(I identifier, String metric, T metricUnit);

}

具體可參考:LoggingMetricEvictionHandler類(默認(rèn)實現(xiàn))
你還要實現(xiàn)接口:

public interface MetricSequencerFactory {

    GenericUserMetricSequencer<String, BigInt> getBigIntMetricSequencer();

    GenericUserMetricSequencer<String, Numeric> getNumericMetricSequencer();

    GenericUserMetricSequencer<String, Bool> getBoolMetricSequencer();

}

具體可參考:DefaultMetricSequencerFactory類(默認(rèn)實現(xiàn))
最后,附上vortex metrics的源碼地址:
https://github.com/paganini2008/vortex.git

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點,簡書系信息發(fā)布平臺,僅提供信息存儲服務(wù)。
禁止轉(zhuǎn)載,如需轉(zhuǎn)載請通過簡信或評論聯(lián)系作者。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容