vortex metrics是一款用Java寫的,輕量級的,高性能的分布式時序計算框架。
vortex metrics框架源自作者的另一個分布式流式計算框架vortex,可以看作是vortex關(guān)于時序計算場景的一個具體實現(xiàn)
框架特性:
- 去中心化模式管理應(yīng)用集群
- 支持應(yīng)用動態(tài)水平擴(kuò)展
- 并發(fā)性能極高且穩(wěn)定(網(wǎng)絡(luò)層可適配Netty4,Mina,Grizzly等框架)
- CP分布式架構(gòu),支持?jǐn)?shù)據(jù)的最終一致性
- 數(shù)據(jù)存儲在內(nèi)存中,訪問快捷。
數(shù)據(jù)存儲:
vortex metrics本身不提供外部存儲(但可以擴(kuò)展去實現(xiàn)),相比另一個時序數(shù)據(jù)庫OpenTSDB,vortex metrics更偏向?qū)崟r計算,將延遲盡可能保持在秒級,但由于是內(nèi)存計算,計算結(jié)果是存儲在內(nèi)存中的,有限的存儲空間讓它并不適用于存儲大量指標(biāo)的業(yè)務(wù)場景。但同時,對于一個指標(biāo)的窗口數(shù)據(jù),若需要保留的時間跨度較長時,vortex metrics提供了定制接口,可以保存到外部存儲,比如數(shù)據(jù)庫或緩存,但這需要開發(fā)人員做一些額外的開發(fā)工作。此外,vortex metrics提供了一個簡單的Web查詢界面來查看效果統(tǒng)計,并且這個Web項目也正在持續(xù)改進(jìn)中。
實際上,vortex metrics的功能是很單一的,因為它只做一件事,那就是根據(jù)客戶端源源不斷的上報數(shù)據(jù)實時地計算并輸出單位時間窗口內(nèi)的統(tǒng)計數(shù)據(jù),記住這點很重要。
安裝
Maven:
<dependency>
<groupId>com.github.paganini2008.atlantis</groupId>
<artifactId>vortex-spring-boot-starter</artifactId>
<version>1.0-RC3</version>
</dependency>
兼容性:
- Jdk 1.8 (or later)
- SpringBoot 2.2.0 (or later)
- Redis 3.0 (or later)
- MySQL 5.0 (or later)
vortex metrics是一個基于SpringBoot的Web應(yīng)用程序,前面說過,它依賴vortex框架, 而vortex是通過另一個微服務(wù)分布式協(xié)調(diào)框架tridenter來實現(xiàn)集群模式的,并支持動態(tài)水平擴(kuò)展。
參考配置
# 集群配置
spring.application.name=vortex-metrics
spring.application.cluster.name=vortex-metrics-cluster
# vortex配置
atlantis.framework.vortex.bufferzone.collectionName=metric # 緩沖區(qū)集合名稱
atlantis.framework.vortex.bufferzone.pullSize=1000 # 緩沖區(qū)每次拉取的消息數(shù)量
atlantis.framework.vortex.processor.threads=200 # 數(shù)據(jù)消費者線程數(shù)
服務(wù)的默認(rèn)端口為6150,輸入地址:http://localhost:6150/metric/ 可以看到這個界面

vortex metrics將server數(shù)據(jù)接收端和監(jiān)控界面做在同一個應(yīng)用中的,高并發(fā)場景下,可能會出現(xiàn)數(shù)據(jù)計算完成并已同步,但界面會滯后更新的情況。
HTTP API說明
vortex metrics目前只提供了2個接口:
- 上報數(shù)據(jù)接口:
POST http://localhost:6150/metrics/sequence/{dataType}
比如,要上報小汽車的時速,請求體示例:
{
"name": "car",
"metric": "speed",
"value": 120,
"timestamp": 1613200609281
}
參數(shù)說明:
| 參數(shù)名 | 類型 | 描述 | 舉例 |
|---|---|---|---|
| dateType | 字符串 | 數(shù)據(jù)類型,取值范圍:bigint,numeric,bool | bigint |
| name | 字符串 | 應(yīng)用名稱 | car |
| metric | 字符串 | 上報指標(biāo)名稱 | speed |
| value | 數(shù)值 | 上報數(shù)值 | 120 |
| timestamp | 長整型 | 上報時間戳 | 1613200609281 |
- 查看效果統(tǒng)計數(shù)據(jù)接口:
GET http://localhost:6150/metrics/sequence/{dataType}/{name}/{metric}
參數(shù)同上
這里以上面上報小汽車的時速為例:

對應(yīng)的響應(yīng)體:
{
"dataType": "bigint",
"name": "car",
"metric": "speed",
"data": {
"07:13:00": {
"speed": {
"middleValue": 198,
"count": 2918,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627686839790
}
},
"07:14:00": {
"speed": {
"middleValue": 199,
"count": 10890,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627686899924
}
},
"07:15:00": {
"speed": {
"middleValue": 198,
"count": 10917,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627686959928
}
},
"07:16:00": {
"speed": {
"middleValue": 199,
"count": 10903,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687019591
}
},
"07:17:00": {
"speed": {
"middleValue": 199,
"count": 10841,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687079709
}
},
"07:18:00": {
"speed": {
"middleValue": 199,
"count": 10901,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687139969
}
},
"07:19:00": {
"speed": {
"middleValue": 199,
"count": 10878,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687199267
}
},
"07:20:00": {
"speed": {
"middleValue": 199,
"count": 10887,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687259118
}
},
"07:21:00": {
"speed": {
"middleValue": 199,
"count": 10911,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687319918
}
},
"07:22:00": {
"speed": {
"middleValue": 199,
"count": 10866,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687379834
}
},
"07:23:00": {
"speed": {
"middleValue": 199,
"count": 10910,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687439577
}
},
"07:24:00": {
"speed": {
"middleValue": 200,
"count": 10951,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687499147
}
},
"07:25:00": {
"speed": {
"middleValue": 200,
"count": 10980,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687559897
}
},
"07:26:00": {
"speed": {
"middleValue": 199,
"count": 7811,
"highestValue": 299,
"lowestValue": 100,
"timestamp": 1627687602546
}
},
"07:27:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687620790
}
},
"07:28:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687680790
}
},
"07:29:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687740790
}
},
"07:30:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687800790
}
},
"07:31:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687860790
}
},
"07:32:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687920790
}
},
"07:33:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627687980790
}
},
"07:34:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688040790
}
},
"07:35:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688100790
}
},
"07:36:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688160790
}
},
"07:37:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688220790
}
},
"07:38:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688280790
}
},
"07:39:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688340790
}
},
"07:40:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688400790
}
},
"07:41:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688460790
}
},
"07:42:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688520790
}
},
"07:43:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688580790
}
},
"07:44:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688640790
}
},
"07:45:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688700790
}
},
"07:46:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688760790
}
},
"07:47:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688820790
}
},
"07:48:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688880790
}
},
"07:49:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627688940790
}
},
"07:50:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689000790
}
},
"07:51:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689060790
}
},
"07:52:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689120790
}
},
"07:53:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689180790
}
},
"07:54:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689240790
}
},
"07:55:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689300790
}
},
"07:56:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689360790
}
},
"07:57:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689420790
}
},
"07:58:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689480790
}
},
"07:59:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689540790
}
},
"08:00:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689600790
}
},
"08:01:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689660790
}
},
"08:02:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689720790
}
},
"08:03:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689780790
}
},
"08:04:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689840790
}
},
"08:05:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689900790
}
},
"08:06:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627689960790
}
},
"08:07:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627690020790
}
},
"08:08:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627690080790
}
},
"08:09:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627690140790
}
},
"08:10:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627690200790
}
},
"08:11:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627690260790
}
},
"08:12:00": {
"speed": {
"middleValue": 0,
"count": 0,
"highestValue": 0,
"lowestValue": 0,
"timestamp": 1627690320790
}
}
}
}
可以看到,響應(yīng)體json中的data為時序數(shù)據(jù),vortex metrics默認(rèn)的統(tǒng)計窗口為1分鐘,默認(rèn)滾動保留前60條記錄,也就是用戶可以通過界面看到前60分鐘的統(tǒng)計數(shù)據(jù)。其中,highestValue,lowestValue,middleValue分別代表最大值,最小值,平均值,count是 這1分鐘內(nèi)的數(shù)據(jù)條數(shù),timestamp是時間戳,準(zhǔn)確點講,是這1分鐘內(nèi)最后一條數(shù)據(jù)的時間戳。
說明一下,vortex metrics默認(rèn)的統(tǒng)計窗口,保留前多少條記錄,可以通過配置文件設(shè)置的,但系統(tǒng)需要重啟才能生效。
如何監(jiān)控指標(biāo)
vortex metrics框架為了方便用戶內(nèi)置了3個不同數(shù)據(jù)類型的測試接口:
- http://localhost:6150/metrics/test/bigint/{name}/{metric}
- http://localhost:6150/metrics/test/numeric/{name}/{metric}
-
http://localhost:6150/metrics/test/bool/{name}/{metric}
都是GET方式,設(shè)置name和metric參數(shù)即可
這里還是以統(tǒng)計小汽車的平均速度為例,數(shù)據(jù)上報地址為:
http://localhost:6150/metrics/test/bigint/car/speed
打開壓測工具,這里以JMeter為例
-
設(shè)置線程組
image.png -
設(shè)置HTTP請求
image.png
(關(guān)于Jmeter如何使用,大家可以自行百度,這里僅供測試)
- 點擊綠色箭頭運行
然后在界面中,location輸入框里查詢接口地址:
http://localhost:6150/metrics/sequence/bigint/car/speed
點擊search, 可以看到:

vortex metrics集群如何保證數(shù)據(jù)一致性?
vortex metrics集群內(nèi)的應(yīng)用節(jié)點通過內(nèi)部網(wǎng)絡(luò)通訊(默認(rèn)Netty4實現(xiàn))實現(xiàn)相互復(fù)制,保證數(shù)據(jù)的最終一致性,即每個節(jié)點的數(shù)據(jù)最終都是一致的,全量的。
比如現(xiàn)在再起2臺服務(wù),端口分別為6151和6152
這樣的話,
http://localhost:6151/metrics/sequence/bigint/car/speed
http://localhost:6152/metrics/sequence/bigint/car/speed
http://localhost:6150/metrics/sequence/bigint/car/speed
這3個不同地址的接口返回的數(shù)據(jù)是應(yīng)該是最終一致的
localhost:6151:

localhost:6152

如圖,應(yīng)用啟動之后加入到vortex metrics集群,應(yīng)用間會互相同步數(shù)據(jù),達(dá)到最終一致的。
然后現(xiàn)在對任一一臺服務(wù)進(jìn)行壓測,比如端口為6152的服務(wù)
觀察localhost:6150地址的接口數(shù)據(jù)變化:

結(jié)果驗證了vortex metrics集群中應(yīng)用數(shù)據(jù)是雙向同步的。
前面說過,vortex metrics是內(nèi)存計算型的,即計算結(jié)果是駐留在內(nèi)存中的,同步之后,多個應(yīng)該同樣是占據(jù)大量的內(nèi)存,如果存在大量的指標(biāo)數(shù)據(jù),可能會有延遲,所以vortex metrics不適用于存儲大量指標(biāo)的運算場景,但是你可以部署多個vortex metrics集群來解決這個問題,目前經(jīng)壓測,單集群可以較好的支持1000個左右的指標(biāo)數(shù)據(jù)的業(yè)務(wù)場景,高并發(fā)下延遲保持在1~5分鐘內(nèi)。
如何保存歷史數(shù)據(jù)
前面說到,vortex metrics默認(rèn)的統(tǒng)計窗口為1分鐘,滾動保留前60條記錄,那如果你想保留之前的歷史記錄,需要另外做開發(fā)。
首先要實現(xiàn)接口:
public interface MetricEvictionHandler<I, T extends Metric<T>> {
void onEldestMetricRemoval(I identifier, String metric, T metricUnit);
}
具體可參考:LoggingMetricEvictionHandler類(默認(rèn)實現(xiàn))
你還要實現(xiàn)接口:
public interface MetricSequencerFactory {
GenericUserMetricSequencer<String, BigInt> getBigIntMetricSequencer();
GenericUserMetricSequencer<String, Numeric> getNumericMetricSequencer();
GenericUserMetricSequencer<String, Bool> getBoolMetricSequencer();
}
具體可參考:DefaultMetricSequencerFactory類(默認(rèn)實現(xiàn))
最后,附上vortex metrics的源碼地址:
https://github.com/paganini2008/vortex.git

