當(dāng)網(wǎng)關(guān)和服務(wù)在實(shí)施全鏈路分布式灰度發(fā)布和路由時(shí)候,我們需要一款追蹤系統(tǒng)來(lái)監(jiān)控網(wǎng)關(guān)和服務(wù)走的是哪個(gè)灰度組,哪個(gè)灰度版本,哪個(gè)灰度區(qū)域,甚至監(jiān)控從Http Header頭部全程傳遞的灰度規(guī)則和路由策略。這個(gè)功能意義在于:
- 不僅可以監(jiān)控全鏈路中基本的調(diào)用信息,也可以監(jiān)控額外的灰度信息,有助于我們判斷灰度發(fā)布和路由是否執(zhí)行準(zhǔn)確,一旦有問(wèn)題,也可以快速定位
- 可以監(jiān)控流量何時(shí)切換到新版本,或者新的區(qū)域,或者新的機(jī)器上
- 可以監(jiān)控灰度規(guī)則和路由策略是否配置準(zhǔn)確
- 可以監(jiān)控網(wǎng)關(guān)和服務(wù)灰度上下級(jí)樹狀關(guān)系
- 可以監(jiān)控全鏈路流量拓?fù)鋱D
筆者嘗試調(diào)研了一系列分布式追蹤系統(tǒng)和中間件,包括Opentracing、Uber Jaeger、Twitter Zipkin、Apache Skywalking、Pinpoint、CAT等,最后決定采用Opentracing + Uber Jaeger方式來(lái)實(shí)現(xiàn),重要原因除了易用性和可擴(kuò)展性外,Opentracing支持WebMvc和WebFlux兩種方式,業(yè)界的追蹤系統(tǒng)能支持WebFlux相對(duì)較少
[OpenTracing] OpenTracing已進(jìn)入CNCF,正在為全球的分布式追蹤系統(tǒng)提供統(tǒng)一的概念、規(guī)范、架構(gòu)和數(shù)據(jù)標(biāo)準(zhǔn)。它通過(guò)提供平臺(tái)無(wú)關(guān)、廠商無(wú)關(guān)的API,使得開發(fā)人員能夠方便的添加(或更換)追蹤系統(tǒng)的實(shí)現(xiàn)。對(duì)于存在多樣化的技術(shù)棧共存的調(diào)用鏈中,Opentracing適配Java、C、Go和.Net等技術(shù)棧,實(shí)現(xiàn)全鏈路分布式追蹤功能。迄今為止,Uber Jaeger、Twitter Zipkin和Apache Skywalking已經(jīng)適配了Opentracing規(guī)范
筆者以Nepxion社區(qū)的Discovery開源框架(對(duì)該開源框架感興趣的同學(xué),請(qǐng)?jiān)L問(wèn)如下鏈接)為例子展開整合
源碼主頁(yè),請(qǐng)?jiān)L問(wèn)
https://github.com/Nepxion/Discovery
指南主頁(yè),請(qǐng)?jiān)L問(wèn)
https://github.com/Nepxion/DiscoveryGuide
文檔主頁(yè),請(qǐng)?jiān)L問(wèn)
https://pan.baidu.com/s/1i57rXaNKPuhGRqZ2MONZOA#list/path=%2FNepxion
整合的效果圖
基本概念
灰度調(diào)用鏈主要包括如下11個(gè)參數(shù)。使用者可以自行定義要傳遞的調(diào)用鏈參數(shù),例如:traceId, spanId等;也可以自行定義要傳遞的業(yè)務(wù)調(diào)用鏈參數(shù),例如:mobile, user等
1. n-d-service-group - 服務(wù)所屬組或者應(yīng)用
2. n-d-service-type - 服務(wù)類型,分為“網(wǎng)關(guān)”和“服務(wù)”
3. n-d-service-id - 服務(wù)ID
4. n-d-service-address - 服務(wù)地址,包括Host和Port
5. n-d-service-version - 服務(wù)版本
6. n-d-service-region - 服務(wù)所屬區(qū)域
7. n-d-version - 版本路由值
8. n-d-region - 區(qū)域路由值
9. n-d-address - 地址路由值
10. n-d-version-weight - 版本權(quán)重路由值
11. n-d-region-weight - 區(qū)域權(quán)重路由值
核心實(shí)現(xiàn)
Opentracing通用模塊
源碼參考
https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-opentracing
由于OpenTracing擴(kuò)展需要兼顧到Spring Cloud Gateway、Zuul和服務(wù),它的核心邏輯存在著一定的可封裝性,所以筆者抽取出一個(gè)公共模塊discovery-plugin-strategy-opentracing,包含configuration、operation、context等模塊,著重闡述operation模塊,其它比較簡(jiǎn)單,不一一贅述了
在闡述前,筆者需要解釋一個(gè)配置,該配置將決定核心實(shí)現(xiàn)以及終端界面的顯示
- 如果開啟,灰度信息輸出到獨(dú)立的Span節(jié)點(diǎn)中,意味著在界面顯示中,灰度信息通過(guò)獨(dú)立的GRAY Span節(jié)點(diǎn)來(lái)顯示。優(yōu)點(diǎn)是信息簡(jiǎn)潔明了,缺點(diǎn)是Span節(jié)點(diǎn)會(huì)增長(zhǎng)一倍。我們可以稱呼它為【模式A】
- 如果關(guān)閉,灰度信息輸出到原生的Span節(jié)點(diǎn)中,意味著在界面顯示中,灰度信息會(huì)和原生Span節(jié)點(diǎn)的調(diào)用信息、協(xié)議信息等混在一起,缺點(diǎn)是信息龐雜混合,優(yōu)點(diǎn)是Span節(jié)點(diǎn)數(shù)不會(huì)增長(zhǎng)。我們可以稱呼它為【模式B】
# 啟動(dòng)和關(guān)閉調(diào)用鏈的灰度信息在Opentracing中以獨(dú)立的Span節(jié)點(diǎn)輸出,如果關(guān)閉,則灰度信息輸出到原生的Span節(jié)點(diǎn)中。缺失則默認(rèn)為true
spring.application.strategy.trace.opentracing.separate.span.enabled=true
Opentracing公共操作類 - StrategyOpentracingOperation.java
- 裝配注入Opentracing的Tracer對(duì)象
- opentracingInitialize方法,提供給網(wǎng)關(guān)和服務(wù)的Span節(jié)點(diǎn)初始化
- 【模式A】下,tracer.buildSpan(...).start()實(shí)現(xiàn)新建一個(gè)Span,并把它放置到存儲(chǔ)上下文的StrategyOpentracingContext的ThreadLocal里
- 【模式B】下,不需要做任何工作
- opentracingHeader方法,提供給網(wǎng)關(guān)的灰度調(diào)用鏈輸出
- 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次把customizationMap(自定義的調(diào)用鏈參數(shù))的元素都放入到Tag中,最后把灰度調(diào)用鏈主11個(gè)參數(shù)(通過(guò)strategyContextHolder.getHeader(...)獲?。┖透嗌舷挛男畔⒎湃氲絋ag中
- 【模式B】下,跟【模式A】類似,唯一區(qū)別的是Tags.COMPONENT的處理,由于原生的Span節(jié)點(diǎn)已經(jīng)帶有該信息,所以不需要放入到Tag中
- opentracingLocal方法,提供給服務(wù)的灰度調(diào)用鏈輸出
- 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次把customizationMap(自定義的調(diào)用鏈參數(shù))的元素都放入到Tag中,最后把灰度調(diào)用鏈主11個(gè)參數(shù)(通過(guò)pluginAdapter.getXXX()獲?。┖透嗌舷挛男畔⒎湃氲絋ag中
- 【模式B】下,跟【模式A】類似,唯一區(qū)別的是Tags.COMPONENT的處理,由于原生的Span節(jié)點(diǎn)已經(jīng)帶有該信息,所以不需要放入到Tag中
- opentracingError方法,提供給服務(wù)的灰度調(diào)用鏈異常輸出
- 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次span.log(...)方法實(shí)現(xiàn)異常輸出
- 【模式B】下,不需要做任何工作
- opentracingClear方法,灰度調(diào)用鏈的Span上報(bào)和清除
- 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次span.finish()方法實(shí)現(xiàn)Span上報(bào),最后StrategyOpentracingContext.clearCurrentContext()方法實(shí)現(xiàn)Span清除
- 【模式B】下,不需要做任何工作
- getCurrentSpan方法
- 【模式A】下,返回StrategyOpentracingContext.getCurrentContext().getSpan(),即opentracingInitialize新建的Span對(duì)象
- 【模式B】下,返回tracer.activeSpan(),即原生的Span對(duì)象
public class StrategyOpentracingOperation {
private static final Logger LOG = LoggerFactory.getLogger(StrategyOpentracingOperation.class);
@Autowired
protected PluginAdapter pluginAdapter;
@Autowired
protected StrategyContextHolder strategyContextHolder;
@Autowired
private Tracer tracer;
@Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_ENABLED + ":false}")
protected Boolean traceOpentracingEnabled;
@Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_SEPARATE_SPAN_ENABLED + ":true}")
protected Boolean traceOpentracingSeparateSpanEnabled;
public void opentracingInitialize() {
if (!traceOpentracingEnabled) {
return;
}
if (!traceOpentracingSeparateSpanEnabled) {
return;
}
Span span = tracer.buildSpan(DiscoveryConstant.SPAN_VALUE).start();
StrategyOpentracingContext.getCurrentContext().setSpan(span);
LOG.debug("Trace chain for Opentracing initialized...");
}
public void opentracingHeader(Map<String, String> customizationMap) {
if (!traceOpentracingEnabled) {
return;
}
Span span = getCurrentSpan();
if (span == null) {
LOG.error("Span not found in context to opentracing header");
return;
}
if (MapUtils.isNotEmpty(customizationMap)) {
for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
span.setTag(entry.getKey(), entry.getValue());
}
}
if (traceOpentracingSeparateSpanEnabled) {
span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
}
span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, strategyContextHolder.getHeader(DiscoveryConstant.N_D_SERVICE_GROUP));
...
String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
if (StringUtils.isNotEmpty(routeVersion)) {
span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
}
...
LOG.debug("Trace chain information outputs to Opentracing...");
}
public void opentracingLocal(String className, String methodName, Map<String, String> customizationMap) {
if (!traceOpentracingEnabled) {
return;
}
Span span = getCurrentSpan();
if (span == null) {
LOG.error("Span not found in context to opentracing local");
return;
}
if (MapUtils.isNotEmpty(customizationMap)) {
for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
span.setTag(entry.getKey(), entry.getValue());
}
}
if (traceOpentracingSeparateSpanEnabled) {
span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
}
span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
span.setTag(DiscoveryConstant.CLASS, className);
span.setTag(DiscoveryConstant.METHOD, methodName);
span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, pluginAdapter.getGroup());
...
String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
if (StringUtils.isNotEmpty(routeVersion)) {
span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
}
...
LOG.debug("Trace chain information outputs to Opentracing...");
}
public void opentracingError(String className, String methodName, Throwable e) {
if (!traceOpentracingEnabled) {
return;
}
if (!traceOpentracingSeparateSpanEnabled) {
return;
}
Span span = getCurrentSpan();
if (span == null) {
LOG.error("Span not found in context to opentracing error");
return;
}
span.log(new ImmutableMap.Builder<String, Object>()
.put(DiscoveryConstant.CLASS, className)
.put(DiscoveryConstant.METHOD, methodName)
.put(DiscoveryConstant.EVENT, Tags.ERROR.getKey())
.put(DiscoveryConstant.ERROR_OBJECT, e)
.build());
LOG.debug("Trace chain error outputs to Opentracing...");
}
public void opentracingClear() {
if (!traceOpentracingEnabled) {
return;
}
if (!traceOpentracingSeparateSpanEnabled) {
return;
}
Span span = getCurrentSpan();
if (span != null) {
span.finish();
} else {
LOG.error("Span not found in context to opentracing clear");
}
StrategyOpentracingContext.clearCurrentContext();
LOG.debug("Trace chain context of Opentracing cleared...");
}
public Span getCurrentSpan() {
return traceOpentracingSeparateSpanEnabled ? StrategyOpentracingContext.getCurrentContext().getSpan() : tracer.activeSpan();
}
public String getTraceId() {
if (!traceOpentracingEnabled) {
return null;
}
Span span = getCurrentSpan();
if (span != null) {
return span.context().toTraceId();
}
return null;
}
public String getSpanId() {
if (!traceOpentracingEnabled) {
return null;
}
Span span = getCurrentSpan();
if (span != null) {
return span.context().toSpanId();
}
return null;
}
}
Opentracing Service模塊
實(shí)現(xiàn)OpenTracing對(duì)服務(wù)的擴(kuò)展,包含configuration、tracer等模塊,著重闡述tracer模塊,其它比較簡(jiǎn)單,不一一贅述了
Opentracing的服務(wù)追蹤類 - DefaultServiceStrategyOpentracingTracer.java
- 繼承DefaultServiceStrategyTracer,并注入StrategyOpentracingOperation
- trace方法里先執(zhí)行opentracingInitialize初始化Span,這樣可以讓后面的邏輯都可以從Span中拿到traceId和spanId,執(zhí)行opentracingLocal實(shí)現(xiàn)服務(wù)的灰度調(diào)用鏈輸出
- error方法里執(zhí)行opentracingError實(shí)現(xiàn)服務(wù)的灰度調(diào)用鏈異常輸出
- release方法里執(zhí)行opentracingClear實(shí)現(xiàn)灰度調(diào)用鏈的Span上報(bào)和清除
public class DefaultServiceStrategyOpentracingTracer extends DefaultServiceStrategyTracer {
@Autowired
private StrategyOpentracingOperation strategyOpentracingOperation;
@Override
public void trace(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
strategyOpentracingOperation.opentracingInitialize();
super.trace(interceptor, invocation);
strategyOpentracingOperation.opentracingLocal(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), getCustomizationMap());
}
@Override
public void error(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation, Throwable e) {
super.error(interceptor, invocation, e);
strategyOpentracingOperation.opentracingError(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), e);
}
@Override
public void release(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
super.release(interceptor, invocation);
strategyOpentracingOperation.opentracingClear();
}
@Override
public String getTraceId() {
return strategyOpentracingOperation.getTraceId();
}
@Override
public String getSpanId() {
return strategyOpentracingOperation.getSpanId();
}
}
Opentracing Spring Cloud Gateway模塊
實(shí)現(xiàn)OpenTracing對(duì)Spring Cloud Gateway的擴(kuò)展,跟discovery-plugin-strategy-starter-service-opentracing模塊類似,不一一贅述了
Opentracing Zuul模塊
源碼參考
https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-zuul-opentracing
實(shí)現(xiàn)OpenTracing對(duì)Zuul的擴(kuò)展,跟discovery-plugin-strategy-starter-service-opentracing模塊類似,不一一贅述了
使用說(shuō)明
示例參考
https://github.com/Nepxion/DiscoveryGuide
使用方式
Opentracing輸出方式以Uber Jaeger為例來(lái)說(shuō)明,步驟非常簡(jiǎn)單
- 從https://pan.baidu.com/s/1i57rXaNKPuhGRqZ2MONZOA#list/path=%2FNepxion獲取Jaeger-1.14.0.zip,Windows操作系統(tǒng)下解壓后運(yùn)行jaeger.bat,Mac和Lunix操作系統(tǒng)請(qǐng)自行研究
- 執(zhí)行Postman調(diào)用后,訪問(wèn)http://localhost:16686查看灰度調(diào)用鏈
- 灰度調(diào)用鏈支持WebMvc和WebFlux兩種方式,以GRAY字樣的標(biāo)記來(lái)標(biāo)識(shí)
開關(guān)控制
對(duì)于Opentracing調(diào)用鏈功能的開啟和關(guān)閉,需要通過(guò)如下開關(guān)做控制:
# 啟動(dòng)和關(guān)閉調(diào)用鏈。缺失則默認(rèn)為false
spring.application.strategy.trace.enabled=true
# 啟動(dòng)和關(guān)閉調(diào)用鏈的Opentracing輸出,支持F版或更高版本的配置,其它版本不需要該行配置。缺失則默認(rèn)為false
spring.application.strategy.trace.opentracing.enabled=true
# 啟動(dòng)和關(guān)閉調(diào)用鏈的灰度信息在Opentracing中以獨(dú)立的Span節(jié)點(diǎn)輸出,如果關(guān)閉,則灰度信息輸出到原生的Span節(jié)點(diǎn)中。缺失則默認(rèn)為true
spring.application.strategy.trace.opentracing.separate.span.enabled=true
可選功能
自定義調(diào)用鏈上下文參數(shù)的創(chuàng)建(該類不是必須的),繼承DefaultStrategyTracerAdapter
// 自定義調(diào)用鏈上下文參數(shù)的創(chuàng)建
// 對(duì)于getTraceId和getSpanId方法,在Opentracing等調(diào)用鏈中間件引入的情況下,由調(diào)用鏈中間件決定,在這里定義不會(huì)起作用;在Opentracing等調(diào)用鏈中間件未引入的情況下,在這里定義才有效,下面代碼中表示從Http Header中獲取,并全鏈路傳遞
// 對(duì)于getCustomizationMap方法,表示輸出到調(diào)用鏈中的定制化業(yè)務(wù)參數(shù),可以同時(shí)輸出到日志和Opentracing等調(diào)用鏈中間件,下面代碼中表示從Http Header中獲取,并全鏈路傳遞
public class MyStrategyTracerAdapter extends DefaultStrategyTracerAdapter {
@Override
public String getTraceId() {
return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID) : StringUtils.EMPTY;
}
@Override
public String getSpanId() {
return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID) : StringUtils.EMPTY;
}
@Override
public Map<String, String> getCustomizationMap() {
return new ImmutableMap.Builder<String, String>()
.put("mobile", StringUtils.isNotEmpty(strategyContextHolder.getHeader("mobile")) ? strategyContextHolder.getHeader("mobile") : StringUtils.EMPTY)
.put("user", StringUtils.isNotEmpty(strategyContextHolder.getHeader("user")) ? strategyContextHolder.getHeader("user") : StringUtils.EMPTY)
.build();
}
}
在配置類里@Bean方式進(jìn)行調(diào)用鏈類創(chuàng)建,覆蓋框架內(nèi)置的調(diào)用鏈類
@Bean
public StrategyTracerAdapter strategyTracerAdapter() {
return new MyStrategyTracerAdapter();
}
本文作者
任浩軍, 10 多年開源經(jīng)歷,Github ID:@HaojunRen,Nepxion 開源社區(qū)創(chuàng)始人,Nacos Group Member,Spring Cloud Alibaba & Nacos & Sentinel Committer
請(qǐng)聯(lián)系我
微信、公眾號(hào)和文檔
本文由博客一文多發(fā)平臺(tái) OpenWrite 發(fā)布!