基于Opentracing+Jaeger全鏈路灰度調(diào)用鏈

當(dāng)網(wǎng)關(guān)和服務(wù)在實(shí)施全鏈路分布式灰度發(fā)布和路由時(shí)候,我們需要一款追蹤系統(tǒng)來(lái)監(jiān)控網(wǎng)關(guān)和服務(wù)走的是哪個(gè)灰度組,哪個(gè)灰度版本,哪個(gè)灰度區(qū)域,甚至監(jiān)控從Http Header頭部全程傳遞的灰度規(guī)則和路由策略。這個(gè)功能意義在于:

  • 不僅可以監(jiān)控全鏈路中基本的調(diào)用信息,也可以監(jiān)控額外的灰度信息,有助于我們判斷灰度發(fā)布和路由是否執(zhí)行準(zhǔn)確,一旦有問(wèn)題,也可以快速定位
  • 可以監(jiān)控流量何時(shí)切換到新版本,或者新的區(qū)域,或者新的機(jī)器上
  • 可以監(jiān)控灰度規(guī)則和路由策略是否配置準(zhǔn)確
  • 可以監(jiān)控網(wǎng)關(guān)和服務(wù)灰度上下級(jí)樹狀關(guān)系
  • 可以監(jiān)控全鏈路流量拓?fù)鋱D

筆者嘗試調(diào)研了一系列分布式追蹤系統(tǒng)和中間件,包括Opentracing、Uber Jaeger、Twitter Zipkin、Apache Skywalking、Pinpoint、CAT等,最后決定采用Opentracing + Uber Jaeger方式來(lái)實(shí)現(xiàn),重要原因除了易用性和可擴(kuò)展性外,Opentracing支持WebMvc和WebFlux兩種方式,業(yè)界的追蹤系統(tǒng)能支持WebFlux相對(duì)較少

[OpenTracing] OpenTracing已進(jìn)入CNCF,正在為全球的分布式追蹤系統(tǒng)提供統(tǒng)一的概念、規(guī)范、架構(gòu)和數(shù)據(jù)標(biāo)準(zhǔn)。它通過(guò)提供平臺(tái)無(wú)關(guān)、廠商無(wú)關(guān)的API,使得開發(fā)人員能夠方便的添加(或更換)追蹤系統(tǒng)的實(shí)現(xiàn)。對(duì)于存在多樣化的技術(shù)棧共存的調(diào)用鏈中,Opentracing適配Java、C、Go和.Net等技術(shù)棧,實(shí)現(xiàn)全鏈路分布式追蹤功能。迄今為止,Uber Jaeger、Twitter Zipkin和Apache Skywalking已經(jīng)適配了Opentracing規(guī)范

筆者以Nepxion社區(qū)的Discovery開源框架(對(duì)該開源框架感興趣的同學(xué),請(qǐng)?jiān)L問(wèn)如下鏈接)為例子展開整合

源碼主頁(yè),請(qǐng)?jiān)L問(wèn)
https://github.com/Nepxion/Discovery

指南主頁(yè),請(qǐng)?jiān)L問(wèn)
https://github.com/Nepxion/DiscoveryGuide

文檔主頁(yè),請(qǐng)?jiān)L問(wèn)
https://pan.baidu.com/s/1i57rXaNKPuhGRqZ2MONZOA#list/path=%2FNepxion

整合的效果圖





基本概念

灰度調(diào)用鏈主要包括如下11個(gè)參數(shù)。使用者可以自行定義要傳遞的調(diào)用鏈參數(shù),例如:traceId, spanId等;也可以自行定義要傳遞的業(yè)務(wù)調(diào)用鏈參數(shù),例如:mobile, user等

1. n-d-service-group - 服務(wù)所屬組或者應(yīng)用
2. n-d-service-type - 服務(wù)類型,分為“網(wǎng)關(guān)”和“服務(wù)”
3. n-d-service-id - 服務(wù)ID
4. n-d-service-address - 服務(wù)地址,包括Host和Port
5. n-d-service-version - 服務(wù)版本
6. n-d-service-region - 服務(wù)所屬區(qū)域
7. n-d-version - 版本路由值
8. n-d-region - 區(qū)域路由值
9. n-d-address - 地址路由值
10. n-d-version-weight - 版本權(quán)重路由值
11. n-d-region-weight - 區(qū)域權(quán)重路由值

核心實(shí)現(xiàn)

Opentracing通用模塊

源碼參考
https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-opentracing

由于OpenTracing擴(kuò)展需要兼顧到Spring Cloud Gateway、Zuul和服務(wù),它的核心邏輯存在著一定的可封裝性,所以筆者抽取出一個(gè)公共模塊discovery-plugin-strategy-opentracing,包含configuration、operation、context等模塊,著重闡述operation模塊,其它比較簡(jiǎn)單,不一一贅述了

在闡述前,筆者需要解釋一個(gè)配置,該配置將決定核心實(shí)現(xiàn)以及終端界面的顯示

  1. 如果開啟,灰度信息輸出到獨(dú)立的Span節(jié)點(diǎn)中,意味著在界面顯示中,灰度信息通過(guò)獨(dú)立的GRAY Span節(jié)點(diǎn)來(lái)顯示。優(yōu)點(diǎn)是信息簡(jiǎn)潔明了,缺點(diǎn)是Span節(jié)點(diǎn)會(huì)增長(zhǎng)一倍。我們可以稱呼它為【模式A】
  2. 如果關(guān)閉,灰度信息輸出到原生的Span節(jié)點(diǎn)中,意味著在界面顯示中,灰度信息會(huì)和原生Span節(jié)點(diǎn)的調(diào)用信息、協(xié)議信息等混在一起,缺點(diǎn)是信息龐雜混合,優(yōu)點(diǎn)是Span節(jié)點(diǎn)數(shù)不會(huì)增長(zhǎng)。我們可以稱呼它為【模式B】
# 啟動(dòng)和關(guān)閉調(diào)用鏈的灰度信息在Opentracing中以獨(dú)立的Span節(jié)點(diǎn)輸出,如果關(guān)閉,則灰度信息輸出到原生的Span節(jié)點(diǎn)中。缺失則默認(rèn)為true
spring.application.strategy.trace.opentracing.separate.span.enabled=true

Opentracing公共操作類 - StrategyOpentracingOperation.java

  • 裝配注入Opentracing的Tracer對(duì)象
  • opentracingInitialize方法,提供給網(wǎng)關(guān)和服務(wù)的Span節(jié)點(diǎn)初始化
    • 【模式A】下,tracer.buildSpan(...).start()實(shí)現(xiàn)新建一個(gè)Span,并把它放置到存儲(chǔ)上下文的StrategyOpentracingContext的ThreadLocal里
    • 【模式B】下,不需要做任何工作
  • opentracingHeader方法,提供給網(wǎng)關(guān)的灰度調(diào)用鏈輸出
    • 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次把customizationMap(自定義的調(diào)用鏈參數(shù))的元素都放入到Tag中,最后把灰度調(diào)用鏈主11個(gè)參數(shù)(通過(guò)strategyContextHolder.getHeader(...)獲?。┖透嗌舷挛男畔⒎湃氲絋ag中
    • 【模式B】下,跟【模式A】類似,唯一區(qū)別的是Tags.COMPONENT的處理,由于原生的Span節(jié)點(diǎn)已經(jīng)帶有該信息,所以不需要放入到Tag中
  • opentracingLocal方法,提供給服務(wù)的灰度調(diào)用鏈輸出
    • 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次把customizationMap(自定義的調(diào)用鏈參數(shù))的元素都放入到Tag中,最后把灰度調(diào)用鏈主11個(gè)參數(shù)(通過(guò)pluginAdapter.getXXX()獲?。┖透嗌舷挛男畔⒎湃氲絋ag中
    • 【模式B】下,跟【模式A】類似,唯一區(qū)別的是Tags.COMPONENT的處理,由于原生的Span節(jié)點(diǎn)已經(jīng)帶有該信息,所以不需要放入到Tag中
  • opentracingError方法,提供給服務(wù)的灰度調(diào)用鏈異常輸出
    • 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次span.log(...)方法實(shí)現(xiàn)異常輸出
    • 【模式B】下,不需要做任何工作
  • opentracingClear方法,灰度調(diào)用鏈的Span上報(bào)和清除
    • 【模式A】下,首先從StrategyOpentracingContext的ThreadLocal里獲取Span對(duì)象,其次span.finish()方法實(shí)現(xiàn)Span上報(bào),最后StrategyOpentracingContext.clearCurrentContext()方法實(shí)現(xiàn)Span清除
    • 【模式B】下,不需要做任何工作
  • getCurrentSpan方法
    • 【模式A】下,返回StrategyOpentracingContext.getCurrentContext().getSpan(),即opentracingInitialize新建的Span對(duì)象
    • 【模式B】下,返回tracer.activeSpan(),即原生的Span對(duì)象
public class StrategyOpentracingOperation {
    private static final Logger LOG = LoggerFactory.getLogger(StrategyOpentracingOperation.class);

    @Autowired
    protected PluginAdapter pluginAdapter;

    @Autowired
    protected StrategyContextHolder strategyContextHolder;

    @Autowired
    private Tracer tracer;

    @Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_ENABLED + ":false}")
    protected Boolean traceOpentracingEnabled;

    @Value("${" + StrategyOpentracingConstant.SPRING_APPLICATION_STRATEGY_TRACE_OPENTRACING_SEPARATE_SPAN_ENABLED + ":true}")
    protected Boolean traceOpentracingSeparateSpanEnabled;

    public void opentracingInitialize() {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = tracer.buildSpan(DiscoveryConstant.SPAN_VALUE).start();
        StrategyOpentracingContext.getCurrentContext().setSpan(span);

        LOG.debug("Trace chain for Opentracing initialized...");
    }

    public void opentracingHeader(Map<String, String> customizationMap) {
        if (!traceOpentracingEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing header");

            return;
        }

        if (MapUtils.isNotEmpty(customizationMap)) {
            for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
                span.setTag(entry.getKey(), entry.getValue());
            }
        }

        if (traceOpentracingSeparateSpanEnabled) {
            span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
        }
        span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
        span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
        span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
        span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, strategyContextHolder.getHeader(DiscoveryConstant.N_D_SERVICE_GROUP));
        ...

        String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
        if (StringUtils.isNotEmpty(routeVersion)) {
            span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
        }
        ...

        LOG.debug("Trace chain information outputs to Opentracing...");
    }

    public void opentracingLocal(String className, String methodName, Map<String, String> customizationMap) {
        if (!traceOpentracingEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing local");

            return;
        }

        if (MapUtils.isNotEmpty(customizationMap)) {
            for (Map.Entry<String, String> entry : customizationMap.entrySet()) {
                span.setTag(entry.getKey(), entry.getValue());
            }
        }

        if (traceOpentracingSeparateSpanEnabled) {
            span.setTag(Tags.COMPONENT.getKey(), DiscoveryConstant.TAG_COMPONENT_VALUE);
        }
        span.setTag(DiscoveryConstant.PLUGIN, DiscoveryConstant.PLUGIN_VALUE);
        span.setTag(DiscoveryConstant.CLASS, className);
        span.setTag(DiscoveryConstant.METHOD, methodName);
        span.setTag(DiscoveryConstant.TRACE_ID, span.context().toTraceId());
        span.setTag(DiscoveryConstant.SPAN_ID, span.context().toSpanId());
        span.setTag(DiscoveryConstant.N_D_SERVICE_GROUP, pluginAdapter.getGroup());
        ...

        String routeVersion = strategyContextHolder.getHeader(DiscoveryConstant.N_D_VERSION);
        if (StringUtils.isNotEmpty(routeVersion)) {
            span.setTag(DiscoveryConstant.N_D_VERSION, routeVersion);
        }
        ...

        LOG.debug("Trace chain information outputs to Opentracing...");
    }

    public void opentracingError(String className, String methodName, Throwable e) {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span == null) {
            LOG.error("Span not found in context to opentracing error");

            return;
        }

        span.log(new ImmutableMap.Builder<String, Object>()
                .put(DiscoveryConstant.CLASS, className)
                .put(DiscoveryConstant.METHOD, methodName)
                .put(DiscoveryConstant.EVENT, Tags.ERROR.getKey())
                .put(DiscoveryConstant.ERROR_OBJECT, e)
                .build());

        LOG.debug("Trace chain error outputs to Opentracing...");
    }

    public void opentracingClear() {
        if (!traceOpentracingEnabled) {
            return;
        }

        if (!traceOpentracingSeparateSpanEnabled) {
            return;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            span.finish();
        } else {
            LOG.error("Span not found in context to opentracing clear");
        }
        StrategyOpentracingContext.clearCurrentContext();

        LOG.debug("Trace chain context of Opentracing cleared...");
    }

    public Span getCurrentSpan() {
        return traceOpentracingSeparateSpanEnabled ? StrategyOpentracingContext.getCurrentContext().getSpan() : tracer.activeSpan();
    }

    public String getTraceId() {
        if (!traceOpentracingEnabled) {
            return null;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            return span.context().toTraceId();
        }

        return null;
    }

    public String getSpanId() {
        if (!traceOpentracingEnabled) {
            return null;
        }

        Span span = getCurrentSpan();
        if (span != null) {
            return span.context().toSpanId();
        }

        return null;
    }
}

Opentracing Service模塊

源碼參考
https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-service-opentracing

實(shí)現(xiàn)OpenTracing對(duì)服務(wù)的擴(kuò)展,包含configuration、tracer等模塊,著重闡述tracer模塊,其它比較簡(jiǎn)單,不一一贅述了

Opentracing的服務(wù)追蹤類 - DefaultServiceStrategyOpentracingTracer.java

  • 繼承DefaultServiceStrategyTracer,并注入StrategyOpentracingOperation
  • trace方法里先執(zhí)行opentracingInitialize初始化Span,這樣可以讓后面的邏輯都可以從Span中拿到traceId和spanId,執(zhí)行opentracingLocal實(shí)現(xiàn)服務(wù)的灰度調(diào)用鏈輸出
  • error方法里執(zhí)行opentracingError實(shí)現(xiàn)服務(wù)的灰度調(diào)用鏈異常輸出
  • release方法里執(zhí)行opentracingClear實(shí)現(xiàn)灰度調(diào)用鏈的Span上報(bào)和清除
public class DefaultServiceStrategyOpentracingTracer extends DefaultServiceStrategyTracer {
    @Autowired
    private StrategyOpentracingOperation strategyOpentracingOperation;

    @Override
    public void trace(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
        strategyOpentracingOperation.opentracingInitialize();

        super.trace(interceptor, invocation);

        strategyOpentracingOperation.opentracingLocal(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), getCustomizationMap());
    }

    @Override
    public void error(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation, Throwable e) {
        super.error(interceptor, invocation, e);

        strategyOpentracingOperation.opentracingError(interceptor.getMethod(invocation).getDeclaringClass().getName(), interceptor.getMethodName(invocation), e);
    }

    @Override
    public void release(ServiceStrategyTracerInterceptor interceptor, MethodInvocation invocation) {
        super.release(interceptor, invocation);

        strategyOpentracingOperation.opentracingClear();
    }

    @Override
    public String getTraceId() {
        return strategyOpentracingOperation.getTraceId();
    }

    @Override
    public String getSpanId() {
        return strategyOpentracingOperation.getSpanId();
    }
}

Opentracing Spring Cloud Gateway模塊

源碼參考
https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-gateway-opentracing

實(shí)現(xiàn)OpenTracing對(duì)Spring Cloud Gateway的擴(kuò)展,跟discovery-plugin-strategy-starter-service-opentracing模塊類似,不一一贅述了

Opentracing Zuul模塊

源碼參考
https://github.com/Nepxion/Discovery/tree/master/discovery-plugin-strategy-starter-zuul-opentracing

實(shí)現(xiàn)OpenTracing對(duì)Zuul的擴(kuò)展,跟discovery-plugin-strategy-starter-service-opentracing模塊類似,不一一贅述了

使用說(shuō)明

示例參考
https://github.com/Nepxion/DiscoveryGuide

使用方式

Opentracing輸出方式以Uber Jaeger為例來(lái)說(shuō)明,步驟非常簡(jiǎn)單

  1. https://pan.baidu.com/s/1i57rXaNKPuhGRqZ2MONZOA#list/path=%2FNepxion獲取Jaeger-1.14.0.zip,Windows操作系統(tǒng)下解壓后運(yùn)行jaeger.bat,Mac和Lunix操作系統(tǒng)請(qǐng)自行研究
  2. 執(zhí)行Postman調(diào)用后,訪問(wèn)http://localhost:16686查看灰度調(diào)用鏈
  3. 灰度調(diào)用鏈支持WebMvc和WebFlux兩種方式,以GRAY字樣的標(biāo)記來(lái)標(biāo)識(shí)

開關(guān)控制

對(duì)于Opentracing調(diào)用鏈功能的開啟和關(guān)閉,需要通過(guò)如下開關(guān)做控制:

# 啟動(dòng)和關(guān)閉調(diào)用鏈。缺失則默認(rèn)為false
spring.application.strategy.trace.enabled=true
# 啟動(dòng)和關(guān)閉調(diào)用鏈的Opentracing輸出,支持F版或更高版本的配置,其它版本不需要該行配置。缺失則默認(rèn)為false
spring.application.strategy.trace.opentracing.enabled=true
# 啟動(dòng)和關(guān)閉調(diào)用鏈的灰度信息在Opentracing中以獨(dú)立的Span節(jié)點(diǎn)輸出,如果關(guān)閉,則灰度信息輸出到原生的Span節(jié)點(diǎn)中。缺失則默認(rèn)為true
spring.application.strategy.trace.opentracing.separate.span.enabled=true

可選功能

自定義調(diào)用鏈上下文參數(shù)的創(chuàng)建(該類不是必須的),繼承DefaultStrategyTracerAdapter

// 自定義調(diào)用鏈上下文參數(shù)的創(chuàng)建
// 對(duì)于getTraceId和getSpanId方法,在Opentracing等調(diào)用鏈中間件引入的情況下,由調(diào)用鏈中間件決定,在這里定義不會(huì)起作用;在Opentracing等調(diào)用鏈中間件未引入的情況下,在這里定義才有效,下面代碼中表示從Http Header中獲取,并全鏈路傳遞
// 對(duì)于getCustomizationMap方法,表示輸出到調(diào)用鏈中的定制化業(yè)務(wù)參數(shù),可以同時(shí)輸出到日志和Opentracing等調(diào)用鏈中間件,下面代碼中表示從Http Header中獲取,并全鏈路傳遞
public class MyStrategyTracerAdapter extends DefaultStrategyTracerAdapter {
    @Override
    public String getTraceId() {
        return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.TRACE_ID) : StringUtils.EMPTY;
    }

    @Override
    public String getSpanId() {
        return StringUtils.isNotEmpty(strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID)) ? strategyContextHolder.getHeader(DiscoveryConstant.SPAN_ID) : StringUtils.EMPTY;
    }

    @Override
    public Map<String, String> getCustomizationMap() {
        return new ImmutableMap.Builder<String, String>()
                .put("mobile", StringUtils.isNotEmpty(strategyContextHolder.getHeader("mobile")) ? strategyContextHolder.getHeader("mobile") : StringUtils.EMPTY)
                .put("user", StringUtils.isNotEmpty(strategyContextHolder.getHeader("user")) ? strategyContextHolder.getHeader("user") : StringUtils.EMPTY)
                .build();
    }
}

在配置類里@Bean方式進(jìn)行調(diào)用鏈類創(chuàng)建,覆蓋框架內(nèi)置的調(diào)用鏈類

@Bean
public StrategyTracerAdapter strategyTracerAdapter() {
    return new MyStrategyTracerAdapter();
}

本文作者

任浩軍, 10 多年開源經(jīng)歷,Github ID:@HaojunRen,Nepxion 開源社區(qū)創(chuàng)始人,Nacos Group Member,Spring Cloud Alibaba & Nacos & Sentinel Committer

請(qǐng)聯(lián)系我

微信、公眾號(hào)和文檔

本文由博客一文多發(fā)平臺(tái) OpenWrite 發(fā)布!

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡(jiǎn)書系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容