一. 背景

在使用kylin增量構(gòu)建Cube時(shí)，出現(xiàn)這么一個(gè)情況：

在2018.06.27早晨上班后突然發(fā)現(xiàn)2018.06.26自動(dòng)增量構(gòu)建的Cube任務(wù)失敗了。于是打開 Kylin WebUI 界面想手動(dòng) resume 該 Job，但在 Kylin WebUI Monitor 界面卻找不到錯(cuò)誤構(gòu)建的歷史 Job.

二. 嘗試 Rebuild Job

于是決定 rebuild 該 Job，然而，無法重新構(gòu)建，報(bào) Segments overlap 錯(cuò)誤。如下圖：

原因是 Kylin 元數(shù)據(jù)中已經(jīng)有相同時(shí)間段的 Segment 存在，顧無法再構(gòu)建。

三. 嘗試 Resume Job

那既然 rebuild job 失敗，UI 界面也無歷史Job信息，那就調(diào)用 restful API resume job 吧。但 API 要求提供 jobid，哪里去獲取呢？

谷歌。。。

https://issues.apache.org/jira/browse/KYLIN-2795

該 JIRA 表明，在 kylin2.1 之前官方是沒有獲取 jobid 的 restful api 提供的，在2.2版本才補(bǔ)充上。而公司剛好用的是2.1版本，哭到在廁所。

然后只能扒 Kylin 的元數(shù)據(jù)了，查看 hbase 下 kylin_metadata 表，研究了下其 Rowkey 設(shè)計(jì)方式主要有如下幾種：

/acl/ffd1b1fd-eae0-4c2f-9808-26b58b1c21ac
/cube/kylin_cube.json
/cube_desc/kylin_cube.json
/cube_statistics/kylin_cube/3f7de744-8703-4b1d-9ca4-cd283ec9667c.seq
/dict/LOG.DW_APP_KYLIN_VIEW/ACITON_SOURCE/1335fd4c-bb7d-447b-a678-4972d19449aa.dict

我們通過如下命令拿到指定 Cube 的信息：

get 'kylin_metadata','/cube/kylin_cube.json'

這樣可以看到此cube的元數(shù)據(jù)信息，打印出來，找到指定的segment元數(shù)據(jù)所在的位置。

{
    "uuid" : "5877617f-5fd1-40b5-bdc9-fd59945135af",
    "name" : "20180626000000_20180627000000",
    "storage_location_identifier" : "KYLIN_8SJTX5SBNM",
    "date_range_start" : 1528416000000,
    "date_range_end" : 1528502400000,
    "source_offset_start" : 0,
    "source_offset_end" : 0,
    "status" : "NEW",
    "size_kb" : 0,
    "input_records" : 0,
    "input_records_size" : 0,
    "last_build_time" : 0,
    "last_build_job_id" : null,
    "create_time_utc" : 1529030216244,
    "cuboid_shard_nums" : { },
    "total_shards" : 0,
    "blackout_cuboids" : [ ],
    "binary_signature" : null,
    "dictionaries" : null,
    "snapshots" : null,
    "rowkey_stats" : [ ]
  }

然而，正如你所見，"last_build_job_id" : null，瞬間感覺不愛了，原來構(gòu)建不成功的任務(wù)是不存儲(chǔ) Jobid 的。

四. 嘗試刪除 Segment 重新構(gòu)建任務(wù)

于是想通過 Resume Job 的方式再Append 構(gòu)建 Job 任務(wù)不可行了。

那怎么辦呢？

于是想直接手動(dòng)刪除該 Segment 再重新構(gòu)建新任務(wù)，但當(dāng)調(diào)用 Restful API 刪除時(shí)卻報(bào)錯(cuò)了，如下：

{
    "code": "999",
    "data": null,
    "msg": "Cannot delete segment '20180603000000_20180604000000' as it is neither the first nor the last segment.",
    "stacktrace": "org.apache.kylin.rest.exception.InternalErrorException: Cannot delete segment '20180603000000_20180604000000' as it is neither the first nor the last segment.\n\tat org.apache.kylin.rest.controller.CubeController.deleteSegment(CubeController.java:247)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:606)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:743)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)\n\tat org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:891)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:656)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:731)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)\n\tat org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)\n\tat org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:48)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:213)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)\n\tat org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)\n\tat org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)\n\tat org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)\n\tat org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)\n\tat com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\tat org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\tat org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\tat org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\tat org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:745)\n",
    "exception": "Cannot delete segment '20180603000000_20180604000000' as it is neither the first nor the last segment.",
    "url": "http://10.15.184.21:7071/kylin/api/cubes/kylin_lc_log_cube_new_2/segs/20180603000000_20180604000000"
}

如報(bào)錯(cuò)信息所示，我們不能夠刪除它，因?yàn)樗炔皇堑谝粋€(gè)也不是最后一個(gè)Segment。原來，刪除Segment是有限制的，只能刪除首尾 Segment .

五. 直接修改元數(shù)據(jù)

那怎么辦呢？有了一個(gè)大膽的想法：既然API不支持刪除，那就手動(dòng)修改元數(shù)據(jù)徹底刪除該 Segment。

1. 備份

未防止錯(cuò)誤操作，我們先備份元數(shù)據(jù)，以便操作失誤后 Kylin 集群還可以再恢復(fù)到之前的狀態(tài)。

# 備份元數(shù)據(jù)
$ ./bin/metastore.sh backup
# 備份目錄為：$KYLIN_HOME/meta_backups/meta_2018_06_27_08_30_49

2. 清理

后來想，如果先將 Hbase 中無用的 Segment 清理下是否可以呢？執(zhí)行如下命令：

檢查元數(shù)據(jù)：./bin/metastore.sh clean
清除無效數(shù)據(jù)： ./bin/metastore.sh clean --delete true

3. 修改

結(jié)果不起作用，那就改元數(shù)據(jù)吧！

$ cd  $KYLIN_HOME/meta_backups/meta_2018_06_27_08_30_49/cube
$ vim  kylin_cube.json

找到如下數(shù)據(jù)，刪除，保存。

{
    "uuid" : "5877617f-5fd1-40b5-bdc9-fd59945135af",
    "name" : "20180626000000_20180627000000",
    "storage_location_identifier" : "KYLIN_8SJTX5SBNM",
    "date_range_start" : 1528416000000,
    "date_range_end" : 1528502400000,
    "source_offset_start" : 0,
    "source_offset_end" : 0,
    "status" : "NEW",
    "size_kb" : 0,
    "input_records" : 0,
    "input_records_size" : 0,
    "last_build_time" : 0,
    "last_build_job_id" : null,
    "create_time_utc" : 1529030216244,
    "cuboid_shard_nums" : { },
    "total_shards" : 0,
    "blackout_cuboids" : [ ],
    "binary_signature" : null,
    "dictionaries" : null,
    "snapshots" : null,
    "rowkey_stats" : [ ]
  }

4. 恢復(fù)

最后再用最新的元數(shù)據(jù)恢復(fù)集群，命令如下：

$ ./bin/metastore.sh reset

$ ./bin/metastore.sh restore $KYLIN_HOME/meta_backups/meta_2018_06_27_08_30_49

在 WebUI 界面刷新元數(shù)據(jù)，重新 Build 任務(wù)，構(gòu)建成功！

搞定！