背景

最近在使用fluent-bit做日志解析，有部分日志解析不了，專門研究下怎么解析這種日志。做個(gè)筆記，以后再使用時(shí)直接拿出來使用。

日志格式

2019-02-06 22:32:30 [INFO ][:0] - <DaZhuanPan_Check> start_time:1549209600, end_time: 1549468799
2019-02-06 22:32:30 [INFO ][:0] - <DaZhuanPan_Check> start_time:1549209600, end_time: 1549468799
2019-02-06 22:32:30 [INFO ][:0] - [draw_machine_global] DoGmActivity self._acUniqueId:36, updateCfg->uniqueId:36, startTime:1549209600, endTime:1549814100.
2019-02-06 22:32:30 [INFO ][:0] - [SetQueueLoginGlobal]  DoGmActivity maxCnt:0 warnCnt:0 allowCntMinute:0
2019-02-06 22:32:30 [INFO ][:0] - [QueueLoginMgr] OnSetQueueNum  maxCut:2000 warnCnt:1700 allowCntMinute:6
2019-02-06 22:32:30 [ERROR][:0] - server state change, new state:0, effect time:1548840180, delay:1, interval:1, kick:False, tip:
2019-02-06 22:32:30 [INFO ][:0] - <WuYiDaFangSong_Check> start_time:1549209720.0, end_time: 1549468740.0
2019-02-06 22:32:30 [INFO ][:0] - <WuYiDaFangSong_Check> start_time:1549209720.0, end_time: 1549468740.0
2019-02-06 22:32:31 [ERROR][:0] - server state on effect timer, cnt:1
2019-02-06 22:32:31 [INFO ][:0] - ****** dirty_queue.py done, interval:60 seconds ******

通過ruby的正則編輯工具調(diào)試

https://rubular.com/r/C1vfTpxMIYS59T
fluent-bit parse配置如下：

[PARSER]
    Name    mwptest
    Format  regex
    Regex    ^(?<time>[^\]]*) \[(?<time1>[^\]]*)\]\[.*\] - (?<message>.*)$
    Time_Key    time
    #Time_Format %Y-%m-%d:%H:%M:%S,%L
    #Time_Format %Y-%m-%dT%H:%M:%S,%L
    Time_Format %Y-%m-%d %H:%M:%S

fluent-bit.conf 部分配置

[INPUT]
    Name tail
    Tag  cmge.autotools
    Path /data/log/test    #這里不能為目錄，只能匹配文件，如：*.log
    Parser mwptest
[OUTPUT]
    Name  file
    Match *
    Path output.txt

解析出來的結(jié)果如下

cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"<DaZhuanPan_Check> start_time:1549209600, end_time: 1549468799"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"<DaZhuanPan_Check> start_time:1549209600, end_time: 1549468799"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[draw_machine_global] DoGmActivity self._acUniqueId:36, updateCfg->uniqueId:36, startTime:1549209600, endTime:1549814100."}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[SetQueueLoginGlobal]  DoGmActivity maxCnt:0 warnCnt:0 allowCntMinute:0"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"[QueueLoginMgr] OnSetQueueNum  maxCut:2000 warnCnt:1700 allowCntMinute:6"}]
cmge.autotools: [1549492350.000000, {"time1":"ERROR", "message":"server state change, new state:0, effect time:1548840180, delay:1, interval:1, kick:False, tip:"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"<WuYiDaFangSong_Check> start_time:1549209720.0, end_time: 1549468740.0"}]
cmge.autotools: [1549492350.000000, {"time1":"INFO ", "message":"<WuYiDaFangSong_Check> start_time:1549209720.0, end_time: 1549468740.0"}]
cmge.autotools: [1549492351.000000, {"time1":"ERROR", "message":"server state on effect timer, cnt:1"}]
cmge.autotools: [1549492351.000000, {"time1":"INFO ", "message":"****** dirty_queue.py done, interval:60 seconds ******"}]

總結(jié)下

這里最主要是要學(xué)會(huì)利用ruby的正則編輯工具幫助做解析，可以減少很多自己在環(huán)境中調(diào)試時(shí)間。