1.Fluentd安裝之前的準(zhǔn)備工作

1.在節(jié)點(diǎn)上設(shè)置NTP守護(hù)程序,以獲得準(zhǔn)確的當(dāng)前時間戳。

安裝ntp服務(wù)

//ubuntu
sudo apt install ntp
//centos
yum install ntp -y

編輯 /etc/ntp.conf

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

#新增:日志目錄.
logfile /var/log/ntpd.log

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#新增:時間服務(wù)器列表.
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst

注意:ubuntu下服務(wù)叫做ntp,centos下叫做ntpd
修改配置文件之后，你需要重新加載 ntpd：

systemctl enable ntp
systemctl start ntp

或者

/etc/init.d/ntp stop
/etc/init.d/ntp start

ubuntu主機(jī)下例子:

root@ubuntu:~# systemctl status ntp
● ntp.service - LSB: Start NTP daemon
   Loaded: loaded (/etc/init.d/ntp; bad; vendor preset: enabled)
   Active: active (running) since 四 2018-10-11 14:15:51 CST; 1 months 8 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 2
   Memory: 2.2M
      CPU: 3min 42.282s
   CGroup: /system.slice/ntp.service
           └─1506 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 123:132

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@ubuntu:~# systemctl is-enabled ntp
ntp.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install is-enabled ntp
enabled

centos主機(jī)下例子:

[root@master ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
[root@master ~]# systemctl enable ntpd
[root@master ~]# systemctl is-enabled ntpd
enabled
[root@master ~]# systemctl start ntpd     
[root@master ~]# systemctl status ntpd    
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-11-19 16:55:01 CST; 5s ago
  Process: 7170 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 7172 (ntpd)
    Tasks: 1
   Memory: 1.4M
   CGroup: /system.slice/ntpd.service
           └─7172 /usr/sbin/ntpd -u ntp:ntp -g

Nov 19 16:55:01 master.novalocal systemd[1]: Starting Network Time Service...
Nov 19 16:55:01 master.novalocal ntpd[7172]: proto: precision = 0.089 usec
Nov 19 16:55:01 master.novalocal ntpd[7172]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 19 16:55:01 master.novalocal systemd[1]: Started Network Time Service.

查看節(jié)點(diǎn)同步狀態(tài)

root@ubuntu:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
+ntp.hkg10.hk.le 130.133.1.10     2 u   38   64  377  121.432  -10.265   0.534
*45.125.1.20 (45 101.231.167.217  2 u   30   64  377  353.921   12.721   0.429
+118.140.184.98  223.255.185.2    2 u   34   64  157    9.668   13.966  40.916
-alphyn.canonica 192.53.103.108   2 u   37   64   73  341.456   41.184  37.807
-chilipepper.can 17.253.34.253    2 u   96   64   76  299.193  -17.247  29.960
-golem.canonical 145.238.203.14   2 u   36   64   37  274.246  -28.804  28.527
root@ubuntu:~# ntpstat
synchronised to NTP server (45.125.1.20) at stratum 3 
   time correct to within 236 ms
   polling server every 64 s

2.增加最大文件描述符數(shù)

使用ulimit -n命令檢查當(dāng)前號碼。

$ ulimit -n
1024

如果控制臺顯示1024，是不夠的。在/etc/security/limits.conf文件中添加以下行并重新啟動計算機(jī)。

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

3.優(yōu)化網(wǎng)絡(luò)內(nèi)核參數(shù)。

對于由許多Fluentd實(shí)例組成的高負(fù)載環(huán)境，請將這些參數(shù)添加到您的/etc/sysctl.conf文件中。

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

并輸入sysctl -p或重新啟動節(jié)點(diǎn)以使更改生效。

sysctl -p

2. 安裝Fluentd的客戶端td-agent

參考官網(wǎng)安裝教程
對配置文件進(jìn)行配置

//根據(jù)版本下載相應(yīng)客戶端，我的是16
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
//CentOS Linux release 7.4.1708 (Core)
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
//開始
systemctl start td-agent.service
//查看狀態(tài)
systemctl status td-agent.service
//停止
systemctl stop td-agent.service
//打開配置文件
vim /etc/td-agent/td-agent.conf

配置文件的語法可以參考官網(wǎng)給出的，或者參考博客1和博客2

3.簡明使用

Fluentd 安裝使用可以參考該文章。
參考該博客，有比較詳盡的fluentd工具使用經(jīng)驗，日志收集工具Fluentd使用總結(jié)

fluentd配置文件所在目錄：/etc/td-agent/td-agent.conf

fluentd日志文件所在目錄：/var/log/td-agent/td-agent.log

fluentd二進(jìn)制文件所在目錄：/opt/td-agent/embedded/bin/

安裝插件的方法：td-agent-gem install [插件名稱]

4.監(jiān)聽python和HTTP請求的配置實(shí)例

配置文件的配置如下：

<source>
  @type forward
  port 24224
</source>

#匹配項：監(jiān)聽python的配置
<match fluentd.test.**>
  @type stdout
</match>

#匹配項：監(jiān)聽curl命令產(chǎn)生的HTTP的get和post請求
# listening for HTTP Requests
<source>
  @type http
  port 8888
  bind 0.0.0.0
</source>

# print the data arrived on each incoming request to standard output
<match test.cycle>
  @type stdout
</match>

<match debug.test>
  @type stdout
</match>

開一個終端用于跟蹤Fluentd收集到的應(yīng)用日志使用以下命令：

tail -f /var/log/td-agent/td-agent.log

打開另外一個終端用于測試和監(jiān)聽

curl命令操作

curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

curl -i -X POST -d 'json={"action":"login","user":2}' http://localhost:8888/test.cycle

python文件執(zhí)行操作
創(chuàng)建兩個文件如下圖：

simple.py和test.py

# test.py
from fluent import sender
from fluent import event
sender.setup('fluentd.test', host='localhost', port=24224)
event.Event('follow', {
  'from': 'userA',
  'to':   'userB'
})

# simple.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import itertools
import json
from fluent import sender
from fluent import event

sender.setup('fluentd.test', host='localhost', port=24224)

class cartesian(object):
    def __init__(self):
        self._data_list=[]

    def add_data(self,data=[]): #添加生成笛卡爾積的數(shù)據(jù)列表
        self._data_list.append(data)

    def build(self): #計算笛卡爾積
        for item in itertools.product(*self._data_list):
            print(item)

if __name__=="__main__":
    car=cartesian()
    car.add_data([1,2,3,4])
    car.add_data([5,6,7,8])
    car.add_data([9,10,11,12])
    car.build()
    event.Event('Cartesian', {
       'Set': '[1,2,3,4],[5,6,7,8],[9,10,11,12]',
       'Cartesian product': 'results are stdout,not here'
    })

分別執(zhí)行兩個文件

python test.py
python simple.py

執(zhí)行終端顯示結(jié)果如下

執(zhí)行終端顯示

日志跟蹤終端顯示如下：

root@ubuntu:/var/log/td-agent# tail -f td-agent.log
2018-09-06 10:35:15 +0800 [info]: #0 starting fluentd worker pid=8670 ppid=8665 worker=0
2018-09-06 10:35:15 +0800 [info]: #0 listening port port=24224 bind="0.0.0.0"
2018-09-06 10:35:15 +0800 [info]: #0 fluentd worker is now running worker=0
……
2018-09-06 15:34:56.333452308 +0800 debug.test: {"json":"message"}
2018-09-06 15:35:08.899919433 +0800 test.cycle: {"action":"login","user":2}
2018-09-06 15:35:23.000000000 +0800 fluentd.test.follow: {"to":"userB","from":"userA"}
2018-09-06 15:35:33.000000000 +0800 fluentd.test.Cartesian: {"Cartesian product":"results are stdout,not here","Set":"[1,2,3,4],[5,6,7,8],[9,10,11,12]"}

5.監(jiān)聽docker容器的配置實(shí)例

Docker版本需要在17.05以上才可以使用日志驅(qū)動插件(見官網(wǎng))。
使用參見官網(wǎng)說明：https://docs.docker.com/config/containers/logging/fluentd/

修改 /etc/passwd 文件

vim /etc/passwd

找到如下行，把td-agent用戶ID修改為 0 ，如下所示：

td-agent:x:0:133::/var/lib/td-agent:/bin/false

修改配置文件（/etc/td-agent/td-agent.conf ）如下：

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match docker.*>
  @type stdout
</match>

td-agent.conf

重啟fluentd

systemctl restart td-agent.service

運(yùn)行容器

docker run  -d -p 8081:80 --name nginx_bashlog -v /work/DOCKER/ubuntu-fluentd-test01:/root --log-driver=fluentd --log-opt tag=docker.nginx_test --log-opt fluentd-async-connect nginx

查看fluentd運(yùn)行狀態(tài)

查看收集到的日志

cd /var/log/td-agent
tail -f td-agent.log

收集到的日志如下：

2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"source":"stdout","log":"172.17.0.1 - - [10/Sep/2018:08:01:41 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog"}
2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"log":"2018/09/10 08:01:41 [error] 10#10: *1 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"localhost:8081\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog","source":"stderr"}
2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"container_name":"/nginx_bashlog","source":"stdout","log":"172.17.0.1 - - [10/Sep/2018:08:01:41 +0000] \"GET /favicon.ico HTTP/1.1\" 404 169 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6"}

td-agent.log

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av

Linux下td-agent（fluentd）的安裝和配置

Linux下td-agent（fluentd）的安裝和配置

1.Fluentd安裝之前的準(zhǔn)備工作

2. 安裝Fluentd的客戶端td-agent

3.簡明使用

4.監(jiān)聽python和HTTP請求的配置實(shí)例

開一個終端用于跟蹤Fluentd收集到的應(yīng)用日志使用以下命令：

打開另外一個終端用于測試和監(jiān)聽

5.監(jiān)聽docker容器的配置實(shí)例

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九 欧美,1769亚洲,黄色成人av

Linux下td-agent（fluentd）的安裝和配置

1.Fluentd安裝之前的準(zhǔn)備工作

2. 安裝Fluentd的客戶端td-agent

3.簡明使用

4.監(jiān)聽python和HTTP請求的配置實(shí)例

開一個終端用于跟蹤Fluentd收集到的應(yīng)用日志使用以下命令：

打開另外一個終端用于測試和監(jiān)聽

5.監(jiān)聽docker容器的配置實(shí)例

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av