ELK日志分析系統(tǒng)在CentOS 7.3的部署安裝（RPM）

環(huán)境
server1: elk.test.com CentOS 7.3 192.168.81.11
server2: nginx.test.com CentOS 6.5 192.168.81.12
server3: rsyslog.test.com CentOS 6.5 192.168.81.13

1.基礎(chǔ)環(huán)境檢查

配置本地hosts文件

[root@elk ~]# cat /etc/hosts
192.168.81.11 elk.test.com
192.168.81.12 nginx.test.com
192.168.81.13 rsyslog.test.com

下載軟件包：

[root@elk ~]# cd /usr/local/src
[root@elk src]# wget -c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm
[root@elk src]# wget -c https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm
[root@elk src]# wget https://download.elastic.co/kibana/kibana/kibana-4.5.1-1.x86_64.rpm
[root@elk src]# wget -c https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm
2.安裝elasticsearch

安裝JAVA環(huán)境（jdk版本1.8.0及以上）

[root@elk src]# yum install java-1.8.0-openjdk -y
[root@elk src]# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

安裝elasticsearch

[root@elk src]# yum localinstall elasticsearch-2.3.3.rpm -y
[root@elk src]# systemctl daemon-reload
[root@elk src]# systemctl enable elasticsearch
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@elk src]# systemctl start elasticsearch

檢查服務(wù)狀態(tài)

[root@elk src]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-05-12 04:55:08 EDT; 8min ago
     Docs: http://www.elastic.co
  Process: 10661 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 10663 (java)
   CGroup: /system.slice/elasticsearch.service
           └─10663 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParN...

May 12 04:55:09 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:09,233][INFO...]
May 12 04:55:09 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:09,233][WARN...]
May 12 04:55:10 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:10,806][INFO...d
May 12 04:55:10 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:10,806][INFO....
May 12 04:55:10 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:10,920][INFO...}
May 12 04:55:10 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:10,926][INFO...w
May 12 04:55:14 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:14,010][INFO...)
May 12 04:55:14 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:14,028][INFO...}
May 12 04:55:14 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:14,028][INFO...d
May 12 04:55:14 elk.test.com elasticsearch[10663]: [2017-05-12 04:55:14,047][INFO...e
Hint: Some lines were ellipsized, use -l to show in full.
[root@elk src]# rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf
[root@elk src]# netstat -nltp | grep java
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      10663/java          
tcp6       0      0 ::1:9200                :::*                    LISTEN      10663/java          
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      10663/java          
tcp6       0      0 ::1:9300                :::*                    LISTEN      10663/java

開(kāi)放firewalld端口

[root@elk src]# firewall-cmd --zone=public --add-port=80/tcp --permanent
success
[root@elk src]# firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
success
[root@elk src]# firewall-cmd --reload
success
[root@elk src]# firewall-cmd  --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources: 
  services: dhcpv6-client ssh
  ports: 9200/tcp 80/tcp 9300/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules:
3.安裝kibana
[root@elk src]#  yum localinstall kibana-4.5.1-1.x86_64.rpm -y
[root@elk src]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
[root@elk src]# systemctl start kibana
[root@elk src]# systemctl status kibana
● kibana.service - no description given
   Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2017-05-12 05:16:28 EDT; 5min ago
 Main PID: 10978 (node)
   CGroup: /system.slice/kibana.service
           └─10978 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli

May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-12...
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:29 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
May 12 05:16:35 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-12...
May 12 05:16:38 elk.test.com kibana[10978]: {"type":"log","@timestamp":"2017-05-..."}
Hint: Some lines were ellipsized, use -l to show in full.

開(kāi)放防火墻端口

[root@elk src]# firewall-cmd --permanent --add-port=5601/tcp
success
[root@elk src]# firewall-cmd --reload
success
[root@elk src]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources: 
  services: dhcpv6-client ssh
  ports: 9200/tcp 3306/tcp 80/tcp 9300/tcp 5601/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules:
訪問(wèn)elkIP:5601驗(yàn)證kibana安裝

做端口轉(zhuǎn)發(fā)，將對(duì)端口80的訪問(wèn)轉(zhuǎn)發(fā)到5601上（方便訪問(wèn),非必須）

[root@elk src]# firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=5601
success
[root@elk src]# firewall-cmd --reload
success
4.安裝logstash，添加配置文件
[root@elk src]# yum localinstall logstash-2.3.2-1.noarch.rpm -y

生成證書

[root@elk tls]# openssl req -subj '/CN=elk.test.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
Generating a 2048 bit RSA private key
.........+++
.+++
writing new private key to 'private/logstash-forwarder.key'
-----

創(chuàng)建logstash的配置文件

[root@elk tls]# vim /etc/logstash/conf.d/01-logstash-initial.conf
input {
  beats {
    port => 5000
    type => "logs"
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog-beat" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    geoip {
      source => "clientip"
    }
    syslog_pri {}
    date {
      match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { }
  stdout { codec => rubydebug }
}

啟動(dòng)logstash檢查端口

[root@elk tls]# systemctl start logstash
[root@elk tls]# /sbin/chkconfig logstash on
[root@elk tls]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1761/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3306/master         
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      10978/node          
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      3491/php-fpm: maste 
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      10663/java          
tcp6       0      0 ::1:9200                :::*                    LISTEN      10663/java          
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      10663/java          
tcp6       0      0 ::1:9300                :::*                    LISTEN      10663/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1761/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      3306/master         
tcp6       0      0 :::5000                 :::*                    LISTEN      11110/java          
tcp6       0      0 :::3306                 :::*                    LISTEN      2135/mysqld

修改防火墻，開(kāi)放5000端口

[root@elk src]# firewall-cmd --permanent --add-port=5000/tcp
success
[root@elk src]# firewall-cmd --reload
success
5.修改elasticsearch配置文件
[root@elk elasticsearch]# mkdir es-01
[root@elk elasticsearch]# mv elasticsearch.yml es-01/
[root@elk elasticsearch]# mv logging.yml es-01/
[root@elk elasticsearch]# tree
.
├── es-01
│   ├── elasticsearch.yml
│   └── logging.yml
└── scripts

2 directories, 2 files

[root@elk elasticsearch]# cat es-01/elasticsearch.yml
http:
  port: 9200
network:
  host: elk.test.com
node:
  name: elk.test.com
path:
  data: /etc/elasticsearch/data/es-01

重啟elasticsearch和logstash服務(wù)

[root@elk elasticsearch]# systemctl restart elasticsearch
[root@elk elasticsearch]# systemctl restart logstash
6.將fiebeat安裝包拷貝到rsyslog、nginx客戶端上
[root@elk elk]# scp filebeat-1.2.3-x86_64.rpm root@rsyslog.test.com:/root/elk
[root@elk elk]# scp filebeat-1.2.3-x86_64.rpm root@nginx.test.com:/root/elk
[root@elk elk]# scp /etc/pki/tls/certs/logstash-forwarder.crt rsyslog.test.com:/root/elk
[root@elk elk]# scp /etc/pki/tls/certs/logstash-forwarder.crt nginx.test.com:/root/elk
7.在rsyslog上安裝filebeat
[root@rsyslog src]# yum localinstall filebeat-1.2.3-x86_64.rpm -y
[root@rsyslog src]# cp logstash-forwarder.crt /etc/pki/tls/certs/.
[root@rsyslog src]# cd /etc/filebeat/
[root@rsyslog filebeat]# tree
.
├── conf.d
│   ├── authlogs.yml
│   └── syslogs.yml
├── filebeat.template.json
└── filebeat.yml
[root@rsyslog filebeat]# vim filebeat.yml
filebeat:
  spool_size: 1024
  idle_timeout: 5s
  registry_file: .filebeat
  config_dir: /etc/filebeat/conf.d
output:
  logstash:
    hosts:
    - elk.test.com:5000
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
    enabled: true
shipper: {}
logging: {}
runoptions: {}
[root@rsyslog filebeat]# vim conf.d/authlogs.yml
filebeat:
  prospectors:
    - paths:
      - /var/log/secure
      encoding: plain
      fields_under_root: false
      input_type: log
      ignore_older: 24h
      document_type: syslog-beat
      scan_frequency: 10s
      harvester_buffer_size: 16384
      tail_files: false
      force_close_files: false
      backoff: 1s
      max_backoff: 1s
      backoff_factor: 2
      partial_line_waiting: 5s
      max_bytes: 10485760
[root@rsyslog filebeat]# vim conf.d/syslogs.yml
filebeat:
  prospectors:
    - paths:
      - /var/log/messages
      encoding: plain
      fields_under_root: false
      input_type: log
      ignore_older: 24h
      document_type: syslog-beat
      scan_frequency: 10s
      harvester_buffer_size: 16384
      tail_files: false
      force_close_files: false
      backoff: 1s
      max_backoff: 1s
      backoff_factor: 2
      partial_line_waiting: 5s
      max_bytes: 10485760
8.修改完成，啟動(dòng)filebeat服務(wù)（時(shí)間要一致，否則報(bào)X509錯(cuò)誤）
[root@rsyslog filebeat]# service filebeat start
Starting filebeat:                                         [確定]
[root@rsyslog filebeat]# chkconfig filebeat on
9.安裝Nginx在nginx上安裝filebeat,拷貝證書，創(chuàng)建收集日志配置文件
[root@nginx ~]# yum -y install nginx-1.8.0-1.el6.ngx.x86_64.rpm
[root@nginx ~]# service nginx start
Starting nginx:                                            [  OK  ]

[root@nginx src]# yum localinstall filebeat-1.2.3-x86_64.rpm -y
[root@nginx src]# cp logstash-forwarder.crt /etc/pki/tls/certs/.
[root@nginx src]# cd /etc/filebeat/
....
....
(配置同上7、8)
10.這時(shí)filebeat進(jìn)程已經(jīng)和elk服務(wù)器連接了，瀏覽器訪問(wèn)elkIP驗(yàn)證
創(chuàng)建
檢查所有日志
檢查nginx服務(wù)器的日志
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容