Kubernetes ELLK
本次方案是按照 Elasticsearch + Logspout + Logstash + kibana 并且手機(jī)的日志可以被簡單切分：
Elasticsearch-rc 配置文件：
apiVersion: v1
kind: ReplicationController
metadata:
??name: elasticsearch-logging-v1
??labels:
????k8s-app: elasticsearch-logging
????version: v1
????kubernetes.io/cluster-service:?"true"
spec:
??replicas: 1
??selector:
????k8s-app: elasticsearch-logging
????version: v1
??template:
????metadata:
??????labels:
????????k8s-app: elasticsearch-logging
????????version: v1
????????kubernetes.io/cluster-service:?"true"
????spec:
??????nodeSelector:
????????role: elk
??????containers:
??????- image: registry.aliyuncs.com/slzcc/elasticsearch
????????name: elasticsearch
????????resources:
??????????limits:
????????????cpu: 1000m
??????????requests:
????????????cpu: 100m
????????ports:
????????- containerPort: 9200
??????????name: db
??????????protocol: TCP
????????- containerPort: 9300
??????????name: transport
??????????protocol: TCP
????????volumeMounts:
????????- name: es-persistent-storage
??????????mountPath:?"/usr/share/elasticsearch/data"
??????volumes:
??????- name: es-persistent-storage
????????hostPath:
??????????path:?"/data/elasticsearch"
Elasticsearch-svc 配置文件：
apiVersion: v1
kind: Service
metadata:
??name: elasticsearch-logging
??labels:
????k8s-app: elasticsearch-logging
????kubernetes.io/cluster-service:?"true"
????kubernetes.io/name:?"Elasticsearch"
spec:
??ports:
??- port: 9200
????name: http
????protocol: TCP
????targetPort: db
??selector:
????k8s-app: elasticsearch-logging
Kibana-rc 配置文件：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
??name: kibana-logging
??labels:
????k8s-app: kibana-logging
????kubernetes.io/cluster-service:?"true"
spec:
??replicas: 1
??selector:
????matchLabels:
??????k8s-app: kibana-logging
??template:
????metadata:
??????labels:
????????k8s-app: kibana-logging
????spec:
??????nodeSelector:
????????role: elk
??????containers:
??????- name: kibana-logging
????????image: registry.aliyuncs.com/slzcc/kibana
????????resources:
??????????# keep request = limit to keep this container in guaranteed class
??????????limits:
????????????cpu: 100m
??????????requests:
????????????cpu: 100m
????????env:
??????????- name:?"ELASTICSEARCH_URL"
????????????value:?"http://elasticsearch-logging:9200"
????????ports:
????????- containerPort: 5601
??????????name: ui
??????????protocol: TCP
Kibana-svc 配置文件：
apiVersion: v1
kind: Service
metadata:
??name: kibana-logging
??labels:
????k8s-app: kibana-logging
????kubernetes.io/cluster-service:?"true"
????kubernetes.io/name:?"Kibana"
spec:
??ports:
??- port: 5601
????name: http
????protocol: TCP
????targetPort: ui
??selector:
????k8s-app: kibana-logging
Logstash-configmap 配置文件：
apiVersion: v1
kind: ConfigMap
metadata:
??name: logstash
data:
??logstash.conf: |-
??????input {
????????udp {
????????port => 514
????????type?=> syslog
????????codec => json
????????}
????????tcp {
????????port => 514
????????type?=> syslog
????????codec => json
????????}
??????}
??????filter {
????????if?[type] ==?"syslog"?{
??????????grok {
????????????match => {?"message"?=>?"%{SYSLOG5424PRI}%{NONNEGINT:ver} +(?:%{TIMESTAMP_ISO8601:ts}|-) +(?:%{HOSTNAME:containerid}|-) +(?:%??? {NOTSPACE:containername}|-) +(?:%{NOTSPACE:proc}|-) +(?:%{WORD:msgid}|-) +(?:%{SYSLOG5424SD:sd}|-|) +%{GREEDYDATA:msg}"?}
??????????}
??????????syslog_pri { }
??????????date?{
??????????????match => [?"syslog_timestamp",?"MMM? d HH:mm:ss",?"MMM dd HH:mm:ss"?]
??????????}
??????????if?!("_grokparsefailure"?in?[tags]) {
??????????????mutate {
????????????????replace => [?"@source_host",?"%{syslog_hostname}"?]
????????????????replace => [?"@message",?"%{syslog_message}"?]
??????????????}
??????????}
??????????mutate {
????????????remove_field => [?"syslog_hostname",?"syslog_message",?"syslog_timestamp"?]
??????????}
????????}
??????}
??????output {
??????????elasticsearch {
??????????????hosts => ["elasticsearch-logging:9200"]
??????????????index =>?"k8s-%{type}-%{+YYYY.MM.dd}"
??????????????document_type =>?"%{type}"
??????????????workers => 1
??????????????flush_size => 20000
??????????????idle_flush_time => 10
??????????????template_overwrite =>?true
??????????????codec => json
??????????}
??????}
Logstash-rc 配置文件：
apiVersion: v1
kind: ReplicationController
metadata:
??name: logstash
??labels:
????k8s-app: logstash
????version: v1
????kubernetes.io/cluster-service:?"true"
spec:
??replicas: 1
??selector:
????k8s-app: logstash
????version: v1
??template:
????metadata:
??????labels:
????????k8s-app: logstash
????????version: v1
????????kubernetes.io/cluster-service:?"true"
????spec:
??????nodeSelector:
????????role: elk
??????containers:
??????- image: registry.aliyuncs.com/slzcc/logstash-build
????????name: logstash
????????resources:
??????????limits:
????????????cpu: 1000m
??????????requests:
????????????cpu: 100m
????????ports:
????????- containerPort: 514
??????????name: input
??????????protocol: TCP
????????- containerPort: 514
??????????name: output
??????????protocol: UDP
????????command:
????????-?'/logstash-5.1.1/bin/logstash'
????????-?'-f'
????????-?'/etc/logstash/logstash.conf'
????????-?'-w 20'
????????volumeMounts:
??????????- mountPath:?"/etc/logstash/"
????????????name: config-volume
??????volumes:
????????- name: config-volume
??????????configMap:
????????????name: logstash?
Logstash-svc 配置文件：
apiVersion: v1
kind: Service
metadata:
??name: logstash
??labels:
????k8s-app: logstash
????kubernetes.io/cluster-service:?"true"
????kubernetes.io/name:?"logstash"
spec:
??ports:
??- port: 514
????name: input
????protocol: TCP
????targetPort: input
#? - port: 514
#??? name: output
#??? protocol: UDP
#??? targetPort: output
??selector:
????k8s-app: logstash
??clusterIP: None
Lospout-daemon 配置文件：
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
??name: logspout-elasticsearch
??labels:
????k8s-app: logspout-logging
spec:
??template:
????metadata:
??????labels:
????????name: logspout-elasticsearch
????spec:
??????containers:
??????nodeSelector:
????????role: elk
??????- name: logspout-elasticsearch
????????image: registry.aliyuncs.com/slzcc/logspout-logstash
????????resources:
??????????limits:
????????????memory: 200Mi
??????????requests:
????????????cpu: 100m
????????????memory: 200Mi
????????env:
??????????- name:?"ROUTE_URIS"
????????????value:?"logstash+tcp://logstash:514"
????????volumeMounts:
??????????- mountPath:?"/var/run/docker.sock"
????????????name: sock
??????volumes:
????????- hostPath:
????????????path:?"/var/run/docker.sock"
??????????name: sock
??????terminationGracePeriodSeconds: 30
