docker安裝mysql、logstash同步數(shù)據(jù)到es

一、安裝mysql

• 1.拉取鏡像

docker pull mysql:5.7.26

• 2.啟動(dòng)

docker run --name mysql -e MYSQL_ROOT_PASSWORD=123456 -d -i -p 3306:3306 mysql:5.7.26

• 3.授予遠(yuǎn)程訪問權(quán)限

docker exec -it mysql bash
mysql -uroot -p123456

use mysql;
select host,user from user;
ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
flush privileges;

• 4.開啟mysql binlog

docker exec -it mysql bash
apt-get update
apt-get install vim
vim etc/mysql/mysql.conf.d/mysqld.cnf

在mysqld配置項(xiàng)下添加如下幾行內(nèi)容

log-bin=/var/lib/mysql/mysql-bin
server-id=1
binlog-format=ROW

mysqld.cnf

重啟mysql。

• 5.查看binlog是否開啟，數(shù)據(jù)庫(kù)客戶端執(zhí)行如下命令

show variables like '%log_bin%';

log_bin

• 6.創(chuàng)建mysql賬戶，用于讓canal同步數(shù)據(jù)

use mysql;
create user canal identified by 'canal';
grant select,replication slave,replication client on *.* to 'canal'@'%';
flush privileges;

• 7.創(chuàng)建測(cè)試用數(shù)據(jù)庫(kù)

create database test;
use test;
CREATE TABLE `user` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(50) NOT NULL,
  `age` int(11) NOT NULL,
  `createtime` datetime NOT NULL,
  `lastupdatetime` datetime NOT NULL,
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
INSERT INTO `test`.`user`(`name`, `age`, `createtime`, `lastupdatetime`) VALUES ('jack', 18, '2022-03-19 10:27:12', '2022-03-19 10:27:12');
INSERT INTO `test`.`user`( `name`, `age`, `createtime`, `lastupdatetime`) VALUES ( 'William', 18, '2022-03-19 10:27:12', '2022-03-19 10:27:12');

二、安裝logstash

• 1.拉取鏡像

docker pull logstash:7.17.0

• 2.啟動(dòng)

docker run -d -p 5044:5044 -p 9600:9600 -it --name logstash  logstash:7.17.0

• 3.修改配置文件

docker exec -it -u root logstash bash
apt-get update
apt-get install vim

修改logstash.yml

vim config/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.url: http://192.168.0.103:9200

修改log4j2.properties

vim config/log4j2.properties

在末尾增加如下兩項(xiàng)

logger.elasticsearchoutput.name = logstash.outputs.elasticsearch
logger.elasticsearchoutput.level = debug

vim config/pipelines.yml

相關(guān)值修改如下

- pipeline.id: logstash  
path.config: "/usr/share/logstash/config/*.conf"  
pipeline.workers: 3

修改mysql.conf，沒有的話會(huì)創(chuàng)建

vim config/mysql.conf

將如下內(nèi)容粘貼進(jìn)去

input {
   stdin {
   }
   jdbc {
     jdbc_connection_string => "jdbc:mysql://192.168.0.103:3306/test"
     jdbc_user => "canal"
     jdbc_password => "canal"
     jdbc_driver_library => "/usr/share/logstash/config/mysql-connector-java-5.1.27-bin.jar" # 這個(gè)jar文件需要下載
     jdbc_driver_class => "com.mysql.jdbc.Driver"
     statement => "SELECT * FROM user"
     type => "project"
   }
 }
 
 filter {
   json {
     source => "message"
     remove_field => ["message"]
   }
 }
 
 output {
   elasticsearch {
     hosts => "http://192.168.0.103:9200"
     index => "project"
     document_id => "%{id}"
   }
 
   stdout {
     codec => json_lines
   }
 }

mysql-connector-java下載

apt-get install wget
wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.27.tar.gz
tar zxvf mysql-connector-java-5.1.27.tar.gz

將解壓后的.jar文件放到config目錄下即可。

重啟logstash。

• 4.執(zhí)行加載配置項(xiàng)

docker exec -it logstash bash
bin/logstash -f config/mysql.conf

kibana

注：Logstash could not be started because there is already another instance usining...如果報(bào)這個(gè)錯(cuò)誤，就進(jìn)入logstash終端，找到data目錄，ls -all 展示所有文件目錄，把里面的.lock文件刪除即可。

• 5.測(cè)試

查詢

命中結(jié)果

其他文章請(qǐng)看：
docker安裝kibana
docker安裝elasticsearch