利用Logstash同步MySql數(shù)據(jù)至ElasticSearch

安裝Logstash

參考鏈接Installing Logstash
Download and install the public signing key:

rpm --importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

[logstash-7.x]name=Elastic repository for 7.x packages

baseurl=https://artifacts.elastic.co/packages/7.x/yum

gpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1autorefresh=1type=rpm-md

And your repository is ready for use. You can install it with:

sudo yum install logstash

yum方式安裝好后，程序目錄位于/usr/share/logstash/, 配置文件位于/etc/logstash/，

配置MySql數(shù)據(jù)同步

在/etc/logstash中創(chuàng)建mysql.conf配置文件,參考鏈接jdbc input plugin

input {
  jdbc {
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://xxx.mysql.rds.aliyuncs.com/xxx"
    jdbc_user => "root"
    jdbc_password => "******"
    schedule => "* * * * *"
    statement => "SELECT * from biz_goods  where create_time > :sql_last_value AND create_time < NOW() AND is_delete=0 ORDER BY create_time desc"
  }
}

output {
    elasticsearch {
         hosts => ["127.0.0.1:9200"]
         index => "goods"
         document_id => "%{id}"
    }
}

在官方文檔中input->jdbc中需要指定jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"，此處如果將mysq-connector-java.jar放在常規(guī)目錄通過路徑指定，將會(huì)出現(xiàn)如下錯(cuò)誤

LogStash::ConfigurationError
com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?

解決方式為在mvnrepository下載jar文件后，將該文件拷貝至/usr/share/logstash/logstash-core/lib/jars 中，然后無需在input->jdbc中配置jdbc_driver_library

運(yùn)行l(wèi)ogstash開啟同步

./logstash -f /etc/logstash/mysql.conf