軟件版本：elasticsearch版本（6.3.2）
數(shù)據(jù)遷移方案
datax導(dǎo)入數(shù)據(jù)到elasticsearch步驟：
- 安裝elasticsearchwriter插件，安裝教程：
- 從datax的github地址（https://github.com/alibaba/DataX
）下載源碼工程（Datax-master） - 修改pom.xml，配置modules模塊,按需保留elasticsearchwriter模塊
- 從datax的github地址（https://github.com/alibaba/DataX
<modules>
<module>common</module>
<module>core</module>
<module>transformer</module>
<!-- reader -->
<module>mysqlreader</module>
<!-- writer -->
<module>elasticsearchwriter</module>
<!-- common support module -->
<module>plugin-rdbms-util</module>
<module>plugin-unstructured-storage-util</module>
<module>hbase20xsqlreader</module>
<module>hbase20xsqlwriter</module>
</moudles>
打包，mvn clean install -Dmaven.test.skip=true，復(fù)制當(dāng)前目錄下的/elasticsearchwriter/target/datax/plugin/writer/elasticsearchwriter目錄到datax的plugin目錄下
配置datax腳本：
參數(shù)含義參照：
https://help.aliyun.com/knowledge_detail/61990.html
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"percentage": 0
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "xxx",
"password": "xxx",
"connection": [{
"querySql": ["數(shù)據(jù)查詢SQL "],
"jdbcUrl": ["數(shù)據(jù)庫(kù)地址"]
}]
}
},
"writer": {
"name": "elasticsearchwriter",
"parameter": {
"endpoint":"es的地址：xxx:9200",
"accessId": "連接權(quán)限配置，如果不需要權(quán)限認(rèn)證，可隨意配置，但不能為空，或者不配",
"accessKey": "連接權(quán)限配置，如果不需要權(quán)限認(rèn)證，可隨意配置",
"index": "xxx索引",
"type": "_doc",
"cleanup": false,
"discovery":false,
"column": [
{
"name": "id",
"type": "long"
},
{
"name": "userId",
"type": "long"
},
{
"name": "memberId",
"type": "long"
},
{
"name": "receiveTime",
"type": "date"
},
{
"name": "couponTemplateId",
"type": "long"
},
{
"name": "couponStatus",
"type": "keyword"
},
{
"name": "consumeTime",
"type": "date"
},
{
"name": "platformId",
"type": "long"
},
{
"name": "orderId",
"type": "keyword"
},
{
"name": "startDate",
"type": "date"
},
{
"name": "endDate",
"type": "date"
},
{
"name": "lockDate",
"type": "date"
},
{
"name": "reduceAmount",
"type": "double"
},
{
"name": "prodId",
"type": "long"
},
{
"name": "orderAmount",
"type": "double"
},
{
"name": "activityId",
"type": "long"
},
{
"name": "activityType",
"type": "keyword"
},
{
"name": "createUser",
"type": "keyword"
},
{
"name": "updateUser",
"type": "keyword"
},
{
"name": "createDatetime",
"type": "date"
},
{
"name": "updateDatetime",
"type": "date"
}
]
}
}
}
]
}
}
問題：
經(jīng)DataX智能分析,該任務(wù)最可能的錯(cuò)誤原因是:
com.alibaba.datax.common.exception.DataXException:
Code:[Framework-02], Description:
[DataX引擎運(yùn)行過(guò)程出錯(cuò)，具體原因請(qǐng)參看DataX運(yùn)行結(jié)束時(shí)的錯(cuò)誤診斷信息 .].
- java.lang.IllegalArgumentException:
Preemptive authentication set without credentials provider
排查原因：
- datax腳本，elasticsearchwriter未配置accessId、accessKey，或者accessId、accessKey的值為空(""),如果es不需要權(quán)限認(rèn)證，可以設(shè)置accessId、accessKey為任意非空值。
ERROR RetryUtil - Exception when calling callable,
異常Msg:No Server is assigned to client to connect‘
io.searchbox.client.config.exception.NoServerConfiguredException:
No Server is assigned to client to connect
排查原因：elasticsearchwriter的配置參數(shù)discovery:true導(dǎo)致，應(yīng)修改為默認(rèn)值false，該屬性描述含義：?jiǎn)⒂霉?jié)點(diǎn)發(fā)現(xiàn)將(輪詢)并定期更新客戶機(jī)中的服務(wù)器列表，官方es插件在發(fā)文前聲明只在5.x的es版本號(hào)上測(cè)試，在6.x及以上的未給出聲明。
