在使用Scala將DataFrame的數(shù)據(jù)放入ES中時(shí)

1.建立連接時(shí)發(fā)生錯(cuò)誤

Caused by: java.lang.IllegalStateException: availableProcessors is already set to [8], rejecting [8]

解決方法

配置中加入
System.setProperty("es.set.netty.runtime.available.processors", "false")

    // 新建es配置

    val settings: Settings = Settings.builder().put("cluster.name", eSConfig.clusterName).build()

    // 新建一個(gè)es客戶(hù)端
    System.setProperty("es.set.netty.runtime.available.processors", "false")

    //解決netty沖突 ---  es的和spring的netty版本沖突
    val esClient = new PreBuiltTransportClient(settings)

2.寫(xiě)入數(shù)據(jù)時(shí)發(fā)生錯(cuò)誤

Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: resource must 

查看日志

Caused by: java.lang.IllegalArgumentException: Invalid format: "null"
    at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[joda-time-2.9.5.jar:2.9.5]
    at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[joda-time-2.9.5.jar:2.9.5]
    at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:240) ~[elasticsearch-5.6.2.jar:5.6.2]
    at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:465) ~[elasticsearch-5.6.2.jar:5.6.2]
    at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.6.2.jar:5.6.2]
    ... 36 more

原因：在數(shù)據(jù)預(yù)處理時(shí)對(duì)數(shù)據(jù)空值采用null來(lái)填補(bǔ)，es在緩存過(guò)程中Java識(shí)別到null關(guān)鍵字發(fā)生錯(cuò)誤，解決方法為將null字段修改為別的填充數(shù)據(jù)

3.org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [XXX:9200] returned Bad Request(400) - failed to parse [XXX]; Bailing out..

常見(jiàn)錯(cuò)誤為es中的字段值錯(cuò)誤，我在開(kāi)發(fā)過(guò)程中的錯(cuò)誤為 failed to parse [release_date]

格式為2000-01-01
2000-01-01：String類(lèi)型存儲(chǔ)到es中會(huì)自動(dòng)轉(zhuǎn)換為date類(lèi)型，但是在數(shù)據(jù)預(yù)處理時(shí)對(duì)于沒(méi)有 release_date的字段設(shè)置為空，因此轉(zhuǎn)換出錯(cuò)
解決方法：設(shè)置一個(gè)默認(rèn)值 比如 2000-01-01