hudi報(bào)錯(cuò): Unable to alter table. The following columns have types incompatible with the existing co...

hudi 0.8,使用insert_overwrite_table模式覆寫hudi表。之前寫入正常,但hive由1.x升級(jí)到2.7.2后,報(bào)下面的錯(cuò)誤:
Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions

Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `dw_t1`.`contact` REPLACE COLUMNS(`_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `id` bigint, `ts` bigint, `tag_sqlrqx1` bigint, `tag_y_testmulmetadat` double, `tag_sjsmc1` bigint, `tag_qgbq1` string, `tag_sjzb1` double, `tag_sjph1` ARRAY< string>, `tag_sqlbq1` bigint, `tag_sqlsz1` int ) cascade
        at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
        at org.apache.hudi.hive.HoodieHiveClient.updateTableDefinition(HoodieHiveClient.java:251)
        at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:206)
        at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:154)
        at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:108)
        ... 50 more
    Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions :
    tag_sqlrqx1,tag_sjsmc1,tag_qgbq1,tag_sjph1
        at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:385)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
        at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)

查閱資料后,發(fā)現(xiàn)此問(wèn)題是hive的問(wèn)題,和hudi無(wú)關(guān)

問(wèn)題原因:

而hive.metastore.disallow.incompatible.col.type.changes這個(gè)參數(shù),在Hive2.0后,默認(rèn)為True。即在字段類型轉(zhuǎn)換時(shí),不再允許顯式轉(zhuǎn)換(強(qiáng)制轉(zhuǎn)換),而是只能進(jìn)行隱式轉(zhuǎn)換,即由低級(jí)類型轉(zhuǎn)換為高級(jí)類型

解決方法:

在hive服務(wù)端的hive-site.xml文件中添加下面配置,然后重啟hive和hivemetastore服務(wù)

<property>
    <name>hive.metastore.  disallow.incompatible.col.type.changes</name>
    <value>false</value>
</property>

補(bǔ)充:自測(cè),在hive客戶端,執(zhí)行set hive.metastore.disallow.incompatible.col.type.changes=false后,再執(zhí)行REPLACE COLUMNS的sql,還是會(huì)報(bào)錯(cuò)。

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡(jiǎn)書系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容