pyspark的DataFrame，在直接保存為hive表時，如果字符串中帶有換行符，會導致?lián)Q行錯誤。以spark 3.0.0版本為例。我們向hive表保存1條包含換行符字符串的數(shù)據(jù)，統(tǒng)計行數(shù)時卻得到2行：

>>> df = spark.createDataFrame([(1,'hello\nworld')], ('id','msg'))
>>> df.write.format('hive').saveAsTable('test.newline0')
>>> spark.sql('SELECT COUNT(1) FROM test.newline0').show()
+--------+
|count(1)|
+--------+
| 2|
+--------+

這一問題的相關文檔我找了很久，最后發(fā)現(xiàn)是在Specifying storage format for Hive tables一節(jié)。直接使用hive格式保存時，底層是'textfile'且默認換行符是'\n'，因此自然會出現(xiàn)換行錯誤。可以通過以下代碼進行驗證：

>>> df.write.format('hive').option('fileFormat', 'textfile').option('lineDelim', '\x13').saveAsTable('test.newline1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/share/spark-3.0.0-bin-hadoop2.7/python/pyspark/sql/readwriter.py", line 868, in saveAsTable
self._jwrite.saveAsTable(name)
File "/usr/share/spark-3.0.0-bin-hadoop2.7/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in call
File "/usr/share/spark-3.0.0-bin-hadoop2.7/python/pyspark/sql/utils.py", line 137, in deco
raise_from(converted)
File "<string>", line 3, in raise_from
pyspark.sql.utils.IllegalArgumentException: Hive data source only support newline '\n' as line delimiter, but given: ?

解決的方法也很簡單，使用其他格式進行保存：

>>> df.write.format('hive').option('fileFormat', 'parquet').saveAsTable('test.newline1')
>>> spark.sql('SELECT COUNT(1) FROM test.newline1').show()
+--------+
|count(1)|
+--------+
| 1|
+--------+