在寫(xiě)Spark代碼時(shí), 發(fā)現(xiàn)有個(gè)場(chǎng)景需要合并兩個(gè)List / Array / Seq為一個(gè):
即: 合并兩個(gè)Array[String]

在這里插入圖片描述

輸入: Array("1","2","3","4") Array("5","6","7","8")
輸出: Array("1","2","3","4","5","6","7","8")

或者:

輸入: ["1","2","3","4"] ["5","6","7","8"]
輸出: ["1","2","3","4","5","6","7","8"]

但是Spark中沒(méi)有實(shí)現(xiàn)類(lèi)似功能的算子, 于是自己寫(xiě)了一個(gè)UDF去實(shí)現(xiàn), 話(huà)不多說(shuō), 貼代碼供大家參考.

UDF代碼:

object UdfUtils {
  def udfUnionString(): UserDefinedFunction = {
    def unionString(arrayOne: Seq[String], arrayTwo: Seq[String]): Array[String] = {
      if ((arrayOne == null || arrayOne.isEmpty) && (arrayTwo == null || arrayTwo.isEmpty)) {
        Array[String]()
      } else if (arrayOne == null || arrayOne.isEmpty) {
        arrayTwo.toArray
      } else if (arrayTwo == null || arrayTwo.isEmpty) {
        arrayOne.toArray
      } else {
        (arrayOne ++ arrayTwo).toArray
      }
    }

    udf((arrayOne: Seq[String], arrayTwo: Seq[String]) =>
      unionString(arrayOne, arrayTwo),
      ArrayType(StringType)
    )
  }
}

調(diào)用UDF代碼:

testDF.select(
    UdfUtils.udfUnionString()(testDf1("array1"),testDf2("array2"))
)

示例:

testDf1("array1"):
    ["1","2","3","4"] 
    ["a1","b1","c1","d1"]
    ...

testDf2("array2"):
    ["5","6","7","8"] 
    ["a2","b2","c2","d2"]
    ...
    
結(jié)果:
    ["1","2","3","4","5","6","7","8"]
    ["a1","b1","c1","d1","a2","b2","c2","d2"]