spark

很多時(shí)候我們需要把一些參數(shù)保存在文件里，等程序運(yùn)行時(shí)動(dòng)態(tài)加載，分享兩種scala調(diào)用的方式：1.Typesafe的Config庫 2.java.util.Properties類簡(jiǎn)單介紹其一些用法

1.Typesafe的Config庫

純Java寫成、零外部依賴、代碼精簡(jiǎn)、功能靈活、API友好。支持Java properties、JSON、JSON超集格式HOCON以及環(huán)境變量。它也是Akka的配置管理庫。

maven依賴

<!-- https://mvnrepository.com/artifact/com.typesafe/config -->
    <dependency>
      <groupId>com.typesafe</groupId>
      <artifactId>config</artifactId>
      <version>1.4.0</version>
    </dependency>

概述

• 純java實(shí)現(xiàn)，無任何依賴
• 充分的測(cè)試
• 支持: Java properties, JSON, and a human-friendly JSON superset
• 可以合并各種格式的配置文件
• 可以通過文件、urls、classpath加載配置
• 支持多層嵌套的配置方式
• 識(shí)別Java system properties, 如java -Dmyapp.foo.bar=10
• 可以轉(zhuǎn)換長(zhǎng)短，大小等單位。如配置文件中timeout=10s,則可以轉(zhuǎn)換成任意的毫秒或者
• 類型轉(zhuǎn)換，比如yes可以轉(zhuǎn)換為boolean類型的true

默認(rèn)加載classpath下的application.conf,application.json和application.properties文件。通過ConfigFactory.load()加載。

java實(shí)現(xiàn)方式

import com.typesafe.config.Config;
import com.typesafe.config.ConfigFactory;

public class CommonConfig {

    private Config config;

    //指定配置文件
    public CommonConfig(Config config) {
        this.config = config;
        config.checkValid(ConfigFactory.defaultReference(), "simple-lib");
    }

    // 默認(rèn)加載classpath下的application.*
    public CommonConfig() {
        this(ConfigFactory.load());
    }

    //打印
    public void printSetting(String path) {
        System.out.println("The setting '" + path + "' is: " + config.getString(path));
    }

    public static void main(String[] args) {
        CommonConfig s = new CommonConfig();
        s.printSetting("simple-app.answer");
    }
}

scala實(shí)現(xiàn)方式

1.工具類

package com.suning.sospdm.utils

import java.io.FileNotFoundException
import java.util

import com.typesafe.config.{Config, ConfigFactory}
import org.apache.commons.collections.CollectionUtils
import org.apache.commons.lang3.{ArrayUtils, StringUtils}

/**
  * Description: 
  * Author: 18051165 WuYu 
  * Date: 2019/12/30 14:58
  */
object ConfigLoader {
  private var config:Config = _
  private val PROFILES_ACTIVE_KEY = "app.profiles.active"

  /**
    * 獲取配置信息
    * @return Config對(duì)象
    */
  def getConfig:Config = {
    if (config == null || config.isEmpty) {
      // 默認(rèn)方式
      var cfg = ConfigFactory.load()
      // 配置處理
      if(cfg.hasPath(PROFILES_ACTIVE_KEY)){
        val activeProfiles = cfg.getAnyRef(PROFILES_ACTIVE_KEY)
        activeProfiles match {
          case file: String =>
            if (!StringUtils.isEmpty(file)) {
              try {
                cfg = ConfigFactory.load(file)
              } catch {
                case e: FileNotFoundException => println(e.getMessage)
              }
            }
          case files: util.List[String] =>
            if (!CollectionUtils.isEmpty(files)) {
              val itr = files.iterator()
              while (itr.hasNext) {
                try {
                  val f = itr.next()
                  val tmpConfig = ConfigFactory.load(f)
                  cfg = cfg.withFallback(tmpConfig)
                } catch {
                  case e: FileNotFoundException => println(e.getMessage)
                }
              }
            }
          case _ =>
        }
      }
      cfg
    } else {
      config
    }
  }

  /**
    * 刷新config
    * @return Unit
    */
  def refresh():Unit = {config = ConfigFactory.load()}
}

ConfigFactory.load()會(huì)加載配置文件，默認(rèn)加載classpath下的application.conf,application.json和application.properties文件。
也可以調(diào)用ConfigFactory.load(confFileName)加載指定的配置文件。

2.application.json文件

{
  "app": {
    "profiles":{
      "active": ["./config/dev0.json","./config/dev1.json"]
    },
    "name": "template",
    "master": "local",
    "spark": {
      "config": {
        "testing": {
          "memory": "512000000"
        },
        "hive": {
          "enable": false
        }
      }
    },
    "resources":{
      "data": "E:\\Spark\\myAlgorithmDemo\\src\\main\\resources\\data\\iris.data",
      "model": "E:\\Spark\\myAlgorithmDemo\\src\\main\\resources\\model"
    }
  }
}

3.實(shí)例

package com.suning.sospdm.utils

import com.suning.sospdm.utils.ConfigLoader
import com.typesafe.config.Config

/**
  * Description: 
  * Author: 18051165 WuYu 
  * Date: 2019/12/30 15:27
  */
object ConfigTest {
  def main(args: Array[String]): Unit = {
    val c = ConfigLoader.getConfig
    // 默認(rèn)地址讀取配置
    printConfig("app.name", c)
    printConfig("app.resources.data", c)
    // 指定地址讀取
    printConfig("app.test0", c)
    printConfig("app.test1", c)
  }

  def printConfig(configName:String, c:Config): Unit = {
    println("The setting " + configName + " is: " + c.getString(configName))
  }
}

結(jié)果如下，在dev0和dev1中可以實(shí)現(xiàn)相同key，不同value的調(diào)用，方便不同環(huán)境使用不同的參數(shù)，而不用修改其他代碼

The setting app.name is: config test
The setting app.resources.data is: E:\Spark\myAlgorithmDemo\src\main\resources\data\iris.data
The setting app.test0 is: this is dev0 config
The setting app.test1 is: this is dev1 config

2.java.util.Properties

Properties是一個(gè)鍵值對(duì)存儲(chǔ)容器，且鍵和值的類型都是String，其中鍵不會(huì)重復(fù)，文件中后面的會(huì)覆蓋前面的同名鍵值對(duì)。

*.properties例子

spark.enableHiveSupport = false
spark.memory = 512000000
dataPath = E:/Spark/myAlgorithmDemo/src/main/resources/data
modelPath = E:/Spark/myAlgorithmDemo/src/main/resources/model

scala實(shí)現(xiàn)

1.工具類

package com.suning.sospdm.utils

import java.io.InputStream
import java.util.Properties

import org.slf4j.{Logger, LoggerFactory}

/**
  * Description: 參數(shù)配置
  * Author: 18051165 WuYu 
  * Date: 2019/11/10 15:01
  */
object PropertyUtils extends Serializable {
  val log: Logger = LoggerFactory.getLogger(this.getClass)
  var p: Properties = _

  def apply(): Properties = {
    p = new Properties
    val fileNames = PropertyFileNameEnum.values
    for (fileName <- fileNames) {
      loadProperties(fileName.toString)
    }
    p
  }

  def loadProperties(fileName: String): Unit = {
    var is: InputStream = null
    try {
      is = this.getClass.getResourceAsStream(fileName)
      p.load(is)
    } catch {
      case e: Exception =>
        throw new RuntimeException(e)
    } finally {
      if (null != is)
        try
          is.close()
        catch {
          case e: Exception => throw new RuntimeException(e)
        }
    }
  }

  private[utils] object PropertyFileNameEnum extends Enumeration {
    type PropertyFileNameEnum = Value
    val SPARK: PropertyUtils.PropertyFileNameEnum.Value = Value("/spark.properties")
  }
}

默認(rèn)情況下可以直接讀取resources下*.properties文件，在pom文件中配置maven工具可以設(shè)置不同環(huán)境，在打包的過程中可以分配配置不同環(huán)境下的參數(shù)，方便實(shí)際工程應(yīng)用

2.pom文件配置

  <build>
    <resources>
      <resource>
        <directory>src/main/resources</directory>
        <excludes>
          <exclude>dev/*</exclude>
          <exclude>pre/*</exclude>
          <exclude>prd/*</exclude>
          <exclude>*.properties</exclude>
        </excludes>
      </resource>
      <resource>
        <directory>src/main/resources/${profiles.active}</directory>
      </resource>
    </resources>
  </build>
  <!-- 打包命令:clean compile install assembly:assembly -P pre -->
  <profiles>
    <profile>
      <id>dev</id>
      <properties>
        <!-- 默認(rèn)本地環(huán)境 -->
        <profiles.active>dev</profiles.active>
      </properties>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
    </profile>
    <profile>
      <id>pre</id>
      <properties>
        <!-- 其他環(huán)境 pre -->
        <profiles.active>pre</profiles.active>
      </properties>
    </profile>
    <profile>
      <id>prd</id>
      <properties>
        <!-- 其他環(huán)境 生產(chǎn)環(huán)境prd -->
        <profiles.active>prd</profiles.active>
      </properties>
    </profile>
  </profiles>

pom文件配置好后，IDEA的Maven工具就會(huì)如下顯示：

IDEA顯示截圖

3.實(shí)例

package com.suning.sospdm.utils

import java.util.Properties
import com.suning.sospdm.utils.PropertyUtils

/**
  * Description: 
  * Author: 18051165 WuYu 
  * Date: 2019/12/30 16:34
  */
object PropertyTest {
  val prop: Properties = PropertyUtils()
  
  def main(args: Array[String]): Unit = {
    printConfig("profiles", prop)
    printConfig("spark.enableHiveSupport", prop)
    printConfig("spark.memory", prop)
  }

  def printConfig(propertyName:String, p:Properties): Unit = {
    println("The setting " + propertyName + " is: " + p.getProperty(propertyName))
  }
}

Maven Profiles默認(rèn)dev環(huán)境的結(jié)果：

The setting profiles is: dev
The setting spark.enableHiveSupport is: false
The setting spark.memory is: 512000000

Maven Profiles選擇prd環(huán)境的結(jié)果：

The setting profiles is: prd
The setting spark.enableHiveSupport is: true
The setting spark.memory is: 2048000000