什么是spark 的dynamic allocate

• 它允許你的spark 根據(jù) spark的運行時的負(fù)載情況動態(tài)增加或者減少executors 的個數(shù)

• 下面展示的是一張靜態(tài)資源分配下spark實際使用的資源與分配的資源圖

  
  

• 下面展示的是動態(tài)資源分配下spark 實際使用的資源與分配的資源 圖

  
  

什么場景下使用spark 的dynamic allocate

• 任何需要大的shuffle拉取的情況

  
  

開啟spark dynamic allocate 步驟設(shè)置

1. 設(shè)置環(huán)境變量

$ export HADOOP_CONF_DIR=/hadoop/hadoop-2.7XX/etc/hadoop
$ export HADOOP_HOME=/hadoop/hadoop-2.7XX
$ export SPARK_HOME=/hadoop/spark-2.4.0-bin-hadoop2.7
$ hds=(`cat ${HADOOP_CONF_DIR}/slaves` 'namenode1' 'namenode2') 
# 配置hadoop cluster 的所有機器的host 
# remeber to unset hds at the end

2. yarn文件配置

• bakup yarn-site.xml

  $ for i in ${hds[@]}  ; do echo $i ; ssh $i "cp ${HADOOP_CONF_DIR}/yarn-site.xml ${HADOOP_CONF_DIR}/yarn-site.xml.pre_spark_shuffle.bak"  ; done;
  $ for i in ${hds[@]}  ; do echo $i ; ssh $i "ls ${HADOOP_CONF_DIR} | grep pre_spark_shuffle.bak"  ; done;
  

• modify yarn-site.xml

  $ more ${HADOOP_CONF_DIR}/yarn-site.xml | grep -B 1 -A 2 "aux-services"
  

• output

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,spark_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
    <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  </property>
  

• broadcast yarn-site.xml

   $ for i in ${hds[@]} ; do echo $i ; scp ${HADOOP_CONF_DIR}/yarn-site.xml ${i}:${HADOOP_CONF_DIR}/ ; done ;
   $ for i in ${hds[@]} ; do echo $i ; ssh $i  "cat ${HADOOP_CONF_DIR}/yarn-site.xml | grep -B 1 -A 2 'aux-services' " ; done ;
  

• Check heapsize

  $ more ${HADOOP_CONF_DIR}/yarn-env.sh | grep "YARN_HEAPSIZE"
  

  • output

  YARN_HEAPSIZE=2000 # 根據(jù)實際情況調(diào)整
  

• check yarn class path

  $ yarn classpath | sed -r 's/:/\n/g'
  $ more ${HADOOP_CONF_DIR}/yarn-site.xml | grep "yarn.application.classpath"  
  # if finds nothing  , we can use default path $HADOOP_HOME/share/hadoop/yarn/
  

• check yarn shuffle jar

  $ find ${SPARK_HOME} -iname "*yarn-shuffle.jar" 
  # get result :  spark-2.4.0-yarn-shuffle.jar 
  

• copy yarn shuffle jar

  $ for i in ${hds[@]} ; do echo $i ; scp `find ${SPARK_HOME} -iname "*yarn-shuffle.jar"` ${i}:$HADOOP_HOME/share/hadoop/yarn/ ; done ;
  $ for i in ${hds[@]}  ; do echo $i ; ll -ltr $HADOOP_HOME/share/hadoop/yarn/   | grep shuffle  ; done ;
  

• Restart yarn

    $ bash $HADOOP_HOME/sbin/stop-yarn.sh
    $ bash $HADOOP_HOME/sbin/start-yarn.sh
  

• check application

  $ for i in ${hds[@]}  ; do echo $i ; ssh $i ". /etc/profile ; jps" | grep -i manager  ; done;
  

3. spark 配置

• spark-default.conf

  $ more ${SPARK_HOME}/conf/spark-defaults.conf
  

• and add the following entries:

   spark.dynamicAllocation.enabled true
   spark.shuffle.service.enabled  true
   spark.dynamicAllocation.minExecutors 1
   spark.dynamicAllocation.maxExecutors 4
   spark.dynamicAllocation.executorIdleTimeout 60
  

refrence

ppt
config