Hadoop下載

• Hadoop官網(wǎng)

  Hadoop官網(wǎng)

• 選擇的版本

  下載連接

Hadoop安裝配置

• 準(zhǔn)備內(nèi)容

  1.虛擬機3臺(centos6.9)

  2.配置在同一網(wǎng)段

• 服務(wù)器配置

  1.配置hosts(三臺機子同樣配置)

  192.168.0.101 node1
  192.168.0.102 node2
  192.168.0.103 node3
  

  2.配置JDK1.8(/etc/profile)

  JAVA_HOME=/usr/java/jdk1.8.0_171/
  PATH=$JAVA_HOME/bin:$PATH
  CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  export JAVA_HOME
  export PATH
  export CLASSPATH
  

  source /etc/profile
  

  3.設(shè)置hadoop用戶

  useradd hadoop && echo hadoop | passwd --stdin hadoop
  echo "hadoopALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
  su - hadoop
  

  4.安裝hadoop2.7

  安裝

  解壓安裝在/home/hadoop目錄下
  

  配置環(huán)境變量

  export HADOOP_HOME=/home/hadoop/hadoop/
  export PATH=$HADOOP_HOME/bin:$PATH
  

  創(chuàng)建目錄

  mkdir -p /home/hadoop/dfs/{name,data}
  mkdir -p /home/hadoop/tmp
  

  創(chuàng)建備份目錄

  mkdir -p /data/hdfs/{name,data}
  chown -R hadoop:hadoop /data/
  

設(shè)置ssh

• 設(shè)置主節(jié)點和其它節(jié)點(${username}指的是默認(rèn)登錄用戶名)

ssh-keygen -t rsa
ssh-copy-id ${username}@192.168.0.101
ssh-copy-id ${username}@192.168.0.102
ssh-copy-id ${username}@192.168.0.103

• 測試ssh登錄

ssh ${username}@192.168.0.101

修改hadoop配置文件(/home/hadoop/hadoop/etc/hadoop)

• hadoop-env.sh(配置JAVA_HOME)

# The java implementation to use.
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/java/jdk1.8.0_171/

• yarn-evn.sh(配置JAVA_HOME)

# some Java parameters
# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
export JAVA_HOME=/usr/java/jdk1.8.0_171/

• slaves(配置主機名)

node1
node2
node3

• core-site.xml

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://node1:9000</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>131072</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:/home/hadoop/tmp</value>
    <description>Abase for other temporary directories.</description>
  </property>
</configuration>

• hdfs-site.xml

<configuration>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>node1:9001</value>
    <description># 通過web界面來查看HDFS狀態(tài) </description>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/hadoop/dfs/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/home/hadoop/dfs/data</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
    <description># 每個Block有2個備份</description>
  </property>
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
</configuration>

• mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>node1:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>node1:19888</value>
  </property>
</configuration>

• yarn-site.xml

<?xml version="1.0"?>
<!-- yarn-site.xml  -->
<configuration>
  <!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>node1:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>node1:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>node1:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>node1:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>node1:8088</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
  </property>
</configuration>

• 復(fù)制hadoop到其它節(jié)點

scp -r /home/hadoop/hadoop/ 192.168.0.102:/home/hadoop/
scp -r /home/hadoop/hadoop/ 192.168.0.103:/home/hadoop/

初始化和運行

• 初始化(只在node1主節(jié)點運行)

/home/hadoop/hadoop/bin/hdfs namenode -format

yum install tree
tree /home/hadoop/dfs

• 啟動hadoop(hadoop權(quán)限)

/home/hadoop/hadoop/sbin/start-dfs.sh

查看進(jìn)程

ps aux | grep --color namenode
ps aux | grep --color datanode

• 關(guān)閉hadoop(hadoop權(quán)限)

/home/hadoop/hadoop/sbin/stop-dfs.sh

• 啟動yarn分布式計算框架

/home/hadoop/hadoop/sbin/start-yarn.sh starting yarn daemons

ps aux | grep --color resourcemanager
ps aux | grep --color nodemanager

• 簡易啟動/停止

/home/hadoop/hadoop/sbin/start-all.sh
/home/hadoop/hadoop/sbin/stop-all.sh

• 查看hdfs分布式文件系統(tǒng)狀態(tài)

/home/hadoop/hadoop/bin/hdfs dfsadmin -report

• 用網(wǎng)頁查看內(nèi)容

192.168.0.101:50070