@[TOC]
前言
在Mac上搭建Hadoop環(huán)境(1) — 虛擬機(jī)的安裝及SSH免密設(shè)置中，我們已經(jīng)配置好了節(jié)點(diǎn)之間基本的網(wǎng)絡(luò)環(huán)境，接下來(lái)，只要在master(MBP)上下載安裝hadoop即可。
下載hadoop
可以前往Apache Hadoop官網(wǎng)，選擇你想要的版本進(jìn)行下載。 這里，我選擇下載的是hadoop-2.7.7的binary版本。
安裝hadoop
在下載完hadoop-2.7.7.tar.gz后，將其解壓到你想安裝的目錄即可。 我將其解壓到 ==/opt== 文件夾下。
sudo tar -C /opt -xvf ~/Downloads/hadoop-2.7.7.tar.gz;
然后對(duì)目錄重命名并創(chuàng)建相應(yīng)的子目錄
sudo mv /opt/hadoop-2.7.7 /opt/hadoop;
mkdir /opt/hadoop/dfs;
mkdir /opt/hadoop/dfs/name;
mkdir /opt/hadoop/dfs/data;
mkdir /opt/hadoop/tmp;
配置hadoop
hadoop-env.sh
編輯 /opt/hadoop/etc/hadoop/hadoop-env.sh，設(shè)置JAVA_HOME值，如下
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home
yarn-env.sh
編輯 /opt/hadoop/etc/hadoop/yarn-env.sh，設(shè)置JAVA_HOME值，如下
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home
slaves
編輯 /opt/hadoop/etc/hadoop/slaves，加入之前設(shè)置的slave節(jié)點(diǎn)的hostname：
slave1
slave2
更新*.xml配置文件
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>hadoop.proxyuser.lestat.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.lestat.groups</name>
<value>*</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
將hadoop復(fù)制到slave1和slave1上
scp -r /opt/hadoop parallels@slave1:~/
scp -r /opt/hadoop parallels@slave2:~/
然后分別在slave節(jié)點(diǎn)執(zhí)行
sudo mv ~/hadoop /opt/hadoop;
sudo chown -R parallels:parallels /opt/hadoop
修改slave上的hadoop-env.sh和yarn-env.sh中的JAVA_HOME值
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64
添加hadoop環(huán)境變量
export PATH=$PATH:/opt/hadoop/bin:/opt/hadoop/sbin
啟動(dòng)hadoop
在啟動(dòng)之前，先執(zhí)行hadoop namenode -format 進(jìn)行格式化，
然后執(zhí)行 start-all.sh來(lái)啟動(dòng)hadoop集群，以下是我的啟動(dòng)日志
Lestats-MBP:~ lestat$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/03/24 09:57:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: namenode running as process 45208. Stop it first.
slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave1.out
slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-parallels-datanode-slave2.out
Starting secondary namenodes [master]
master: secondarynamenode running as process 45331. Stop it first.
19/03/24 09:57:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 45435. Stop it first.
slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave1.out
slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-parallels-nodemanager-slave2.out
然后可以執(zhí)行master上jps來(lái)查看啟動(dòng)的JVM。
Lestats-MBP:~ lestat$ jps
45331 SecondaryNameNode
45846 Jps
45208 NameNode
45435 ResourceManager
執(zhí)行slave上jps來(lái)查看啟動(dòng)的JVM。
如果jps命令找不到， 可以運(yùn)行sudo yum install java-1.8.0-openjdk-devel進(jìn)行安裝
[parallels@slave1 ~]$ jps
28832 Jps
26454 DataNode
26590 NodeManager
[parallels@slave2 ~]$ jps
26034 DataNode
26180 NodeManager
27931 Jps
這樣hadoop偽分布式集群的安裝就告一段落了。
測(cè)試
例如我們執(zhí)行一些hadoop fs的命令：
lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -mkdir /input
19/03/24 10:09:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Lestats-MBP:~ lestat$ hadoop fs -ls /
19/03/24 10:09:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - lestat supergroup 0 2019-03-24 10:09 /input
