JAVA環(huán)境配置

$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

mac查看Java的安裝位置信息

$  /usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home

SSH配置

文件和目錄的權(quán)限千萬別設(shè)置成chmod 777.這個(gè)權(quán)限太大了，不安全

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

如果沒有ssh公鑰,執(zhí)行下面命令

ssh-keygen -t rsa

開啟遠(yuǎn)程登錄

image.png

測試遠(yuǎn)程登錄是否開啟

ssh localhost

安裝hadoop

最終的安裝目錄在/usr/local/Cellar/hadoop/ 安裝的版本是3.1.2

$ brew install hadoop
Updating Homebrew...
==> Downloading https://www.apache.org/dyn/closer.cgi?path=hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
==> Downloading from http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
######################################################################## 100.0%
??  /usr/local/Cellar/hadoop/3.1.2: 21,686 files, 774.1MB, built in 10 minutes 1 second

配置

需要修改配置文件都在/usr/local/Cellar/hadoop/3.1.2/libexec/etc/hadoop這個(gè)目錄下

$ vim hadoop-env.sh
$ vim core-site.xml
$ vim hdfs-site.xml

hadoop-env.sh

配置JAVA_HOME

image.png

將/usr/libexec/java_home查到的 Java 路徑，記得去掉注釋 #。

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home

core-site.xml

修改core-site.xml 文件參數(shù),配置NameNode的主機(jī)名和端口號

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

hdfs-site.xml

變量dfs.replication指定了每個(gè)HDFS數(shù)據(jù)庫的復(fù)制次數(shù)。 通常為3, 由于我們只有一臺主機(jī)和一個(gè)偽分布式模式的DataNode，將此值修改為1

<configuration>
 <property>
 <name>dfs.replication</name>
 <value>1</value>
 </property>
</configuration>

格式化

格式化hdfs操作只要第一次才使用，否則會造成數(shù)據(jù)全部丟失
hdfs namenode -format

image.png

啟動(dòng) NameNode 和 DataNode：

$ cd /usr/local/Cellar/hadoop/3.1.2/sbin
$ ./start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [baxiangs-Mac-mini.local]
baxiangs-Mac-mini.local: Warning: Permanently added 'baxiangs-mac-mini.local,192.168.1.115' (ECDSA) to the list of known hosts.
2019-08-04 01:25:14,753 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

http://localhost:9870/

image.png

$ jps
5665 DataNode
5890 Jps
5797 SecondaryNameNode
1274 Launcher
317
5566 NameNode

YARN服務(wù)

./start-yarn.sh

關(guān)閉YARN服務(wù)

./stop-yarn.sh

啟動(dòng)成功后，我們在瀏覽器中輸入http://localhost:8088/cluster

image.png

啟動(dòng)/關(guān)閉Hadoop服務(wù)(等效上面兩個(gè))
./start-all.sh
./stop-all.sh

執(zhí)行wordcount

創(chuàng)建文件

$ hadoop fs -mkdir  /test
$ echo "hello hdfs" >> demo.txt
$ hadoop fs -put demo.txt /test/demo.txt
$ $ hadoop fs -cat /test/demo.txt

執(zhí)行mr統(tǒng)計(jì)wordcount

$ hadoop jar /usr/local/Cellar/hadoop/3.1.2/libexec/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.1.2-sources.jar org.apache.hadoop.examples.WordCount /test/demo.txt /test/out/word-out

查看執(zhí)行結(jié)果

$hadoop fs -cat  /test/out/word-out/part-r-00000
2019-08-07 22:46:46,974 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hdfs    1
hello   1

安裝spark

=> Downloading https://www.apache.org/dyn/closer.lua?path=spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
==> Downloading from http://45.252.224.79/files/623300000DD89759/mirror.bit.edu.cn/apache/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
######################################################################## 100.0%
??  /usr/local/Cellar/apache-spark/2.4.3: 1,059 files, 248.4MB, built in 29 seconds