搭建HDFS

• HDFS是一個(gè)高度容錯(cuò)性的系統(tǒng)，適合部署在廉價(jià)的機(jī)器上。HDFS能提供高吞吐量的數(shù)據(jù)訪問(wèn)，非常適合大規(guī)模數(shù)據(jù)集上的應(yīng)用。

• 安裝hadoop:

  ?  bin brew install hadoop
  

• 配置ssh免密碼登錄
  用dsa密鑰認(rèn)證來(lái)生成一對(duì)公鑰和私鑰:

  ?  hadoop ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
  

• 將生成的公鑰加入到用于認(rèn)證的公鑰文件中：

  ?  hadoop cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_key
  

• 驗(yàn)證配置是否成功:

  ?  hadoop ssh localhost
  

• 配置core-site.xml(/usr/local/Cellar/hadoop/2.8.0/libexec/etc/hadoop):

  <configuration>
     <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
     </property>
  </configuration>
  

• 配置hdfs-site.xml:

  <configuration>
     <property>
       <name>dfs.replication</name>
       <value>1</value>
     </property>
  </configuration>
  

• 配置mapred-site.xml:

  <configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
  </configuration>
  

• 配置yarn-site.xml:

  <configuration>
  
  <!-- Site specific YARN configuration properties -->
  <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
  </property>
  </configuration>
  

• 進(jìn)入hadoop目錄(/usr/local/Cellar/hadoop/2.8.0/libexec):

• 格式化文件系統(tǒng):

  ?  libexec bin/hdfs namenode -format
  

• 啟動(dòng)NameNode和DataNode的守護(hù)進(jìn)程:

  ?  libexec sbin/start-dfs.sh
  

• 啟動(dòng)ResourceManager和NodeManager的守護(hù)進(jìn)程:

  ?  libexec sbin/start-yarn.sh
  

• 訪問(wèn)localhost:50070和localhost:8088測(cè)試是否正常

  
  
  50070.png
  
  
  8080.png

• 創(chuàng)建hdfs目錄:

  ?  libexec bin/hdfs dfs -mkdir /user
  ?  libexec bin/hdfs dfs -mkdir /user/lkc
  

• 拷貝一些文件:

  ?  libexec bin/hdfs dfs -put /Users/lkc/Desktop/Other input
  

  查看一下拷貝成功:

  
  
  hadoop.png

搭建ZooKeeper

• ZooKeeper是一個(gè)分布式的，開(kāi)放源碼的分布式應(yīng)用程序協(xié)調(diào)服務(wù)，是Google的Chubby一個(gè)開(kāi)源的實(shí)現(xiàn)，是Hadoop和Hbase的重要組件。它是一個(gè)為分布式應(yīng)用提供一致性服務(wù)的軟件，提供的功能包括：配置維護(hù)、域名服務(wù)、分布式同步、組服務(wù)等。

• 從官網(wǎng)下載:ZooKeeper官網(wǎng)

• 解壓:

  tar zxvf zookeeper-3.4.10.tar.gz
  

• 進(jìn)入主目錄新建一個(gè)zoo.cfg的文件,內(nèi)容如下:

  tickTime=2000  
  dataDir= /Users/lkc/Desktop/Other/zookeeper-3.4.10/data  (填寫(xiě)自己的data目錄)  
  dataLogDir=/Users/lkc/Desktop/Other/出單zookeeper-3.4.10/logs  
  clientPort=2181
  

• 運(yùn)行zookeeper:

  ?

  ./bin/zkServer.sh start
  

• 執(zhí)行

  telnet 127.0.0.1 2181 
  

  查看是否啟動(dòng)成功.

• 停止命令:

  ./bin/zkServer.sh stop
  

搭建HBase

• Hbase是Hadoop的數(shù)據(jù)庫(kù), 而Hive數(shù)據(jù)庫(kù)的管理工具, hbase具有分布式, 可擴(kuò)展及面向列存儲(chǔ)的特點(diǎn)(基于谷歌BigTable). HBase可以使用本地文件系統(tǒng)和HDFS文件存儲(chǔ)系統(tǒng), 存儲(chǔ)的是松散的數(shù)據(jù)(key-value的映射關(guān)系).

  HBase位于HDFS的上層, 向下提供存儲(chǔ), 向上提供運(yùn)算

• 安裝:

  brew install hbase
  

• 配置:

  在conf/hbase-env.sh設(shè)置JAVA_HOME

  export JAVA_HOME="/usr/bin/java"
  

• 在conf/hbase-site.xml設(shè)置HBase的核心配置

  <configuration>
    <property>
        <name>hbase.rootdir</name>
            //這里設(shè)置讓HBase存儲(chǔ)文件的地方
        <value>file:///Users/lkc/Desktop/Other/hbase</value>
    </property>
    <property>
     <name>hbase.zookeeper.property.dataDir</name>
                  //這里設(shè)置讓HBase存儲(chǔ)內(nèi)建zookeeper文件的地方
          <value>/Users/lkc/Desktop/Other/zookeeper-3.4.10</value>
     </property>
  </configuration>
  

• /usr/local/Cellar/hbase/1.2.2/bin/start-hbase.sh提供HBase的啟動(dòng)

• 驗(yàn)證是否成功:

  ?  bin ./start-hbase.sh
  starting master, logging to /usr/local/var/log/hbase/hbase-lkc-master-localhost.out
  Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
  Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
  ?  bin jps
  15171 Jps
  7907 DataNode
  7828 NameNode
  15093 HMaster
  8006 SecondaryNameNode
  

• 啟動(dòng)HBase shell:

  ?  bin ./hbase shell
  2017-06-11 23:18:38,460 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.2.2/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/Users/lkc/Downloads/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  HBase Shell; enter 'help<RETURN>' for list of supported commands.
  Type "exit<RETURN>" to leave the HBase Shell
  Version 1.2.2, r3f671c1ead70d249ea4598f1bbcc5151322b3a13, Fri Jul  1 08:28:55 CDT 2016
  
  hbase(main):001:0> exit
  

• 停止:

  ./bin/stop-hbase.sh
  

  ?