原文3年多前發(fā)表在私人站點，現(xiàn)遷移到簡書
當(dāng)時裝的是5.1.0，現(xiàn)在最新的版本是5.4.2，因為有在線業(yè)務(wù)使用，所以暫時不升級。
cdh
獨立下載hadoop各個組件再安裝比較繁瑣(hdfs+yarn+hbsae+zk+hive)，沒有選好版本可能會沖突，CDH的版本都是選定好的，安裝和升級文檔齊全,非常方便
安裝前配置
官方流程 大致分一下3個步驟：
配置yum源
wget http://archive.cloudera.com/cdh5/one-click-install/redhat/5/x86_64/cloudera-cdh-5-0.x86_64.rpm
sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm #安裝rpm，會加一個clouder的yum源：
yum clean all 、 yum makecache # 重新構(gòu)建yum緩存
sudo rpm --import http://archive.cloudera.com/cdh5/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera #導(dǎo)入GPG驗證的key
* 可能的問題:
1.運行yum的可能遇到錯誤:
It's possible that the above module doesn't match the current version of Python, which is:2.7.3 (default, May 19
2014, 15:04:50) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)]
需要修改yum的python依賴版本：
修改文件： vim /usr/bin/yum
修改頭#!/usr/bin/python => #!/usr/bin/python2.4
2.找不到host命令，需要裝下bind-utils：yum install bind-utils
安裝jdk
yum -y install unzip
curl -L -s get.jenv.io | bash
source /home/admin/.jenv/bin/jenv-init.sh
jenv install java 1.7.0_45
jdk通過USER賬號安裝，cdh系列的需要在自己的特定賬號下執(zhí)行，比如hdfs賬號，所以會出現(xiàn)找不到JAVA_HOME的問題,解決方法：
在/etc/sudoers 配置：Defaults env_keep+=JAVA_HOME
設(shè)置ROOT下的JAVA_HOME指向USER。。，需要修改USER為可執(zhí)行權(quán)限
還有另一個方法，是在/etc/default/bigtop-utils 下配置javahome（chmod 755 /home/USER）
export JAVA_HOME=/home/USER/.jenv/candidates/java/current
chmod 755 /home/USER/
HDFS
安裝和配置
NameNode、Client
sudo yum install hadoop-hdfs-namenode
sudo yum install hadoop-client
安裝DataNode
在DataNode機(jī)器上執(zhí)行：
sudo yum install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce
設(shè)置hdfs config文件到自己的目錄下
sudo cp -r /etc/hadoop/conf.empty /etc/hadoop/conf.my_cluster
sudo /usr/sbin/alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.my_cluster 50
sudo /usr/sbin/alternatives --set hadoop-conf /etc/hadoop/conf.my_cluster
sudo chmod -R 777 /etc/hadoop/conf.my_cluster
（alternatives --config java好像無效）
創(chuàng)建數(shù)據(jù)目錄（用戶組hdfs:hdfs 權(quán)限700）:
datanode：sudo mkdir -p /data/hadoop/hdfs/dn
sudo chown -R hdfs:hdfs /data/hadoop
hadoop-env.sh
hadoop默認(rèn)為namenode、datanode都是1G的內(nèi)存：
export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Xmx3072m -verbose:gc -Xloggc:/var/log/hadoop-hdfs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -Xmx2048m -verbose:gc -Xloggc:/var/log/hadoop-hdfs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
core-site.xml
<property>
<!-- namenode地址和端口 -->
<name>fs.defaultFS</name>
<value>hdfs://cdhhadoop1:8020</value>
</property>
<!-- 回收站，默認(rèn)保留一天 -->
<property>
<name>fs.trash.interval</name>
<value>1440</value>
</property>
<property>
<name>fs.trash.checkpoint.interval</name>
<value>0</value>
</property>
<!-- 配置Snappy壓縮 -->
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
配置hdfs-site.xml
<!-- 超級用戶 -->
<property>
<name>dfs.permissions.superusergroup</name>
<value>admin</value>
</property>
<!-- hdfs副本 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<!-- dfs.namenode.name.dir 作為namenode存放元信息的目錄，如果設(shè)置多個則會有一個拷貝，可以在另外一臺機(jī)器上搭一個NFS共享目錄，作為備份 ->
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/hadoop/hdfs/nn</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/data/hadoop/hdfs/dn</value>
</property>
- 其他配置:
1.如果datanode的目錄有一個寫失敗，DataNode就會停止，這樣這個DataNode上的所有目錄的副本都會增加，如果要避免這種情況，可以設(shè)置容忍失敗的目錄個數(shù)
2.可以配置負(fù)載均衡，默認(rèn)的分配策略是隨機(jī)的，可以配置一個策略比如磁盤大小
3.沒有配置web hdfs
啟動
- 格式化namenode
sudo -u hdfs hdfs namenode -format
日志文件目錄：/var/log/hadoop-hdfs
- 啟動namenode
sudo service hadoop-hdfs-namenode start
- 啟動datanode
sudo service hadoop-hdfs-datanode start
- 初始化
hdfs運行以后，推薦在hdfs上創(chuàng)建tmp目錄，并設(shè)置權(quán)限：
$ sudo -u hdfs hadoop fs -mkdir /tmp
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
測試
http://localhost:50070/dfshealth.html#tab-overview
簡單的測試只要執(zhí)行下hadoop fs命令即可，如果要測試讀寫性能，要等mapreduce裝好
【寫性能測試】
hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
我們集群的一次測試結(jié)果：
----- TestDFSIO ----- : write
Date & time: Sun Jul 13 21:40:41 CST 2014
Number of files: 10
Total MBytes processed: 10000.0（總共10個文件，每個1G）
Throughput mb/sec: 6.452312250618132（總大小/Map總時間）
Average IO rate mb/sec: 6.50354528427124
IO rate std deviation: 0.6099282285067701
Test exec time sec: 197.818（整體執(zhí)行時間）
Throughput是總大小文/每個Map時間之和，如果算并發(fā)吞吐量的話，可以乘以Map數(shù)量，詳細(xì)解讀可以參考：Benchmarking and Stress Testing an Hadoop Cluster With TeraSort, TestDFSIO & Co
【讀性能測試】
hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000
20:38:21 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
20:38:21 INFO fs.TestDFSIO: Date & time: Tue Jul 15 20:38:21 CST 2014
20:38:21 INFO fs.TestDFSIO: Number of files: 10
20:38:21 INFO fs.TestDFSIO: Total MBytes processed: 10000.0
20:38:21 INFO fs.TestDFSIO: Throughput mb/sec: 16.79261125104954
20:38:21 INFO fs.TestDFSIO: Average IO rate mb/sec: 16.829221725463867
20:38:21 INFO fs.TestDFSIO: IO rate std deviation: 0.8154139285912413
20:38:21 INFO fs.TestDFSIO: Test exec time sec: 84.614
測試結(jié)果以后需要清理測試結(jié)果
hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar TestDFSIO -clean
在windows看客戶端下測試Hdfs，需要到
https://github.com/srccodes/hadoop-common-2.2.0-bin 下載并替換hadoopHome下的bin目錄
YARN
安裝和配置
sudo yum install hadoop-yarn-resourcemanager
sudo yum install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver
sudo mkdir -p /data/yarn/local
sudo mkdir -p /data/yarn/logs
sudo chown -R yarn:yarn /data/yarn
hadoop fs -mkdir -p /user/history
hadoop fs -chmod -R 1777 /user/history
hadoop fs -chown mapred:hadoop /user/history
yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>cdhhadoop1</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>cdhhadoop1:8088</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>List of directories to store localized files in.</description>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///data/yarn/local</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>file:///data/yarn/logs</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>hdfs:///log/yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
啟動
- 端口
resourceManager 8088/cluster
nodeManager 8042/node
JobHistory 19888/jobhistory
Name:http://localhost:8088/cluster/nodes
Node1:http://localhost:8042/node
Node2:http://localhost:8043/node
測試
通過hadoop自帶的randowmwriter測試下：
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar randomwriter out
21:14:34 INFO mapreduce.Job: Job job_1405247153654_0004 completed successfully
21:14:34 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=1772230
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2350
HDFS: Number of bytes written=21545727074（寫入20G）
HDFS: Number of read operations=80
HDFS: Number of large read operations=0
HDFS: Number of write operations=40
Job Counters
Killed map tasks=10
Launched map tasks=30
Other local map tasks=30
Total time spent by all maps in occupied slots (ms)=7247472
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=7247472
Total vcore-seconds taken by all map tasks=7247472
Total megabyte-seconds taken by all map tasks=7421411328
Map-Reduce Framework
Map input records=20
Map output records=2043801
Input split bytes=2350
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=8157
CPU time spent (ms)=641440
Physical memory (bytes) snapshot=2889732096
Virtual memory (bytes) snapshot=14388494336
Total committed heap usage (bytes)=2371878912
org.apache.hadoop.examples.RandomWriter$Counters
BYTES_WRITTEN=21475013178
RECORDS_WRITTEN=2043801
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=21545727074
The job took 604 seconds.
ZK
安裝和配置
安裝
sudo yum install zookeeper
sudo yum install zookeeper-server
拷貝配置
sudo cp -r /etc/zookeeper/conf.dist /etc/zookeeper/conf.my_cluster
sudo alternatives --verbose --install /etc/zookeeper/conf zookeeper-conf /etc/zookeeper/conf.my_cluster 50
sudo alternatives --set zookeeper-conf /etc/zookeeper/conf.my_cluster
修改配置文件并在節(jié)點間同步
/etc/zookeeper/conf.my_cluster/zoo.cfg
server.1=A:2888:3888
server.2=B:2888:3888
server.3=C:2888:3888
創(chuàng)建數(shù)據(jù)目錄
mkdir -p /data/zookeeper
chown -R zookeeper:zookeeper /data/zookeeper
啟動
啟動日志在/var/log/zookeeper
在A運行 :
$ service zookeeper-server init --myid=1
$ service zookeeper-server start
在B運行
$ service zookeeper-server init --myid=2
$ service zookeeper-server start
在C運行
$ service zookeeper-server init --myid=3
$ service zookeeper-server start
測試
zookeeper-client -server A:2181
zookeeper-client -server B:2181
目錄列表： ls /
創(chuàng)建目錄： create /test "empty"
HBase
安裝和配置
安裝
所有機(jī)器上： sudo yum install hbase
NameNode：sudo yum install hbase-master
DataNode： sudo yum install hbase-regionserver
拷貝自己的配置文件
sudo cp -r /etc/hbase/conf.dist /etc/hbase/conf.my_cluster
sudo alternatives --verbose --install /etc/hbase/conf hbase-conf /etc/hbase/conf.my_cluster 50
sudo alternatives --set hbase-conf /etc/hbase/conf.my_cluster
修改最大文件數(shù)限制
避免Too many open files（/etc/security/limits.conf）
hdfs - nofile 32768
hbase - nofile 32768
阿里云機(jī)器默認(rèn)已經(jīng)是65535，所以不做修改
hdfs DataNode會限制打開的文件數(shù)（ /etc/hadoop/conf/hdfs-site.xml）
dfs.datanode.max.xcievers
65535
創(chuàng)建目錄
hadoop fs -mkdir /hbase
hadoop fs -chown hbase /hbase
hbase-site.xml
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://myhost:8020/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>A,B,C</value>
</property>
<!--關(guān)閉checksum-->
<property>
<name>hbase.regionserver.checksum.verify</name>
<value>false</value>
<description>
If set to true, HBase will read data and then verify checksums for
hfile blocks. Checksum verification inside HDFS will be switched off.
If the hbase-checksum verification fails, then it will switch back to
using HDFS checksums.
</description>
</property>
<property>
<name>hbase.hstore.checksum.algorithm</name>
<value>NULL</value>
<description>
Name of an algorithm that is used to compute checksums. Possible values
are NULL, CRC32, CRC32C.
</description>
</property>
啟動
service hbase-master start
service hbase-regionserver start
測試
60010是master的端口 http://localhost:60010/master-status?filter=all
60030是regionServer的端口
測試hbase集群是否支持snappy：
hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/benchmarks/hbase snappy
通過hbase shell訪問hbase
Hive
安裝和配置
安裝hive/metastore/hieveserver
sudo yum install -y hive
sudo yum install -y hive-metastore
sudo yum install -y hive-server2
mysql-connector-java.jar
在metastore的機(jī)器，把mysql-connector-java.jar放到/usr/lib/hive/lib/目錄下
java堆配置
官方文檔有誤，實際配置文件是:/etc/hive/conf/hive-env.sh
if [ "$SERVICE" = "hiveserver2或者metastore" ]; then
export HADOOP_OPTS="${HADOOP_OPTS} -Xmx3072m -Xms1024m -Xloggc:/var/log/hive/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
fi
export HADOOP_HEAPSIZE=512
metastore配置(配置文件：hive-site.xml)
metastore 配置hdfs
先初始化下hdfs得配置,再從namenode把最新的配置拷過來：scp /etc/hadoop/conf.my_cluster/hdfs-site.xml /etc/hadoop/conf.my_cluster/core-site.xml host:/etc/hadoop/conf.my_cluster/
hiveserver2配置(配置文件：/etc/hive/conf/hive-site.xml)
hive.support.concurrency
Enable Hive's Table Lock Manager Service
true
hive.zookeeper.quorum
Zookeeper quorum used by Hive's Table Lock Manager
A,B,C
hive.metastore.local
false
hive.metastore.uris
thrift://xxxxx:9083
啟動
sudo /sbin/service hive-metastore start
sudo /sbin/service hive-server2 start
測試
1./usr/lib/hive/bin/beeline
-
2.!connect jdbc:hive2://localhost:10000 username password org.apache.hive.jdbc.HiveDriver
或者： !connect jdbc:hive2://10.241.52.161:10000 username password org.apache.hive.jdbc.HiveDriver
3.show tables;
hive服務(wù)端日志在：/var/log/hive
hive shell日志在/tmp/admin/hive.log，之前有個配置錯誤引起的異常，一直沒找到日志，原來路徑是在/etc/hive/conf下的log4j配置的
