1 概述
本文檔描述CENTOS6.5操作系統(tǒng)部署CDH企業(yè)版的過(guò)程。Cloudera企業(yè)級(jí)數(shù)據(jù)中心的安裝主要分為4個(gè)步驟：
1.集群服務(wù)器配置，包括安裝操作系統(tǒng)、關(guān)閉防火墻、同步服務(wù)器時(shí)鐘等；
2.外部數(shù)據(jù)庫(kù)安裝
3.安裝Cloudera管理器；
4.安裝CDH集群；
這篇文檔將著重介紹Cloudera管理器與CDH的安裝，并基于以下假設(shè)：
1.操作系統(tǒng)版本：CENTOS6.5
2.MySQL數(shù)據(jù)庫(kù)版本為5.1.73
3.CM版本：CM 5.12.0
4.CDH版本：CDH 5.12.0
5.Kudu版本：1.4
6.采用root對(duì)集群進(jìn)行部署
7.您已經(jīng)下載CDH和CM的安裝包
2 前期準(zhǔn)備
1.集群中各個(gè)節(jié)點(diǎn)之間能互相通信使用靜態(tài)IP地址。IP地址和主機(jī)名通過(guò)/etc/hosts配置，主機(jī)名/etc/HOSTNAME進(jìn)行配置。
以cm節(jié)點(diǎn)為例：
/etc/hosts文件如下:
172.31.8.113 cdh1
172.31.14.96 cdh2
172.31.9.234 cdh3
172.31.2.34 cdh4
2.配置ntp時(shí)間同步,將172.31.8.113機(jī)器作為本地ntp服務(wù)器，其他3臺(tái)機(jī)器與其保持同步，配置片段：
172.31.8.113:
$> sudo vim
/etc/ntp.conf
server? 127.127.1.0???????????? # local clock (LCL)
fudge? 127.127.1.0 stratum 10 # LCL is? unsynchronized
其他機(jī)器：
$> sudo vim
/etc/ntp.conf
# server
127.127.1.0
# local clock (LCL)
# fudge?? 127.127.1.0 stratum 10 # LCL is unsynchronized
重啟所有機(jī)器的ntp服務(wù)：
service ntpd restart
驗(yàn)證始終同步：ntpq -p
3.關(guān)閉防火墻：
在每臺(tái)機(jī)器上運(yùn)行下列命令
service iptables stop
chkconfig iptables off
vim /etc/selinux/config改為SELINUX=disabled
檢查所有機(jī)器：以cdh1示例
安裝httpd服務(wù)：
[root@cdh1 ~]# yum -y? install httpd
4.安裝mysql。
[root@cdh1
~]# yum -y install mysql
[root@cdh1? ~]# yum -y install mysql-server
啟動(dòng)mysql并配置mysql
[root@cdh1 ~]#? service mysqld start
[root@cdh1 ~]#? /usr/bin/mysql_secure_installation
NOTE: RUNNING ALL? PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
SERVERS IN PRODUCTION USE!PLEASE READ EACH STEP CAREFULLY!
In order to log into? MySQL to secure it, we'll need the current
password for the root? user.If you've just installed MySQL,? and
you haven't set the? root password yet, the password will be blank,
so you should just? press enter here.
Enter current? password for root (enter for none):
OK, successfully used? password, moving on...
Setting the root? password ensures that nobody can log into the MySQL
root user without the? proper authorisation.
Set root password?? [Y/n] Y
New password:
Re-enter new? password:
Password updated? successfully!
Reloading privilege? tables..
... Success!
By default, a MySQL? installation has an anonymous user, allowing anyone
to log into MySQL? without having to have a user account created for
them.This is intended only for testing, and to? make the installation
go a bit? smoother.You should remove them? before moving into a
production? environment.
Remove anonymous? users? [Y/n] Y
... Success!
Normally, root should? only be allowed to connect from 'localhost'.This
ensures that someone? cannot guess at the root password from the network.
Disallow root login? remotely? [Y/n] n
... skipping.
By default, MySQL? comes with a database named 'test' that anyone can
access.This is also intended only for testing, and? should be removed
before moving into a? production environment.
Remove test database? and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the? privilege tables will ensure that all changes made so far
will take effect? immediately.
Reload privilege? tables now? [Y/n] Y
... Success!
Cleaning up...
All done!If you've completed all of the above steps,? your MySQL
installation should? now be secure.
Thanks for using? MySQL!
建立CM和Hive需要的表
[root@cdh1? ~]# mysql -u root -p
Enter? password:
Welcome? to the MySQL monitor.Commands end? with ; or \g.
Your? MySQL connection id is 8
create? database metastore default character set utf8;
CREATE? USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH
PRIVILEGES;
create? database cm default character set utf8;
CREATE? USER 'cm'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH? PRIVILEGES;
create? database am default character set utf8;
CREATE? USER 'am'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON am. * TO 'am'@'%';
FLUSH? PRIVILEGES;
create? database rm default character set utf8;
CREATE? USER 'rm'@'%' IDENTIFIED BY 'password';
GRANT ALL? PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH
PRIVILEGES;
安裝jdbc驅(qū)動(dòng)：
[root@cdh1 ~]# mv? mysql-connector-java-5.1.34.jar /usr/share/java/
[root@cdh1 ~]# ln -s? mysql-connector-java-5.1.34.jar mysql-connector-java.jar
dctest01:/usr/share/java? # ll
總用量944
-rwxr-xr-x 1 root? root 9603723月16 15:58 mysql-connector-java-5.1.34.jar
lrwxrwxrwx 1 root? root313月16 16:15 mysql-connector-java.jar ->? mysql-connector-java-5.1.34.jar
3 cloudera manager 安裝
配置本地repo源，將Cloudera Manager安裝需要的7個(gè)rpm包下載到本地，放在同一目錄，執(zhí)行createrepo命令生成rpm元數(shù)據(jù)：
[root@cdh1 cm5.12.0]#? ll
total 958396
-rw-r--r-- 1 root? root9674128 Jul7 04:35? cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root 726564488 Jul7 04:35? cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root8704 Jul7 04:35? cloudera-manager-server-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root10608 Jul7 04:36? cloudera-manager-server-db-2-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root31880440 Jul7 04:36? enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
-rw-r--r-- 1 root? root71204325 Jul7 04:36 jdk-6u31-linux-amd64.rpm
-rw-r--r-- 1 root? root 142039186 Jul7 04:35? oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
[root@cdh1 cm5.12.0]#? createrepo .
Spawning worker 0? with 7 pkgs
Workers Finished
Gathering worker? results
Saving Primary? metadata
Saving file lists? metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
配置Web服務(wù)器，比如apache2、nginx等，將上述目錄移動(dòng)到Web服務(wù)器數(shù)據(jù)目錄，這里為/var/www/html，使得用戶可以通過(guò)HTTP訪問(wèn)這些rpm包。
驗(yàn)證安裝JDK
yum -y install oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
通過(guò)yum安裝Cloudera Manager Server
yum -y install? cloudera-manager-server-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
初始化數(shù)據(jù)庫(kù)
sudo /usr/share/cmf/schema/scm_prepare_database.sh? mysql cm cm password
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Verifying that we can? write to /etc/cloudera-scm-server
Creating SCM configuration? file in /etc/cloudera-scm-server
Executing:/usr/java/jdk1.7.0_67-cloudera/bin/java -cp? /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/*? com.cloudera.enterprise.dbutil.DbCommandExecutor? /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main]? DbCommandExecutorINFOSuccessfully connected to? database.
All done, your SCM? database is configured correctly!
啟動(dòng)Cloudera Manager Server
[root@cdh1 init.d]#? service cloudera-scm-server start
這個(gè)服務(wù)啟動(dòng)比較慢，稍微等幾分鐘，等可以看到7180端口，就可以了
[root@cdh1? cm5.12]# netstat -lnpt | grep 7180
tcp00 0.0.0.0:71800.0.0.0:*LISTEN1524/java
通過(guò)http://54.169.251.123:7180/cmf/login訪問(wèn)CM
4 CDH安裝
1.admin/admin登錄到CM。
2.點(diǎn)擊繼續(xù)
3.輸入主機(jī)ip或者名稱，點(diǎn)擊搜索找到主機(jī)后點(diǎn)擊繼續(xù)
4.點(diǎn)擊“更多選項(xiàng)”
5.點(diǎn)擊“-”刪除其他所有地址，輸入http://54.169.251.123/cdh5.12/和http://54.169.251.123/kudu1.4/點(diǎn)擊確定
6.輸入cm的http地址
7.安裝jdk
8.輸入root密碼
9.開(kāi)始安裝cloudera manager相關(guān)到各臺(tái)節(jié)點(diǎn)
10.開(kāi)始安裝cdh和Kudu到各臺(tái)節(jié)點(diǎn)
11.主機(jī)檢查，點(diǎn)擊繼續(xù)
12.選擇需要安裝的服務(wù)
13.分配角色
14.測(cè)試數(shù)據(jù)庫(kù)連接，點(diǎn)擊繼續(xù)
15.設(shè)置目錄
16.等待各個(gè)服務(wù)啟動(dòng)
17.安裝成功
5 Kudu配置
在CM界面添加服務(wù)，選擇Kudu
分配主機(jī)角色
配置相應(yīng)的目錄
點(diǎn)擊繼續(xù)啟動(dòng)Kudu服務(wù)
添加Kudu服務(wù)成功，安裝完畢
6 快速組件服務(wù)驗(yàn)證
1.MapReduce驗(yàn)證
#增加用戶組
[root]$?groupadd? cloudera-dev
#增加用戶
[root]$?useradd? -g cloudera-dev cloudera-dev
#查看用戶組cloudera-dev中的所有用戶
[root]$?id -g?cloudera-dev
#查看用戶cloudera-dev所屬的所有組
[root]$?id? cloudera-dev
# Hadoop創(chuàng)建相應(yīng)的用戶
[root]$?sudo -u? hdfs hadoop fs -mkdir /user/cloudera-dev
[root]$?sudo -u? hdfs hadoop fs -chown cloudera-dev:cloudera-dev /user/cloudera-dev
#運(yùn)行Hadoop wordcount示例程序
[root]$?sudo su? cloudera-dev
[cloudera-dev]$ echo? "Hello World Bye World" > file0
[cloudera-dev]$ echo? "Hello Hadoop Goodbye Hadoop" > file1
[cloudera-dev]$? hadoop fs -mkdir -p /user/cloudera-dev/wordcount/input
[cloudera-dev]$ hadoop?fs? -put file* /user/cloudera-dev/wordcount/input
[cloudera-dev]$? hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-examples.jar wordcount? wordcount/input wordcount/output
[cloudera-dev]$? hadoop fs -getmerge wordcount/output output.txt
[cloudera-dev]$ cat? output.txt
2.Hive驗(yàn)證
[root]$?sudo su? cloudera-dev
[cloudera-dev]$ echo? "Alex,Cloudera" > file2
[cloudera-dev]$? hadoop fs -mkdir -p hive/input
[cloudera-dev]$? hadoop fs -put file2 hive/input
[cloudera-dev]$ cat? > test.hql
create external table? test (
name string,
company string
)
row format delimited
fields? terminated by ','
location? '/user/cloudera-dev/hive/input'
[cloudera-dev]$ hive? -f?test.hql
[cloudera-dev]$ hive? -e "select * from test" 2> /dev/null
[cloudera-dev]$ hive -e "select? count(*) from test" 2> /dev/null
[cloudera-dev]$ hive -e "drop? table test"
[cloudera-dev]$ exit
3.HBase驗(yàn)證
[cloudera-dev]$?? cat > test.hbase
create 'record',? {NAME => 'user'}
put 'record', 'Alex',? 'user:company', 'Cloudera'
get 'record', 'Alex'
exit
[cloudera-dev]$?? hbase shell test.hbase
[cloudera-dev]$? hbase shell
hbase(main):001:0>? disable 'record'
0 row(s) in 2.6130? seconds
hbase(main):002:0>? drop 'record'
0 row(s) in 1.2880? seconds
#運(yùn)行Hive Over HBase測(cè)試(Read & Write)
[cloudera-dev]$?? cat test.hbase
create 'record',? {NAME => 'user'}
put 'record', 'Alex',? 'user:company', 'Cloudera'
get 'record', 'Alex'
exit
[cloudera-dev]$?? hbase shell test.hbase
[cloudera-dev]$? cat? readHiveOverHbase.hql
create external table? test (name string, company string)
stored by? 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties? ("hbase.columns.mapping" = ":key,user:company")
tblproperties? ("hbase.table.name" = "record");
[cloudera-dev]$?? hive -f readHiveOverHbase.hql
[cloudera-dev]$?? hive -e "select * from test" 2> /dev/null
[cloudera-dev]$? hive -e? "select count(*) from test" 2> /dev/null
[cloudera-dev]$? cat? writeHiveOverHbase.hql
create table test1? (name string, firm string)
stored by? 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties? ("hbase.columns.mapping" = ":key,user:firm")
tblproperties ("hbase.table.name" = "record1");
[cloudera-dev]$?? hive -f writeHiveOverHbase.hql
[cloudera-dev]$?? hive -e "insert overwrite table test1 select name, company from? test"
[cloudera-dev]$?? hbase shell
hbase(main):001:0>? list
TABLE
record
record1
2 row(s) in 0.2400? seconds
=>? ["record", "record1"]
hbase(main):002:0>? scan 'record1'
ROWCOLUMN+CELL
Alexcolumn=user:firm,? timestamp=1500707374796, value=Cloudera
1 row(s) in 0.1360? seconds
4. Kudu驗(yàn)證
先在Impala的高級(jí)配置中配置Kudu Master地址：
--kudu_master_hosts=cdh1:7051
[root@cdh1 ~]# impala-shell -i cdh4
[cdh4:21000]? > CREATE TABLE my_first_table
> (
> id BIGINT,
> name STRING,
> PRIMARY KEY(id)
> )
> PARTITION BY HASH PARTITIONS 16
> STORED AS KUDU;
[cdh4:21000]? > INSERT INTO my_first_table VALUES (99,"sarah");
[cdh4:21000]? > INSERT INTO my_first_table VALUES? (1,"john"),(2,"jane"),(3,"jim");
[cdh4:21000]? > select * from my_first_table;
[cdh4:21000]? > delete from my_first_table where id =99;
[cdh4:21000]? > select * from my_first_table;
[cdh4:21000]? > INSERT INTO my_first_table VALUES (99,"sarah");
[cdh4:21000]? > update my_first_table set name='lilei' where id=99;
[cdh4:21000]? > select * from my_first_table;
[cdh4:21000]? > upsertinto my_first_table? values(1, "john"), (4, "tom"), (99, "lilei1");
[cdh4:21000]? > select * from my_first_table;
