在線部署CDH

在線部署CDH

  1. 本文主要記錄了 在線安裝CDH的步驟，其中 JDK 和 數(shù)據(jù)庫 按需安裝。

  2. 服務(wù)器系統(tǒng)為CentOS，不同系統(tǒng)命令可能會有所不同。

  3. 在線安裝的優(yōu)勢在于 方便，操作簡單，基本跟著步驟走下來問題不大。缺點也很明顯，下載慢，在內(nèi)網(wǎng)環(huán)境或者網(wǎng)絡(luò)環(huán)境不好的時候，在線安裝的弊端就顯現(xiàn)出來了，所以內(nèi)網(wǎng)或者網(wǎng)絡(luò)不好更推薦離線部署的方式。

1. 準(zhǔn)備工作

1.1 虛擬機(jī)

我這邊準(zhǔn)備了3臺虛擬機(jī) ， Linux版本為：CentOS

  • 192.168.44.128 - 部署 scm-serser 和 scm-agent
  • 192.168.44.129 - 部署 scm-agent
  • 192.168.44.130 - 部署 scm-agent

1.2 設(shè)置所有節(jié)點的hosts?件

vim /etc/hosts

# add 
192.168.44.128 vmware128
192.168.44.129 vmware129
192.168.44.130 vmware130

1.3 關(guān)閉防火墻

sudo iptables-save > ~/firewall.rules

sudo systemctl disable firewalld
sudo systemctl stop firewalld

1.4 禁用SELinux

vim /etc/selinux/config
# 將 SELINUX=enforcing 改為 SELINUX=disabled
SELINUX=disabled

修改之后重啟機(jī)器

1.5 安裝ntp

yum install ntp

vim /etc/ntp.conf
# add
server vmware128
server vmware129
server vmware130

sudo systemctl start ntpd
sudo systemctl enable ntpd

ntpdate -u vmware128
hwclock --systohc

1.6 更多準(zhǔn)備工作

更多準(zhǔn)備工作參考：https://docs.cloudera.com/documentation/enterprise/6/6.0/topics/installation_reqts.html

2. 安裝CM-server

2.1 配置Cloudera Manager倉庫

sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

2.2 安裝JDK

JDK版本要求在1.8及以上，如果已經(jīng)安裝過JDK（需要配置對應(yīng)環(huán)境變量），可以跳過此步。JDK要求詳見： https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html#java_requirements

sudo yum install oracle-j2sdk1.8

2.3 安裝CM server

這一步在線下載時間會比較長，在規(guī)劃的主機(jī)上執(zhí)行：

這一步安裝了 daemons server agent 三部分

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

2.4 安裝配置CM agent

2.4.1 安裝CM agent

在其他主機(jī)上執(zhí)行，這一步是安裝 agent，也可以在server起來后再裝

這一步是在其他主機(jī)上安裝了 daemons agent

sudo yum install cloudera-manager-daemons cloudera-manager-agent

2.4.2 配置agent

vim /etc/cloudera-scm-agent/config.ini

## 修改下面配置
## Name of the host where Cloudera Manager Server is running.
server_host vmware128

## Port on the host where Cloudera Manager Server is running.
server_port 7180

2.4.3 啟動agent

sudo systemctl start cloudera-scm-agent

2.5 安裝數(shù)據(jù)庫(MySQL)

如果已有 MySQL（MariaDB） 或 其他數(shù)據(jù)庫(PostgreSQL ,oracle)，此步驟可以跳過。

更多數(shù)據(jù)庫及版本兼容性詳見： https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_database_requirements.html#cdh_cm_supported_db

2.5.1 安裝MySQL

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm

sudo yum update

sudo yum install mysql-server

sudo systemctl start mysqld

2.5.2 配置MySQL

  1. 停掉MySQL

    sudo systemctl stop mysqld

  2. 編輯 /etc/my.cnf :

    官方推薦配置

    [mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

2.5.3 啟動MySQL

sudo systemctl enable mysqld
sudo systemctl start mysqld

設(shè)置MySQL root用戶密碼

sudo /usr/bin/mysql_secure_installation

選項按照下列選擇：

[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

2.5.4 安裝 MySQL JDBC driver

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

tar zxvf mysql-connector-java-5.1.46.tar.gz

sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

2.5.5 創(chuàng)建數(shù)據(jù)庫

mysql -u root -p

mysql> CREATE DATABASE <database> DEFAULT CHARACTER SET <character set> DEFAULT COLLATE utf8_general_ci;

mysql> GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';

根據(jù)需要創(chuàng)建對應(yīng)的數(shù)據(jù)庫

Service Database User
Cloudera Manager Server scm scm
Activity Monitor amon amon
Reports Manager rman rman
Hue hue hue
Hive Metastore Server metastore hive
Sentry Server sentry sentry
Cloudera Navigator Audit Server nav nav
Cloudera Navigator Metadata Server navms navms
Oozie oozie oozie

這里創(chuàng)建了 scm 用于存放 CM的元數(shù)據(jù)，amon 用于存放CM的監(jiān)控數(shù)據(jù)。

create database scm DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
flush privileges;

2.6 配置CM數(shù)據(jù)庫

  1. 如果數(shù)據(jù)庫裝在本機(jī)上：

    #sudo /opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType> <databaseName> <databaseUser>

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

  2. 如果裝在其他機(jī)器上：

    #sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm

# -P 指定數(shù)據(jù)庫端口，默認(rèn)3306
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h vmware128 -P 3306  --scm-host vmware128 scm scm

2.7 啟動CM server

sudo systemctl start cloudera-scm-server

## 查看日志
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

## 如果啟動成功會有：
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

web管理地址：http://vmware128:7180，初始帳號密碼 ：admin 。

后續(xù)配置服務(wù)都可以在 web上操作了。

3. 相關(guān)命令


sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
sudo yum install cloudera-manager-daemons cloudera-manager-agent

sudo systemctl start cloudera-scm-server
sudo systemctl start cloudera-scm-agent

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';

4. 參考資料

Cloudera Installation Guide

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容