此次過程為：通過centos鏡像，創(chuàng)建三個容器，在三個容器中搭建一個小型的cdh集群。
centos已經(jīng)具備，接下來就是創(chuàng)建三個容器
docker網(wǎng)絡(luò)
由于cdh集群需要三個容器之間網(wǎng)絡(luò)互通，所以先創(chuàng)建docker網(wǎng)絡(luò)
Docker 提供三種網(wǎng)絡(luò)驅(qū)動：bridge、overlay、macvlan
overlay 和 macvlan 用于創(chuàng)建跨主機的網(wǎng)絡(luò)
這里三個容器是在同一個主機上，所以直接使用默認的網(wǎng)絡(luò)驅(qū)動即可，后續(xù)跨主機再繼續(xù)深究其他兩種網(wǎng)絡(luò)驅(qū)動。
查看docker網(wǎng)絡(luò)
docker network ls
返回
NETWORK ID NAME DRIVER SCOPE
3be276e8c0b7 bridge bridge local
6bdb390e9d18 host host local
d7df99850908 none null local
創(chuàng)建docker網(wǎng)絡(luò)
docker network create --driver bridge --subnet=172.21.0.0/24 --gateway=172.21.0.1 cdh_net
返回
569e6db77a6b3e1a063949fee0d3be9565f8cd621256993b51311ad8a497eb67
參數(shù)說明：
--driver 網(wǎng)絡(luò)驅(qū)動（可不寫，默認為bridge）
--subnet 子網(wǎng)ip
--gateway 網(wǎng)關(guān)
cdh_net 網(wǎng)絡(luò)名稱
再次查看網(wǎng)絡(luò)
docker network ls
返回
NETWORK ID NAME DRIVER SCOPE
3be276e8c0b7 bridge bridge local
569e6db77a6b cdh_net bridge local
6bdb390e9d18 host host local
d7df99850908 none null local
說明需要的網(wǎng)絡(luò)已經(jīng)創(chuàng)建好了
創(chuàng)建容器
網(wǎng)絡(luò)創(chuàng)建好，接下來需要基于創(chuàng)建好的網(wǎng)絡(luò)創(chuàng)建容器，鏡像為centos的鏡像
查看容器
docker container list
返回
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
docker中沒有容器
查看鏡像
docker images
返回
REPOSITORY TAG IMAGE ID CREATED SIZE
centos latest 5d0da3dc9764 19 months ago 231MB
選擇的鏡像為centos:latest
創(chuàng)建容器
docker run -it -d --name cdh01 --network=cdh_net --ip=172.21.0.101 -p2227:22 -p8080:8080 -p8000:8000/udp -p7183:7183 -p7180:7180 -p9870:9870 -p16010:16010 -p18088:18088 --restart always --privileged=true centos:latest /usr/sbin/init
返回
e399964a01bdd5855dffb9aba355ee9bdcc3edc89b0159cba027e067d6bcfdca
docker run -it -d --name cdh02 --network=cdh_net --ip=172.21.0.102 -p2228:22 --restart always --privileged=true centos:latest /usr/sbin/init
返回
666d932c563f327e032416f7fb02a8c62befd0f0b9896da8da0486bf0765f028
docker run -it -d --name cdh03 --network=cdh_net --ip=172.21.0.103 -p2229:22 --restart always --privileged=true centos:latest /usr/sbin/init
返回
f02d6a292ddd12aba34c1090569bbb2e32c18e39dbc919d50969648193750ef8
查看容器，看是否創(chuàng)建成功
docker container list
返回
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f02d6a292ddd centos:latest "/usr/sbin/init" About a minute ago Up About a minute 0.0.0.0:2229->22/tcp cdh03
666d932c563f centos:latest "/usr/sbin/init" 2 minutes ago Up 2 minutes 0.0.0.0:2228->22/tcp
cdh02
e399964a01bd centos:latest "/usr/sbin/init" 3 minutes ago Up 3 minutes 0.0.0.0:7180->7180/tcp, 0.0.0.0:7183->7183/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:9870->9870/tcp, 0.0.0.0:16010->16010/tcp, 0.0.0.0:18088->18088/tcp, 0.0.0.0:8000->8000/udp, 0.0.0.0:2227->22/tcp cdh01
三個容器創(chuàng)建完成
問題
問題1-docker的dnf makecache啟動報錯
但是在docker desktop中點進容器，在日志中發(fā)現(xiàn)有報錯，報錯內(nèi)容為：
image-20230426153952824.png
根據(jù)提示，進入命令行中執(zhí)行“systemctl status dnf-makecache.service”命令看問題
點進Terminal
執(zhí)行
systemctl status dnf-makecache.service
61e7deca063417e478405b967d34a12.png
根據(jù)告警告訴無法下載metadata，沒有準備好跟mirrorlist連接
應(yīng)該是跟yum有關(guān)系
執(zhí)行
yum list
返回
CentOS Linux 8 - AppStream 85 B/s | 38 B 00:00
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist
故進入“/etc/yum.repo.d”中進行修改，我這里把路徑下其他文件都修改了文件名，只留下CentOS-Linux-BaseOS.repo，把文件中的鏡像路徑改為阿里的鏡像，“mirrors.aliyun.com”阿里鏡像地址。
修改完成后
yum clean all
返回
0 files removed
yum list
嘩啦啦刷一大堆
說明鏡像通了，接下來重啟容器，查看日志
image-20230426155512259.png
可發(fā)現(xiàn)之前的報錯，現(xiàn)在ok了，這個問題解決了
此時此刻，理論上應(yīng)該可以遠程容器了
問題2-未安裝openssh服務(wù)
但是，我通過xshell，ssh或者telnet連接容器，都是在報錯
image-20230426155629156.png
image-20230426155656649.png
經(jīng)過查詢發(fā)現(xiàn)，原來是容器沒有安裝openssh服務(wù)，openssh服務(wù)是負責ssh登錄的，一坑接一坑
現(xiàn)在鏡像有了，天下我有，安裝openssh即可
查詢需要看裝的服務(wù)
yum search openssh
返回
Last metadata expiration check: 0:06:51 ago on Wed Apr 26 07:53:29 2023.
====================================================================================================== Name Exactly Matched: openssh =======================================================================================================
openssh.x86_64 : An open source implementation of SSH protocol version 2
===================================================================================================== Name & Summary Matched: openssh ======================================================================================================
openssh-keycat.x86_64 : A mls keycat backend for openssh
========================================================================================================== Name Matched: openssh ===========================================================================================================
openssh-cavs.x86_64 : CAVS tests for FIPS validation
openssh-clients.x86_64 : An open source SSH client applications
openssh-ldap.x86_64 : A LDAP support for open source SSH server daemon
openssh-server.x86_64 : An open source SSH server daemon
經(jīng)查詢只需要安裝openssh-clients跟openssh-server就行
安裝
yum install openssh-clients -y
yum install openssh-server -y
安裝完畢后，還需要啟動服務(wù)
systemctl restart sshd
查看是否啟動成功
systemctl status sshd
返回
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-04-26 08:03:29 UTC; 2s ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 192 (sshd)
Tasks: 1 (limit: 50160)
Memory: 1.1M
CGroup: /docker/f02d6a292ddd12aba34c1090569bbb2e32c18e39dbc919d50969648193750ef8/system.slice/sshd.service
└─192 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-12>
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Changed dead -> start
Apr 26 08:03:29 f02d6a292ddd systemd[1]: Starting OpenSSH server daemon...
Apr 26 08:03:29 f02d6a292ddd systemd[192]: sshd.service: Executing: /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-25>
Apr 26 08:03:29 f02d6a292ddd sshd[192]: Server listening on 0.0.0.0 port 22.
Apr 26 08:03:29 f02d6a292ddd sshd[192]: Server listening on :: port 22.
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Got notification message from PID 192 (READY=1)
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Changed start -> running
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Job sshd.service/start finished, result=done
Apr 26 08:03:29 f02d6a292ddd systemd[1]: Started OpenSSH server daemon.
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Failed to send unit change signal for sshd.service: Connection reset by peer
telnet檢測一下
telnet 10.10.20.198 2229
返回
Connecting to 10.10.20.198:2229...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.
SSH-2.0-OpenSSH_8.0
說明端口可以連接
ssh遠程登陸
但是讓我輸入密碼，但是我創(chuàng)建容器之后沒有地方創(chuàng)建密碼的地方
問題3-遠程未創(chuàng)建密碼
在docker desktop點進對應(yīng)的容器中創(chuàng)建密碼
passwd
返回
sh: passwd: command not found
沒有對應(yīng)服務(wù)，繼續(xù)安裝對應(yīng)服務(wù)
yum install passwd -y
安裝好后
passwd
返回
Changing password for user root.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
連續(xù)輸入兩次需要更改的密碼即可
再通過xshell遠程登陸，這次沒有問題，順利登陸容器
image-20230426161131382.png
至此，三個容器已經(jīng)順利登錄了，接下來就是按照服務(wù)器搭建cdh的方式開始搭建
搭建cdh
前面什么檢查防火墻、selinux那些都不需要了，openssh都沒有的容器，怎么可能有防火墻。
修改hosts文件
修改
vi /etc/hosts
發(fā)送其他容器
scp /etc/hosts 172.21.0.102:/etc
scp /etc/hosts 172.21.0.103:/etc
需要輸入密碼
配置免密碼登錄
進入root目錄
cd /root
生成公鑰與私鑰
ssh-keygen -t rsa
連續(xù)點三次回車
在/root/.ssh目錄下生成兩個文件id_rsa、id_rsa.pub（公鑰、私鑰）
每個節(jié)點都要生成?。?！
將每臺的公鑰發(fā)送至第一臺服務(wù)器，在其他服務(wù)器上執(zhí)行
ssh-copy-id e399964a01bd
需要輸密碼
復(fù)制第一臺服務(wù)器的認證到其他服務(wù)器
scp /root/.ssh/authorized_keys 666d932c563f:/root/.ssh
scp /root/.ssh/authorized_keys f02d6a292ddd:/root/.ssh
需要輸最后一次密碼
驗證
ssh 666d932c563f
ssh f02d6a292ddd
已經(jīng)不需要輸入密碼了，完成
設(shè)置時鐘同步
無需查看是否安裝過ntpdate，容器肯定沒安裝過，直接安裝
先通過yum源查看一下有沒有
yum search ntpdate
返回
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 1:18:04 ago on Wed Apr 26 07:35:55 2023.
No matches found.
居然沒有
但是查看centos7的iso文件里面是由對應(yīng)的包，此時需要使用我內(nèi)部使用的centos7的iso的源。
在/etc/yum.repo.d中在創(chuàng)建一個我內(nèi)部使用的源
再次搜索
yum search ntpdate
返回
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:08 ago on Wed Apr 26 08:58:29 2023.
========================================================================================= Name Exactly Matched: ntpdate ==========================================================================================
ntpdate.x86_64 : Utility to set the date and time via NTP
[root@e399964a01bd yum.repos.d]# yum search ntpdate
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:43 ago on Wed Apr 26 08:58:29 2023.
========================================================================================= Name Exactly Matched: ntpdate ==========================================================================================
ntpdate.x86_64 : Utility to set the date and time via NT
有了，安裝
yum install ntpdate -y
返回
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:03:34 ago on Wed Apr 26 08:58:29 2023.
Error:
Problem: problem with installed package
-7.61.1-18.el8.x86_64
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libcrypto.so.1.1()(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libcrypto.so.1.1(OPENSSL_1_1_0)(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libssl.so.1.1()(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libssl.so.1.1(OPENSSL_1_1_0)(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libssl.so.1.1(OPENSSL_1_1_1)(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires openssl-libs(x86-64) >= 1:1.1.1g, but none of the providers can be installed
- package libcurl-minimal-7.61.1-22.el8.x86_64 requires openssl-libs(x86-64) >= 1:1.1.1k, but none of the providers can be installed
- cannot install both openssl-libs-1:1.0.2k-19.el7.x86_64 and openssl-libs-1:1.1.1g-15.el8_3.x86_64
- cannot install both openssl-libs-1:1.0.2k-19.el7.x86_64 and openssl-libs-1:1.1.1k-4.el8.x86_64
- cannot install both openssl-libs-1:1.0.2k-19.el7.x86_64 and openssl-libs-1:1.1.1k-5.el8_5.x86_64
- package ntpdate-4.2.6p5-29.el7.centos.2.x86_64 requires libcrypto.so.10()(64bit), but none of the providers can be installed
- package ntpdate-4.2.6p5-29.el7.centos.2.x86_64 requires libcrypto.so.10(libcrypto.so.10)(64bit), but none of the providers can be installed
- package ntpdate-4.2.6p5-29.el7.centos.2.x86_64 requires libcrypto.so.10(OPENSSL_1.0.2)(64bit), but none of the providers can be installed
- conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
報錯，嚶嚶嚶
問題看起來應(yīng)該是安裝libcurl-minimal-7.61.1-18.el8.x86_64的時候，缺少了很多這個庫的依賴，libcrypto.so.1.1、libssl.so.1.1、openssl-libs
查看了yum源的問題，我從docker hub拉取的是centos8的鏡像，但是阿里的源提供的是centos7的源，導(dǎo)致無法使用，所以需要從頭再來一次，在docker hub找到一個centos7的鏡像，再來過。
在docker hub找鏡像
image-20230427144921136.png
這次找一個centos7，應(yīng)該沒問題了
拉取鏡像
docker pull centos:centos7@sha256:dead07b4d8ed7e29e98de0f4504d87e8880d4347859d839686a31da35a3b532f
查看鏡像
docker images
網(wǎng)絡(luò)創(chuàng)建過了就不需要創(chuàng)建網(wǎng)絡(luò)了，直接創(chuàng)建容器
重復(fù)前面的步驟
在yum中查找ntp
yum search ntp
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.tuna.tsinghua.edu.cn
* updates: ftp.riken.jp
================================================================================================ N/S matched: ntp ================================================================================================
ntp.x86_64 : The NTP daemon and utilities
ntp-doc.noarch : NTP documentation
ntp-perl.noarch : NTP utilities written in Perl
ntpdate.x86_64 : Utility to set the date and time via NTP
python-ntplib.noarch : Python module that offers a simple interface to query NTP servers
chrony.x86_64 : An NTP client/server
cyrus-imapd.x86_64 : A high-performance mail server with IMAP, POP3, NNTP and SIEVE support
fontpackages-devel.noarch : Templates and macros used to create font packages
fontpackages-filesystem.noarch : Directories used by font packages
fontpackages-tools.noarch : Tools used to check fonts and font packages
sntp.x86_64 : Standard Simple Network Time Protocol program
Name and summary matches only, use "search all" for everything.
這次直接有了
通過yum安裝
yum install ntp -y
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.tuna.tsinghua.edu.cn
* updates: ftp.riken.jp
Resolving Dependencies
--> Running transaction check
---> Package ntp.x86_64 0:4.2.6p5-29.el7.centos.2 will be installed
--> Processing Dependency: libopts.so.25()(64bit) for package: ntp-4.2.6p5-29.el7.centos.2.x86_64
--> Running transaction check
---> Package autogen-libopts.x86_64 0:5.18-5.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================================================================================================
Package Arch Version Repository Size
==================================================================================================================================================================================================================
Installing:
ntp x86_64 4.2.6p5-29.el7.centos.2 base 549 k
Installing for dependencies:
autogen-libopts x86_64 5.18-5.el7 base 66 k
Transaction Summary
==================================================================================================================================================================================================================
Install 1 Package (+1 Dependent package)
Total download size: 615 k
Installed size: 1.5 M
Downloading packages:
(1/2): autogen-libopts-5.18-5.el7.x86_64.rpm | 66 kB 00:00:00
(2/2): ntp-4.2.6p5-29.el7.centos.2.x86_64.rpm | 549 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.2 MB/s | 615 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : autogen-libopts-5.18-5.el7.x86_64 1/2
Installing : ntp-4.2.6p5-29.el7.centos.2.x86_64 2/2
Verifying : autogen-libopts-5.18-5.el7.x86_64 1/2
Verifying : ntp-4.2.6p5-29.el7.centos.2.x86_64 2/2
Installed:
ntp.x86_64 0:4.2.6p5-29.el7.centos.2
Dependency Installed:
autogen-libopts.x86_64 0:5.18-5.el7
Complete!
編輯ntp服務(wù)節(jié)點配置文件
vi /etc/ntp.conf
修改后文件內(nèi)容
1683255544796.png
重啟ntp服務(wù)
systemctl start ntpd
查看狀態(tài)
systemctl status ntpd
確認沒有問題
設(shè)置開機啟動
systemctl enable ntpd
編輯ntp客戶端節(jié)點配置文件
vi /etc/ntp.conf
修改后文件內(nèi)容
1683256090185.png
重啟ntp服務(wù)
systemctl start ntpd
查看狀態(tài)
systemctl status ntpd
查看配置是否生效
ntpq -p
返回
remote refid st t when poll reach delay offset jitter
==============================================================================
*a6d039150d75 LOCAL(0) 6 u 20 64 17 0.216 0.093 0.073
確認沒有問題
設(shè)置開機啟動
systemctl enable ntpd
主節(jié)點部署MySQL
查看是否安裝過MySQL
rpm -qa | grep -i mysql
沒有返回，說明沒有安裝過
安裝MySQL
下載rpm包
wget https://repo.mysql.com/mysql57-community-release-el7-10.noarch.rpm
安裝mysql源
yum localinstall -y mysql57-community-release-el7-10.noarch.rpm
檢查mysql源
yum repolist enabled | grep "mysql.*-community.*"
返回
mysql-connectors-community/x86_64 MySQL Connectors Community 203
mysql-tools-community/x86_64 MySQL Tools Community 129
mysql57-community/x86_64 MySQL 5.7 Community Server 504
安裝
yum -y install mysql-community-server
查看狀態(tài)
systemctl status mysqld.service
返回
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
啟動
systemctl restart mysqld.service
查看狀態(tài)
systemctl status mysqld.service
說明安裝正常
修改MySQL登錄密碼
查看默認密碼
grep 'temporary password' /var/log/mysqld.log
返回
2023-05-05T08:19:46.844203Z 1 [Note] A temporary password is generated for root@localhost: tBGvK=)I8oy?
密碼為：tBGvK=)I8oy?
修改登錄密碼
mysql -u root -p
輸入一次密碼
進入MySQL命令行
ALTER USER 'root'@'localhost' IDENTIFIED BY '（要改的密碼）';
小插曲，這里修改的是‘123456’的密碼，返回告訴我密碼不夠安全
返回
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
這里需要修改MySQL的密碼強度要求
set global validate_password_policy=LOW;
再去修改，告訴我還是不夠安全，原因是密碼長度要求為8位，再把長度修改為6位
set global validate_password_length=6;
再再次修改，成功
ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
返回
Query OK, 0 rows affected (0.01 sec)
退出，再次登錄驗證是否修改成功
quit
返回
Bye
登陸驗證
mysql -u root -p
輸入修改的新密碼，登陸成功
MySQL密碼修改完畢
修改MySQL配置文件
vi /etc/my.cnf
在文件中添加內(nèi)容
validate_password=off
character_set_server=utf8
init_connect='SET NAMES utf8'
重啟mysql，將配置進行生效
systemctl restart mysqld
此配置內(nèi)容不修改，創(chuàng)建CDH相關(guān)庫時會出現(xiàn)問題
允許root用戶遠程登錄
進入MySQL命令行
mysql -u root -p
輸入密碼
執(zhí)行
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '（密碼）' WITH GRANT OPTION;
FLUSH PRIVILEGES;
創(chuàng)建CDH相關(guān)mysql庫
CREATE DATABASE scm DEFAULT CHARACTER SET utf8;
CREATE USER 'scm'@'%'IDENTIFIED BY 'scm';
GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL PRIVILEGES ON *.* TO 'scm'@'%' IDENTIFIED BY 'scm' WITH GRANT OPTION;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8;
CREATE USER 'amon'@'%'IDENTIFIED BY 'amon';
GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL PRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY 'amon' WITH GRANT OPTION;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8;
CREATE USER 'hive'@'%'IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8;
CREATE USER 'hue'@'%'IDENTIFIED BY 'hue';
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL PRIVILEGES ON *.* TO 'hue'@'%' IDENTIFIED BY 'hue' WITH GRANT OPTION;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8;
CREATE USER 'oozie'@'%'IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY 'oozie' WITH GRANT OPTION;
FLUSH PRIVILEGES;
驗證數(shù)據(jù)庫是否創(chuàng)建
show databases;
返回
+--------------------+
| Database |
+--------------------+
| information_schema |
| amon |
| hive |
| hue |
| mysql |
| oozie |
| performance_schema |
| scm |
| sys |
+--------------------+
9 rows in set (0.00 sec)
創(chuàng)建成功
配置cdh相關(guān)的yum源
我這里有之前配置過的yum源，直接拿配置用，如果沒有配置過請參考http://www.itdecent.cn/p/6f120a99cbae第8節(jié)
所有服務(wù)器安裝JDK
在所有節(jié)點執(zhí)行
yum install -y oracle-j2sdk1.8-1.8.0+update181-1.x86_64
配置jdk環(huán)境
vi /etc/profile
添加內(nèi)容
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=:$JAVA_HOME/bin:$PATH
保存退出
生效
source /etc/profile
安裝cloudera-agent
對所有節(jié)點安裝daemons、agent
yum install -y python27 python2
yum install -y cloudera-manager-daemons
yum install -y cloudera-manager-agent.x86_64
修改所有節(jié)點agent配置
vi /etc/cloudera-scm-agent/config.ini
修改server_host值
server_host=a6d039150d75
在主節(jié)點安裝cloudera-manager-server及scm數(shù)據(jù)庫初始化
安裝server服務(wù)
yum install -y cloudera-manager-server
初始化scm數(shù)據(jù)庫
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm
小插曲，這里有個報錯
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/
../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.[ main] DbCommandExecutor INFO Unable to find JDBC driver for database type: MySQL
[ main] DbCommandExecutor ERROR JDBC Driver com.mysql.jdbc.Driver not found.
[ main] DbCommandExecutor ERROR Exiting with exit code 3
--> Error 3, giving up (use --force if you wish to ignore the error)
連接MySQL缺少一個jar包，jar包的路徑在“/usr/share/java”下，順著目錄查找就根本沒有這個目錄，秉著沒有啥就給啥的路子
就創(chuàng)建/usr/share/java目錄，把mysql-connector-java-5.1.46.jar，上傳，改成mysql-connector-java.jar名字
再執(zhí)行，通過
返回
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/
../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Sat May 06 05:42:40 UTC 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be estab
lished by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
啟動cloudera-manager-server及所有cloudera-manager-agent
在主節(jié)點啟動
systemctl start cloudera-scm-server
查看狀態(tài)
systemctl status cloudera-scm-server
返回
● cloudera-scm-server.service - Cloudera CM Server Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
Active: inactive (dead)
[root@a6d039150d75 java]# systemctl restart cloudera-scm-server
[root@a6d039150d75 java]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2023-05-06 05:48:50 UTC; 1s ago
Process: 6155 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
Main PID: 6157 (java)
CGroup: /docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/system.slice/cloudera-scm-server.service
└─6157 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server...
? 6157 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server...
May 06 05:48:50 a6d039150d75 systemd[1]: cloudera-scm-server.service: control process exited, code=exited status=0
May 06 05:48:50 a6d039150d75 systemd[1]: cloudera-scm-server.service got final SIGCHLD for state start-pre
May 06 05:48:50 a6d039150d75 systemd[1]: About to execute: /opt/cloudera/cm/bin/cm-server
May 06 05:48:50 a6d039150d75 systemd[1]: Forked /opt/cloudera/cm/bin/cm-server as 6157
May 06 05:48:50 a6d039150d75 systemd[1]: Cleaning 6157 from watches.
May 06 05:48:50 a6d039150d75 systemd[1]: Watching 6157 through watch_pids1.
May 06 05:48:50 a6d039150d75 systemd[1]: cloudera-scm-server.service changed start-pre -> running
May 06 05:48:50 a6d039150d75 systemd[1]: Job cloudera-scm-server.service/start finished, result=done
May 06 05:48:50 a6d039150d75 systemd[1]: Started Cloudera CM Server Service.
May 06 05:48:50 a6d039150d75 systemd[1]: Watching 6157 through watch_pids1.
所有節(jié)點啟動
systemctl start cloudera-scm-agent
查看狀態(tài)
systemctl status cloudera-scm-agent
返回
● cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2023-05-06 05:49:56 UTC; 21s ago
Main PID: 6356 (cmagent)
CGroup: /docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/system.slice/cloudera-scm-agent.service
└─6356 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
? 6356 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread tmpfs INFO Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread logging INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
May 06 05:50:02 a6d039150d75 cm[6356]: status_server: added process group
May 06 05:50:02 a6d039150d75 cm[6356]: flood: added process group
May 06 05:50:02 a6d039150d75 cm[6356]: /opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_pslinux.py:477: RuntimeWarning: dirty, writeback, mapped, commit_limit memory stats couldn't b...d were set to 0
May 06 05:50:02 a6d039150d75 cm[6356]: warnings.warn(msg, RuntimeWarning)
Hint: Some lines were ellipsized, use -l to show in full.
集群搭建
進入本機ip+7180端口
1683352347845.png
1683352384040.png
1683352406081.png
1683352437631.png
1683352481227.png
1683352526645.png
1683352553412.png
給集群起個名字
1683352553412.png
通過ip搜索可以，因為提前給所有主機安裝了agent，直接可以查找到
1683352598065.png
1683352716262.png
1683352728669.png
1683352746988.png
1683353530953.png
主要是修改swap分區(qū)的問題
臨時修改
sysctl vm.swappiness=10
永久修改
vi /etc/sysctl.conf
添加內(nèi)容
vm. swappiness=10
重新檢查
通過
基本已經(jīng)搭建完成，還有許多后續(xù)問題，繼續(xù)記錄
