1. 借鑒

Hadoop集群添加新節(jié)點(diǎn)
為什么Hive里，要用mysql?
centos7下安裝mysql5.7（rpm）
error: Failed dependencies: perl(Data::Dumper) is needed by MySQL-server-5.6.46-1.el7.x8
如何在DOCKER容器中運(yùn)行MySQL服務(wù)
rpm 完全卸載mysql
Tez 0.9安裝部署+hive on tez配置 + Tez-UI

2. 開(kāi)始

我們的節(jié)點(diǎn)規(guī)劃如下：

hive01[172.173.16.22]

鏡像準(zhǔn)備

hive是依托于hadoop的，所以hive機(jī)器上需要hadoop環(huán)境

docker hub 下載

  docker pull caiserkaiser/hadoop:2.7.2

構(gòu)建
caiser/hadoop:2.7.2 鏡像

創(chuàng)建自定義網(wǎng)絡(luò)

Docker 網(wǎng)絡(luò)操作

docker network create -d bridge --subnet "172.173.16.0/24" --gateway "172.173.16.1"  datastore_net

啟動(dòng)容器

docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hadoop:2.7.2

下載并配置hive

下載hive

拷貝到容器內(nèi)

docker cp ~/Downloads/apache-hive-1.2.2-bin.tar.gz e2d21b78fdb0:/opt/envs

解壓并重命名

mkdir hive-1.2.2 && tar -zxvf apache-hive-1.2.2-bin.tar.gz -C hive-1.2.2 --strip-components 1

配置

hive-env.sh

a. 備份

cp /opt/envs/hive-1.2.2/conf/hive-env.sh.template /opt/envs/hive-1.2.2/conf/hive-env.sh

b. 編輯hive-env.sh

 vi /opt/envs/hive-1.2.2/conf/hive-env.sh

c. 配置

①. 配置 HADOOP_HOME

 export HADOOP_HOME=/opt/envs/hadoop-2.7.2

②. 配置 HIVE_CONF_DIR

export HIVE_CONF_DIR=/opt/envs/hive-1.2.2/conf

安裝which

不安裝which，啟動(dòng)datanode時(shí)會(huì)報(bào)找不到文件的錯(cuò)誤

yum install which

安裝mysql

為啥需要mysql

[借鑒引用]
你在哪路徑下，執(zhí)行hive指令，就在哪路徑下生成metastore_db建一套數(shù)據(jù)庫(kù)文件，這樣是極其不合適的，公司里每個(gè)人若不一樣，則會(huì)顯得非?；祀s。導(dǎo)致員工之間無(wú)法公用交流，也不便于管理。

a . 重要如果安裝了mysql，先刪除，再重新安裝

 rpm -qa | grep -i mysql
 rpm -ev --nodeps xxx

 find / -name mysql
 find / -name my.cnf

 rm -rf  xxx/mysql
 rm -rf /etc/my.cnf

b. 下載mysql

 MySQL-client-5.6.48-1.el7.x86_64.rpm
 MySQL-server-5.6.48-1.el7.x86_64.rpm

c. 拷貝到容器

 docker cp ~/Downloads/MySQL-client-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs
 docker cp ~/Downloads/MySQL-server-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs

d. 安裝mysql服務(wù)端

① 安裝服務(wù)端

rpm -ivh MySQL-server-5.6.48-1.el7.x86_64.rpm

② 出現(xiàn)問(wèn)題:

/usr/bin/perl is needed by MySQL-server-5.6.48-1.el7.x86_64
libaio.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libaio.so.1(LIBAIO_0.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libaio.so.1(LIBAIO_0.4)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libnuma.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libnuma.so.1(libnuma_1.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
libnuma.so.1(libnuma_1.2)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
net-tools is needed by MySQL-server-5.6.48-1.el7.x86_64
perl(Data::Dumper) is needed by MySQL-server-5.6.48-1.el7.x86_64

③ 安裝以下依賴

 yum install -y perl perl-devel autoconf libaio numactl net-tools iproute

④ server安裝完成[看到以下內(nèi)容證明完成了]

A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.

You must change that password on your first connect,
no other statement but 'SET PASSWORD' will be accepted.
See the manual for the semantics of the 'password expired' flag.

Also, the account for the anonymous user has been removed.

In addition, you can run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test database.
This is strongly recommended for production servers.

See the manual for more instructions.

Please report any problems at http://bugs.mysql.com/

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

New default config file was created as /usr/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

e. 查看隨機(jī)生成的密碼
```
cat /root/.mysql_secret
```

f. 啟動(dòng)mysql

① 啟動(dòng)

/etc/init.d/mysql start
或 service mysql start

② 啟動(dòng)成功日志如下：

Starting MySQL.Logging to '/var/lib/mysql/e2d21b78fdb0.err'.
SUCCESS!

g. 安裝mysql客戶端

 rpm -ivh MySQL-client-5.6.48-1.el7.x86_64.rpm

h. 連接mysql，使用root用戶
```
mysql -uroot -p
```
i. 修改mysql root用戶密碼
```
set password=password('密碼');
quit;
```

j. 修改用戶表，將host表的host列修改為任意ip可訪問(wèn)，并刪除其他host（否則在后面啟動(dòng)時(shí)會(huì)報(bào)沒(méi)有權(quán)限）

use mysql;

update user set host = '%' where host = 'localhost';

delete from user where host = 'e2d21b78fdb0';
delete from user where Host='127.0.0.1';
delete from user where Host='::1';

flush privileges;

k. 安裝mysql-connector-java.jar

① 下載mysql-connector-java.jar

② 將jar包拷貝到 hive-1.2.2/lib 下

配置Metastore到Mysql

在hive-1.2.2/conf下創(chuàng)建hive-site.xml
```
vi hive-site.xml
```

配置如下:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
   <property>
     <name>javax.jdo.option.ConnectionURL</name>
     <value>jdbc:mysql://hive01:3306/metastore?createDatabaseIfNotExist=true</value>
     <description>JDBC connect string for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionDriverName</name>
     <value>com.mysql.jdbc.Driver</value>
     <description>Driver class name for a JDBC metastore</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionUserName</name>
     <value>root</value>
     <description>username to use against metastore database</description>
   </property>
   <property>
     <name>javax.jdo.option.ConnectionPassword</name>
     <value>密碼</value>
     <description>password to use against metastore database</description>
   </property>
   <!-- 顯示當(dāng)前數(shù)據(jù)庫(kù)，顯示查詢表的頭信息 -->
   <property>
     <name>hive.cli.print.header</name>
     <value>true</value>
   </property>
   <property>
     <name>hive.cli.print.current.db</name>
     <value>true</value>
   </property>
</configuration>

配置運(yùn)行日志便于查詢

備份

cp /opt/envs/hive-1.2.2/conf/hive-log4j.properties.template /opt/envs/hive-1.2.2/conf/hive-log4j.properties

修改

vi /opt/envs/hive-1.2.2/conf/hive-log4j.properties

修改hive.log.dir屬性
```
hive.log.dir=/opt/envs/hive-1.2.2/logs
```

保存為鏡像并移除容器

docker commit e2d21b78fdb0 caiser/hive:1.2.2
docker rm e2d21b78fdb0

啟動(dòng)容器

docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hive:1.2.2 bin/bash

配置ssh免密登錄

進(jìn)入容器
```
docker exec -it hive01 /bin/bash
```
到~/.ssh目錄下生成秘鑰
```
ssh-keygen -t rsa
```
拷貝秘鑰到hadoop01，hadoop02和hadoop03

a.[如果沒(méi)開(kāi)啟]開(kāi)啟ssh服務(wù)[ps -ef | grep ssh]
```
/usr/sbin/sshd -D &
```
b. 拷貝秘鑰到hadoop01，hadoop02，hadoop03
```
ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03
```
hadoop01,hadoop02和hadoop03依次執(zhí)行
```
ssh-copy-id hive01
```

向hadoop集群中添加節(jié)點(diǎn)

動(dòng)態(tài)
① 在hive01機(jī)器的hadoop/sbin目錄下啟動(dòng)datanode：
```
hadoop-daemon.sh start datanode
```
② 在hive01機(jī)器的hadoop/sbin目錄下啟動(dòng)nodemanager
```
yarn-daemon.sh start nodemanager
```
③ jps查看datanode和nodemanager是否已啟動(dòng)
④ 回到namenode節(jié)點(diǎn)打印集群信息，或網(wǎng)頁(yè)登錄50070端口查看節(jié)點(diǎn)數(shù)量
```
hdfs dfsadmin -report
```
靜態(tài)

① 停止hadoop集群
② 保證各節(jié)點(diǎn)hadoop的配置文件內(nèi)容一致
③ 分別配置4個(gè)節(jié)點(diǎn)hadoop的salves文件
```
 hadoop01
 hadoop02
 hadoop03
 hive01
```
④ 重啟hadoop集群

啟動(dòng)hive

啟動(dòng)
```
bin/hive
```

看到這種就是成了

Logging initialized using configuration in file:/opt/envs/hive-1.2.2/conf/hive-log4j.properties
hive (default)>

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av

Docker 構(gòu)建 Hive 1.2.2 單節(jié)點(diǎn)

Docker 構(gòu)建 Hive 1.2.2 單節(jié)點(diǎn)

1. 借鑒

2. 開(kāi)始

鏡像準(zhǔn)備

創(chuàng)建自定義網(wǎng)絡(luò)

啟動(dòng)容器

下載并配置hive

配置

hive-env.sh

安裝which

安裝mysql

配置Metastore到Mysql

配置運(yùn)行日志便于查詢

保存為鏡像并移除容器

啟動(dòng)容器

配置ssh免密登錄

向hadoop集群中添加節(jié)點(diǎn)

啟動(dòng)hive

3. 大功告成

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九 欧美,1769亚洲,黄色成人av

Docker 構(gòu)建 Hive 1.2.2 單節(jié)點(diǎn)

1. 借鑒

2. 開(kāi)始

鏡像準(zhǔn)備

創(chuàng)建自定義網(wǎng)絡(luò)

啟動(dòng)容器

下載并配置hive

配置

hive-env.sh

安裝which

安裝mysql

配置Metastore到Mysql

配置運(yùn)行日志便于查詢

保存為鏡像并移除容器

啟動(dòng)容器

配置ssh免密登錄

向hadoop集群中添加節(jié)點(diǎn)

啟動(dòng)hive

3. 大功告成

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av