一、Hadoop Hbase問(wèn)題
前幾天寫(xiě)了安裝HBase,結(jié)果這幾天發(fā)現(xiàn)通過(guò)hbase shell執(zhí)行進(jìn)去,status總是報(bào)錯(cuò)
#命令進(jìn)入hbase shell
hbase shell
#查看hbase狀態(tài)
status
#總是報(bào)以下錯(cuò)誤
ERROR:Can't get master address from Zookeeper;znode data==null
經(jīng)過(guò)了三天的排查,終于發(fā)現(xiàn)了是,hadoop 2.7.0的版本太舊了,需要重新安裝個(gè)更新的hadoop 2.9.3版本,下面咱們開(kāi)始把hadoop和hbase一起重新安裝一遍,就當(dāng)復(fù)習(xí)了。
二、清理原有服務(wù)配置服務(wù)器以及安裝java
#停止并刪除原有的hadoop路徑
cd /usr/local/hadoop/sbin
./stop-all.sh
cd ../..
rm -rf ./hadoop
#編輯yum安裝源
vi /etc/yum.repos.d/CentOS-Base.repo
#修改為如下內(nèi)容
[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/
gpgcheck=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6
#released updates
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/updates/x86_64/
gpgcheck=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/extras/x86_64/
gpgcheck=1
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/centosplus/x86_64/
gpgcheck=1
enabled=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6
#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/contrib/x86_64/
gpgcheck=1
enabled=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/6.8/os/x86_64/RPM-GPG-KEY-CentOS-6
#修改https驗(yàn)證,增加下面的忽略ssl驗(yàn)證
vi /etc/yum.conf
sslverify=false
#安裝jdk
yum install java-1.8.0-openjdk.x86_64
#編輯環(huán)境變量.bash_profile
vi ~/.bash_profile
#.bash_profile修改具體內(nèi)容如下
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export PATH
#編輯環(huán)境變量.bashrc
vi ~/.bashrc
#.bashrc修改具體內(nèi)容如下
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export PATH
#生效環(huán)境變量
source ~/.bashrc
source ~/.bash_profile
三、配置Hadoop
#先將hadoop和hbase文件復(fù)制到docker中,此執(zhí)行在我的mac中
docker cp Downloads/hadoop-2.9.2.tar.gz master:/usr/local
docker cp Downloads/hbase-1.7.1-bin.tar.gz master:/usr/local
#解壓hadoop并修改路徑名稱(chēng)
cd /usr/local/
tar -xzvf hadoop-2.9.2.tar.gz
mv hadoop-2.9.2 hadoop
#進(jìn)入hadoop配置路徑
cd /usr/local/hadoop/etc/hadoop/
#修改core-site.xml
vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
</configuration>
#修改hdfs-site.xml
vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/hadoop/data</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/hadoop/name</value>
</property>
</configuration>
#修改mapped-site.xml
vi mapped-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
#備份并修改yarn-site.xml
cp yarn-site.xml yarn-site.xml.bak
rm yarn-site.xml
vi yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value> </property> <property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
#建立路徑,與上面的配置保持一致
mkdir -p /hadoop/name
mkdir -p /hadoop/data
mkdir -p /hadoop/tmp
#到/usr/local/hadoop/bin/目錄下運(yùn)行,驗(yàn)證初始化是否有問(wèn)題
cd /usr/local/hadoop/bin/
./hadoop namenode -format./hadoop namenode -format
--啟動(dòng)集群
cd ../sbin
./start-all.sh
--jps查看服務(wù)是否啟動(dòng)成功
jps
223 DataNode
612 ResourceManager
129 NameNode
704 NodeManager
1725 Jps
388 SecondaryNameNode
四、配置HBase
#解壓,在Hadoop部分已經(jīng)cp到docker中了
tar -xvzf hbase-1.7.1-bin.tar.gz
#編輯配置文件hbase-site.xml
cd /usr/local/hbase-1.7.1/conf
vi hbase-site.xml
<configuration>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>./tmp</value>
</property>
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>16010</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>16201</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>16301</value>
</property>
</configuration>
#編輯配置文件hbase-env.sh
vi hbase-env.sh
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
#配置hbase環(huán)境變量
vi ~/.bash_profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
export HADOOP_HOME=/usr/local/hadoop
export HBASE_HOME=/usr/local/hbase-1.7.1
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin
export PATH
source ~/.bash_profile
#編輯環(huán)境變量.bashrc
vi ~/.bashrc
#.bashrc修改具體內(nèi)容如下
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
export HADOOP_HOME=/usr/local/hadoop
export HBASE_HOME=/usr/local/hbase-1.7.1
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin
export PATH
source ~/.bashrc
#建立hbase在Hadoop上的路徑,與上面的hbase-site.xml配置的路徑對(duì)應(yīng)
hadoop fs -mkdir /hbase
#啟動(dòng)hbase
./start-hbase.sh
#查看啟動(dòng)狀態(tài),這次出現(xiàn)了HMaster,HRegionServer,HQuorumPeer,啟動(dòng)成功
jps
3505 DataNode
7711 HMaster
8276 Jps
7824 HRegionServer
3699 SecondaryNameNode
3952 NodeManager
7611 HQuorumPeer
3857 ResourceManager
3408 NameNode
#接下來(lái)看看一開(kāi)始的問(wèn)題,執(zhí)行hbase shell進(jìn)入shell
hbase shell
#進(jìn)入shell命令行
hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load
五、HBase常用命令
#查看HBase數(shù)據(jù)庫(kù)狀態(tài)
hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load
#查看當(dāng)前操作HBase的用戶
hbase(main):002:0> whoami
root (auth:SIMPLE)
groups: root
#查看所有表
hbase(main):003:0> list
TABLE
0 row(s) in 0.0950 seconds
=> []
#建表
hbase(main):004:0> create 'census','personal','professional'
0 row(s) in 1.6170 seconds
=> Hbase::Table - census
#再次查看所有表,可以看到剛建立的census
hbase(main):005:0> list
TABLE
census
1 row(s) in 0.0170 seconds
=> ["census"]
#通過(guò)describe查看表census信息
hbase(main):006:0> describe 'census'
Table census is ENABLED
census
COLUMN FAMILIES DESCRIPTION
{NAME => 'personal', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE =>
'0'}
{NAME => 'professional', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOP
E => '0'}
2 row(s) in 0.1050 seconds
#查看census數(shù)據(jù)條數(shù)
hbase(main):007:0> count 'census'
0 row(s) in 0.1060 seconds
=> 0
#插入數(shù)據(jù),注意按照順序來(lái)分別是:表名'census',行ID 1,列族personal,列名name,值Zhang San
hbase(main):009:0> put 'census', 1,'personal:name','Zhang San'
0 row(s) in 0.1590 seconds
#插入數(shù)據(jù)personal列族數(shù)據(jù)中的婚姻狀態(tài)
hbase(main):010:0> put 'census', 1,'personal:marital_stauts','unmarried'
0 row(s) in 0.0090 seconds
#查看數(shù)據(jù),id都為1表明是同一條數(shù)據(jù),另外注意每條數(shù)據(jù)有個(gè)timestamp,表示這條數(shù)據(jù)的插入時(shí)間戳,以便來(lái)區(qū)分不同版本
hbase(main):011:0> scan 'census'
ROW COLUMN+CELL
1 column=personal:marital_stauts, timestamp=1648028740145, value=unmarried
1 column=personal:name, timestamp=1648028690316, value=Zhang San
1 row(s) in 0.0260 seconds
#繼續(xù)插入數(shù)據(jù)
hbase(main):012:0> put 'census', 1,'personal:gender','male'
0 row(s) in 0.0120 seconds
hbase(main):014:0> put 'census', 1,'professional:employed','yes'
0 row(s) in 0.0130 seconds
hbase(main):016:0> put 'census', 1,'professional:education_level','high scholl'
0 row(s) in 0.0070 seconds
hbase(main):017:0> put 'census', 1,'professional:filed','construction'
0 row(s) in 0.0110 seconds
#查看數(shù)據(jù)
hbase(main):018:0> scan 'census'
ROW COLUMN+CELL
1 column=personal:gender, timestamp=1648030369641, value=male
1 column=personal:marital_stauts, timestamp=1648028740145, value=unmarried
1 column=personal:name, timestamp=1648028690316, value=Zhang San
1 column=professional:education_level, timestamp=1648030509236, value=high scholl
1 column=professional:employed, timestamp=1648030420105, value=yes
1 column=professional:filed, timestamp=1648030551921, value=construction
#繼續(xù)插入新ID
hbase(main):019:0> put 'census', 3,'personal:name','li si'
0 row(s) in 0.0280 seconds
hbase(main):020:0> put 'census', 3,'personal:marital_stauts','married'
0 row(s) in 0.0150 seconds
hbase(main):021:0> put 'census', 3,'personal:spouse','zhang san'
0 row(s) in 0.0140 seconds
hbase(main):016:0> put 'census', 3,'professional:education_level','middle scholl'
0 row(s) in 0.5720 seconds
#查看可以看到兩條數(shù)據(jù)
hbase(main):002:0> scan 'census'
ROW COLUMN+CELL
1 column=personal:gender, timestamp=1648030369641, value=male
1 column=personal:marital_stauts, timestamp=1648028740145, value=unmarried
1 column=personal:name, timestamp=1648028690316, value=Zhang San
1 column=professional:education_level, timestamp=1648030509236, value=high scholl
1 column=professional:employed, timestamp=1648030420105, value=yes
1 column=professional:filed, timestamp=1648030551921, value=construction
3 column=personal:marital_stauts, timestamp=1648030685792, value=married
3 column=personal:name, timestamp=1648030680907, value=li si
3 column=personal:spouse, timestamp=1648030690084, value=zhang san
3 column=professional:education_level, timestamp=1648030880173, value=middle scholl
2 row(s) in 0.1200 seconds
#更新數(shù)據(jù)
hbase(main):003:0> put 'census', 3,'professional:education_level','high scholl'
0 row(s) in 0.0260 seconds
#可以看到學(xué)歷變成了高中
hbase(main):004:0> scan 'census'
ROW COLUMN+CELL
1 column=personal:gender, timestamp=1648030369641, value=male
1 column=personal:marital_stauts, timestamp=1648028740145, value=unmarried
1 column=personal:name, timestamp=1648028690316, value=Zhang San
1 column=professional:education_level, timestamp=1648030509236, value=high scholl
1 column=professional:employed, timestamp=1648030420105, value=yes
1 column=professional:filed, timestamp=1648030551921, value=construction
3 column=personal:marital_stauts, timestamp=1648030685792, value=married
3 column=personal:name, timestamp=1648030680907, value=li si
3 column=personal:spouse, timestamp=1648030690084, value=zhang san
3 column=professional:education_level, timestamp=1648031013249, value=high scholl
2 row(s) in 0.0810 seconds
#最后我們?cè)黾右粋€(gè)id為2
hbase(main):005:0> put 'census', 2,'personal:name','wang wu'
0 row(s) in 0.0200 seconds
#查詢(xún)結(jié)果可以看到,ROW的id是按照升序排列的
hbase(main):006:0> scan 'census'
ROW COLUMN+CELL
1 column=personal:gender, timestamp=1648030369641, value=male
1 column=personal:marital_stauts, timestamp=1648028740145, value=unmarried
1 column=personal:name, timestamp=1648028690316, value=Zhang San
1 column=professional:education_level, timestamp=1648030509236, value=high scholl
1 column=professional:employed, timestamp=1648030420105, value=yes
1 column=professional:filed, timestamp=1648030551921, value=construction
2 column=personal:name, timestamp=1648031084938, value=wang wu
3 column=personal:marital_stauts, timestamp=1648030685792, value=married
3 column=personal:name, timestamp=1648030680907, value=li si
3 column=personal:spouse, timestamp=1648030690084, value=zhang san
3 column=professional:education_level, timestamp=1648031013249, value=high scholl
3 row(s) in 0.0600 seconds
#數(shù)一下是3條數(shù)據(jù)了
hbase(main):007:0> count 'census'
3 row(s) in 0.0270 seconds
=> 3
#查詢(xún)id為1的數(shù)據(jù)
hbase(main):008:0> get 'census',1
COLUMN CELL
personal:gender timestamp=1648030369641, value=male
personal:marital_stauts timestamp=1648028740145, value=unmarried
personal:name timestamp=1648028690316, value=Zhang San
professional:education_level timestamp=1648030509236, value=high scholl
professional:employed timestamp=1648030420105, value=yes
professional:filed timestamp=1648030551921, value=construction
1 row(s) in 0.0700 seconds
#查詢(xún)id為1的數(shù)據(jù)的name字段的值
hbase(main):011:0> get 'census',1,'personal:name'
COLUMN CELL
personal:name timestamp=1648028690316, value=Zhang San
1 row(s) in 0.0400 seconds
#查詢(xún)id為1的數(shù)據(jù)的name和employed字段的值
hbase(main):012:0> get 'census',1,'personal:name','professional:employed'
COLUMN CELL
personal:name timestamp=1648028690316, value=Zhang San
professional:employed timestamp=1648030420105, value=yes
1 row(s) in 0.0150 seconds
#查看所有的,只顯示name字段值
hbase(main):014:0> scan 'census',{COLUMN=>['personal:name']}
ROW COLUMN+CELL
1 column=personal:name, timestamp=1648028690316, value=Zhang San
2 column=personal:name, timestamp=1648031084938, value=wang wu
3 column=personal:name, timestamp=1648030680907, value=li si
3 row(s) in 0.0150 seconds
#查看所有的,只顯示name字段值,只查詢(xún)一條
hbase(main):015:0> scan 'census',{COLUMN=>['personal:name'],LIMIT=>1}
ROW COLUMN+CELL
1 column=personal:name, timestamp=1648028690316, value=Zhang San
1 row(s) in 0.0210 seconds
#查看所有的,只顯示name字段值,從第二條數(shù)據(jù)開(kāi)始只查詢(xún)一條
hbase(main):017:0> scan 'census',{COLUMN=>['personal:name'],LIMIT=>1,STARTROW=>"2"}
ROW COLUMN+CELL
2 column=personal:name, timestamp=1648031084938, value=wang wu
1 row(s) in 0.0250 seconds
#刪除數(shù)據(jù)
hbase(main):019:0> delete 'census',1,'personal:marital_stauts'
0 row(s) in 0.0540 seconds
#查看id為1的數(shù)據(jù),沒(méi)有了婚姻狀態(tài)
hbase(main):021:0> get 'census',1
COLUMN CELL
personal:gender timestamp=1648030369641, value=male
personal:name timestamp=1648028690316, value=Zhang San
professional:education_level timestamp=1648030509236, value=high scholl
professional:employed timestamp=1648030420105, value=yes
professional:filed timestamp=1648030551921, value=construction
1 row(s) in 0.0180 seconds
#刪除表前,需要先disable表
hbase(main):023:0> disable 'census'
0 row(s) in 2.3830 seconds
#如果不想刪除則enable表即可
hbase(main):024:0> enable 'census'
0 row(s) in 1.3250 seconds
#沒(méi)有disable的表刪除會(huì)報(bào)錯(cuò)
hbase(main):025:0> drop 'census'
ERROR: Table census is enabled. Disable it first.
Here is some help for this command:
Drop the named table. Table must first be disabled:
hbase> drop 't1'
hbase> drop 'ns1:t1'
#刪除表前,需要先disable表
hbase(main):026:0> disable 'census'
0 row(s) in 2.2280 seconds
#刪除表
hbase(main):027:0> drop 'census'
0 row(s) in 1.3040 seconds
#再次查看已經(jīng)沒(méi)有該表了
hbase(main):028:0> list
TABLE
0 row(s) in 0.0290 seconds
=> []
終于把這部分完成了,相當(dāng)于又把hadoop重新搭建了好幾遍才解決這個(gè)問(wèn)題,其實(shí)就是幾個(gè)配置的錯(cuò)誤,來(lái)回查看才發(fā)現(xiàn)問(wèn)題,網(wǎng)上找的帖子都是騙子,一點(diǎn)都不好用。。。。
謝各位的閱讀,謝謝您動(dòng)動(dòng)手指點(diǎn)贊,萬(wàn)分感謝各位。另外以下是我之前寫(xiě)過(guò)的文章,感興趣的可以點(diǎn)進(jìn)去繼續(xù)閱讀。
歷史文章
Hadoop系列-入門(mén)安裝
Hadoop系列-HDFS命令
Hadoop系列-Hive安裝
Hadoop系列-Hive數(shù)據(jù)庫(kù)常見(jiàn)SQL命令
Hadoop系列-HBase數(shù)據(jù)庫(kù)
Hadoop系列-HBase數(shù)據(jù)庫(kù)(二)
Hadoop系列-HBase數(shù)據(jù)庫(kù)JAVA篇
Hadoop系列-Spark安裝以及HelloWorld
Hadoop系列-MapReduce小例子
Hadoop系列-Spark小例子
JAVA面試匯總(五)數(shù)據(jù)庫(kù)(一)
JAVA面試匯總(五)數(shù)據(jù)庫(kù)(二)
JAVA面試匯總(五)數(shù)據(jù)庫(kù)(三)
JAVA面試匯總(四)JVM(一)
JAVA面試匯總(四)JVM(二)
JAVA面試匯總(四)JVM(三)
JAVA面試匯總(三)集合(一)
JAVA面試匯總(三)集合(二)
JAVA面試匯總(三)集合(三)
JAVA面試匯總(三)集合(四)
JAVA面試匯總(二)多線程(一)
JAVA面試匯總(二)多線程(二)
JAVA面試匯總(二)多線程(三)
JAVA面試匯總(二)多線程(四)
JAVA面試匯總(二)多線程(五)
JAVA面試匯總(二)多線程(六)
JAVA面試匯總(二)多線程(七)
JAVA面試匯總(一)Java基礎(chǔ)知識(shí)