1、運行環(huán)境
主機IP 主機名
xxx.xxx.xxx.xxx po-master1
xxx.xxx.xxx.xxx po-master2
xxx.xxx.xxx.xxx po-slave1
xxx.xxx.xxx.xxx po-slave2
xxx.xxx.xxx.xxx po-slave3
2、配置主機名(分別在五臺機器上執(zhí)行)
vi /etc/sysconfig/network
hostname +主機名
例如: hostname po-master1
3、配置映射關系(把以下五條命令在五臺機器上執(zhí)行)
echo "xxx.xxx.xxx.xxx po-master1">>/etc/hosts
echo "xxx.xxx.xxx.xxx po-master2">>/etc/hosts
echo "xxx.xxx.xxx.xxx po-slave1">>/etc/hosts
echo "xxx.xxx.xxx.xxx po-slave2">>/etc/hosts
echo "xxx.xxx.xxx.xxx po-slave3">>/etc/hosts
4、安裝JDK(在po-master1上執(zhí)行)
4.1、下載JDK安裝包:jdk-8u102-linux-x64.tar.gz
注:作者放到/soft/java具體位置可自行安排
4.2、 安裝
cd /soft/java
mkdir jdk1.8.0_121
rpm -ivh jdk-7u76-linux-x64.rpm --prefix=/soft/java
ln -s -f jdk1.8.0_121/ jdk---創(chuàng)建連接
5、開放端口(五臺機器上都需要配置)
/sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEP
/sbin/iptables -A INPUT -s 0.0.0.0/0 -p tcp --dport 22 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 22 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50010 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 1004 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50075 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 1006 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50020 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8020 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50070 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50470 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50090 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 50495 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8485 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8480 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8032 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8030 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8031 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8033 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8088 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8040 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8042 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8041 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 10020 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 13562 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 19888 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 60000 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 60010 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 60020 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 60030 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 2181 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 2888 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 3888 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8080 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8085 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 9090 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 9095 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 9090 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 9083 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 10000 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 16000 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 2181 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 2888 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 3888 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 3181 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 4181 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 8019 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 9010 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 11000 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 11001 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 14000 -j ACCEPT
/sbin/iptables -A INPUT -s x.x.x.x -p tcp --dport 14001 -j ACCEPT
/etc/rc.d/init.d/iptables save
PS:關閉端口詳解-參考CDH官網(wǎng)
https://www.cloudera.com/documentation/cdh/5-1-x/CDH5-Installation-Guide/cdh5ig_ports_cdh5.html
測試(可忽略)
/etc/init.d/iptables status
6、五臺機器配置互相免秘鑰登陸
6.1、創(chuàng)建ssh文件
如果已經(jīng)創(chuàng)建不要覆蓋
cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys
分別把五臺機器的公鑰加載到authorized_keys
6.2、
vi /etc/ssh/sshd_config
打開如下內(nèi)容
HostKey /etc/ssh/ssh_host_rsa_key
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
6.3、重啟
/etc/init.d/sshd restart
6.4、測試ssh
ssh po-master1
ssh po-master2
ssh po-slave1
ssh po-slave2
ssh po-slave3
7、向其他機器分發(fā)jdk
scp -rp /soft/java/ root@po-master2:/soft/java
scp -rp /soft/java/ root@po-salve1:/soft/java
scp -rp /soft/java/ root@po-salve2:/soft/java
scp -rp /soft/java/ root@po-salve3:/soft/java
8、配置環(huán)境變量(分別在五臺機器上執(zhí)行)
echo "export JAVA_HOME=/soft/java/jdk" >> /etc/profile
echo "export PATH=$PATH:$HOME/bin:$JAVA_HOME:$JAVA_HOME/bin:/usr/bin/" >> /etc/profile
echo "export CLASSPATH=.:$JAVA_HOME/lib" >> /etc/profile
. /etc/profile
9、配置NTP服務器和客戶端(因為使用阿里云此處省略)
10、配置mysql
10.1、上傳mysql文件(博主放到/soft/mysql目錄下)
10.2、解壓
cd /soft/mysql
tar -zxvf mysql-5.7.17-linux-glibc2.5-x86_64.tar.gz -C /usr/local
10.3、將目錄重命名
cd /usr/local
mv mysql-5.7.17-linux-glibc2.5-x86_64/ mysql
10.4、創(chuàng)建data目錄
mkdir /usr/local/mysql/data
yum install libaio
10.5、安裝mysql
cd /usr/local/mysql/bin
./mysql_install_db --user=root --basedir=/usr/local/mysql --datadir=//usr/local/mysql/data
10.5.1、官網(wǎng)下載yum源
https://dev.mysql.com/downloads/repo/yum/
10.5.2、安裝yum源
yum localinstall mysql57-community-release-el6-9.noarch.rpm
10.5.3、安裝mysql
yum install mysql-community-server
10.5.4、創(chuàng)建組和用戶
groupadd mysql
useradd mysql -g mysql
10.5.5、修改配置文件開啟二進制日志
vi /etc/my.cnf (在[mysqld]下面添加如下內(nèi)容)
server-id=1
log-bin=/home/mysql/log/logbin.log
10.5.6、開啟服務
service mysqld start
10.5.7、查看mysql默認的密碼
grep 'temporary password' /var/log/mysqld.log
10.5.8、根據(jù)密碼進入mysql
mysql -u root -p
ALTER USER 'root'@'localhost' IDENTIFIED BY 'xxxxxx';
例如:
ALTER USER 'root'@'localhost' IDENTIFIED BY 'xxxxxx';
Query OK, 0 rows affected (0.01 sec)
注:MySQL's
validate_password plugin is installed by default. This will require
that passwords contain at least one upper case letter, one lower case
letter, one digit, and one special character, and that the total
password length is at least 8 characters.
10.5.9、授權(給其他四臺機器授權)
grant all privileges on oozie.* to 'oozie'@'localhost' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on oozie.* to 'oozie'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on oozie.* to 'oozie'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on oozie.* to 'oozie'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on oozie.* to 'oozie'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
GRANT all privileges on *.* to 'root'@'localhost' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on hive.* to 'hive'@'localhost' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on hive.* to 'hive'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on hive.* to 'hive'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on hive.* to 'hive'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
grant all privileges on hive.* to 'hive'@'xxx.xxx.xxx.xxx' IDENTIFIED BY 'xxxxxx' WITH GRANT OPTION;
flush privileges;
關于新版本的賬戶說明:
https://dev.mysql.com/doc/refman/5.7/en/adding-users.html
10.5.10、創(chuàng)建數(shù)據(jù)庫
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
11、安裝cloudera manager
11.1、下載
地址:http://archive-primary.cloudera.com/cm5/cm/5/
放在/soft/bigdata/clouderamanager下
cd /soft/bigdata/clouderamanager
tar -xvf cloudera-manager-wheezy-cm5.10.0_amd64.tar.gz
11.2、創(chuàng)建用戶(所有節(jié)點)
useradd --system --home=/soft/bigdata/clouderamanager/cm-5.10.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
11.3、修改主機名和端口號
cd /soft/bigdata/clouderamanager/cm-5.10.0/etc/cloudera-scm-agent
vi config.ini
server_host=po-master1
server_port=7182
11.4、下載mysql驅(qū)動包
下載mysql-connector-java-5.1.7-bin.jar放到
/soft/bigdata/clouderamanager/cm-5.10.0/share/cmf/lib 目錄下
11.5、為Cloudera Manager 5建立數(shù)據(jù)庫
/soft/bigdata/clouderamanager/cm-5.10.0/share/cmf/schema/scm_prepare_database.sh mysql scm -hlocalhost -uroot -pxxxxxx --scm-host localhost scm xxxxxx scm
格式是:scm_prepare_database.sh 數(shù)據(jù)庫類型 數(shù)據(jù)庫 服務器 用戶名 密碼 –scm-host Cloudera_Manager_Server所在的機器,后面那三個不知道代表什么,直接照抄官網(wǎng)的了。
開啟Cloudera Manager 5 Server端。
11.6、向其他機器分發(fā)CDH
scp -rp /soft/bigdata/clouderamanager root@po-master2:/soft/bigdata
scp -rp /soft/bigdata/clouderamanager root@po-slave1:/soft/bigdata
scp -rp /soft/bigdata/clouderamanager root@po-slave2:/soft/bigdata
scp -rp /soft/bigdata/clouderamanager root@po-slave3:/soft/bigdata
11.7、準備Parcels,用以安裝CDH5
放在/soft/bigdata/clouderamanager/cloudera/parcel-repo,路徑必須包含cloudera/parcel-repo
官方地址:
http://archive.cloudera.com/cdh5/parcels
需要下載以下兩個文件
? CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel
? manifest.json
打開 manifest.json找到CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel的hash值里的內(nèi)容
"hash": "52f95da433f203a05c2fd33eb0f144e6a5c9d558"
echo '52f95da433f203a05c2fd33eb0f144e6a5c9d558' >> CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha
11.8、啟動
/soft/bigdata/clouderamanager/cm-5.10.0/etc/init.d/cloudera-scm-server start(主節(jié)點啟動)
/soft/bigdata/clouderamanager/cm-5.10.0/etc/init.d/cloudera-scm-agent start(所有節(jié)點上啟動)
11.9、登陸
http://po-master1:7180
默認用戶密碼都是admin

點擊繼續(xù)

選擇免費的點擊繼續(xù)

勾選機器

點擊更多選項修改parcel路徑
/soft/bigdata/clouderamanager/cloudera/parcel-repo

需要重啟所有節(jié)點的服務
/soft/bigdata/clouderamanager/cm-5.10.0/etc/init.d/cloudera-scm-server restart(主節(jié)點啟動)
/soft/bigdata/clouderamanager/cm-5.10.0/etc/init.d/cloudera-scm-agent restart(所有節(jié)點上啟動)
選擇如下內(nèi)容點擊繼續(xù)

等待安裝...

安裝完成,點擊繼續(xù)

安裝過程有個小提示:
已啟用透明大頁面壓縮,可能會導致重大性能問題。請運行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”以禁用此設置,然后將同一命令添加到 /etc/rc.local 等初始腳本中,以便在系統(tǒng)重啟時予以設置。以下主機將受到影響:

選擇自定義服務,選擇自己需要的服務




等待安裝

安裝過程中會遇到錯誤:
是缺少jdbc驅(qū)動把文件考入到lib下即可

配置NameNode HA
進入HDFS界面,點擊“啟用High Availability”

輸入NameService名稱,這里設置為:nameservice1,點擊繼續(xù)按鈕。

配置JourNode的路徑,修改為/opt/dfs/jn

錯誤整理;
Fatal error during KafkaServer startup. Prepare to shutdownkafka.common.InconsistentBrokerIdException: Configured broker.id 52 doesn’t match stored broker.id 102 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:648)at kafka.server.KafkaServer.startup(KafkaServer.scala:187)at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)at kafka.Kafka$.main(Kafka.scala:67)at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
進入到/var/local/kafka/data目錄查看meta.propertie里面的kakfa 的broker id是什么
[main]: Metastore Thrift Server threw an exception…javax.jdo.JDOFatalInternalException: Error creating transactional connection factoryat org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)at java.security.AccessController.doPrivileged(Native Method)at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:411)at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:440)at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:335)at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:291)at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)at org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:648)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:626)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:679)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5989)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5984)at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6236)at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6161)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:221)at org.apache.hadoop.util.RunJar.main(RunJar.java:136)NestedThrowablesStackTrace:java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:282)at org.datanucleus.store.AbstractStoreManager.(AbstractStoreManager.java:240)at org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:286)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)at java.security.AccessController.doPrivileged(Native Method)at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:411)at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:440)at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:335)at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:291)at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)at org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:648)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:626)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:679)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5989)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5984)at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6236)at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6161)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:221)at org.apache.hadoop.util.RunJar.main(RunJar.java:136)Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the “BONECP” plugin to create a ConnectionPool gave an error : The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)at org.datanucleus.store.rdbms.ConnectionFactoryImpl.(ConnectionFactoryImpl.java:85)… 54 moreCaused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)… 56 more
把驅(qū)動程序放在/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hive/lib
SERVER[po-master1] E0103: Could not load service classes, Cannot load JDBC driver class ‘com.mysql.jdbc.Driver’org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot load JDBC driver class ‘com.mysql.jdbc.Driver’at org.apache.oozie.service.Services.loadServices(Services.java:309)at org.apache.oozie.service.Services.init(Services.java:213)at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4236)at org.apache.catalina.core.StandardContext.start(StandardContext.java:4739)at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)at org.apache.catalina.core.StandardService.start(StandardService.java:525)at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)at org.apache.catalina.startup.Catalina.start(Catalina.java:595)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)Caused by: org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver class ‘com.mysql.jdbc.Driver’at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:102)at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)at org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1518)at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:531)at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:456)at org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:120)at org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)at org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)at org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)at org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)at org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:644)at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:203)at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:156)at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:227)at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:154)at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:60)at org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:514)at org.apache.oozie.service.JPAService.init(JPAService.java:215)at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)at org.apache.oozie.service.Services.setService(Services.java:372)at org.apache.oozie.service.Services.loadServices(Services.java:305)… 26 moreCaused by: org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class ‘com.mysql.jdbc.Driver’at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1429)at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:91)… 46 moreCaused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driverat org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1420)… 51 more
把mysql-connector-java.jar,mysql-connector-java-5.1.39.jar驅(qū)動程序放在:/var/lib/oozie
[main]: Query for candidates of org.apache.hadoop.hive.metastore.model.MDatabase and subclasses resulted in no possible candidatesRequired table missing : “DBS
“ in Catalog “” Schema “”. DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “datanucleus.autoCreateTables”org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : “DBS
“ in Catalog “” Schema “”. DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “datanucleus.autoCreateTables”at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:485)at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3380)at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:3190)at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2841)at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:122)at org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1605)at org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:954)at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:679)at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:408)at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:947)at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:370)at org.datanucleus.store.query.Query.executeQuery(Query.java:1744)at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)at org.datanucleus.store.query.Query.execute(Query.java:1654)at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221)at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.ensureDbInit(MetaStoreDirectSql.java:185)at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:136)at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:340)at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:291)at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)at org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:57)at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:648)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:626)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:679)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5989)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5984)at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6236)at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6161)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:221)at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
SERVER[po-master1] E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Table 'oozie.VALIDATE_CONN' doesn't exist)
org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Table ‘oozie.VALIDATE_CONN’ doesn’t exist)at org.apache.oozie.service.Services.loadServices(Services.java:309)at org.apache.oozie.service.Services.init(Services.java:213)at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4236)at org.apache.catalina.core.StandardContext.start(StandardContext.java:4739)at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)at org.apache.catalina.core.StandardService.start(StandardService.java:525)at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)at org.apache.catalina.startup.Catalina.start(Catalina.java:595)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
報這個錯誤需要修改hive的配置。搜索autoCreateSchema 改為true
SERVER[po-master1] E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Table ‘oozie.VALIDATE_CONN’ doesn’t exist)org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Table ‘oozie.VALIDATE_CONN’ doesn’t exist)at org.apache.oozie.service.Services.loadServices(Services.java:309)at org.apache.oozie.service.Services.init(Services.java:213)at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4236)at org.apache.catalina.core.StandardContext.start(StandardContext.java:4739)at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780)at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944)at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779)at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505)at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069)at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)at org.apache.catalina.core.StandardService.start(StandardService.java:525)at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)at org.apache.catalina.startup.Catalina.start(Catalina.java:595)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
點擊界面上的Oozie 點擊操作,創(chuàng)建Oozie數(shù)據(jù)庫表
最后導入環(huán)境變量就可以測試了export ZK_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/zookeeper/export HBASE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hbase/export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/export HIVE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hive/export SQOOP_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/sqoop/export OOZIE_HOME=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/oozie/export PATH=$PATH:$HOME/bin:$JAVA_HOME:$JAVA_HOME/bin:/usr/bin/:$HADOOP_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin:$ZK_HOME/bin:$HBASE_HOME/bin
安裝完畢...