測試環(huán)境搭建

安裝

https://hive.apache.org/downloads.html
因?yàn)樾枰褂肞hoenix Storage Handler,所以選擇支持hadoop2.x.y的 1.2+版本,hive1.2.2

配置環(huán)境變量

~/.bash_profile 配置hadoop&&hive變量

export HADOOP_HOME="/Users/titengjiang/Documents/DeveloperTool/hadoop/hadoop-2.7.5"
export PATH="$HADOOP_HOME/bin:$PATH"
export HIVE_HOME="/Users/titengjiang/Documents/DeveloperTool/hive/apache-hive-1.2.2-bin"
export PATH="$HIVE_HOME/bin:$PATH"

配置hive

mv hive-default.xml.template hive-site.xml

hive-site.xml增加如下配置

  <property>
    <name>system:java.io.tmpdir</name>
    <value>/Users/titengjiang/Documents/DeveloperTool/hive/local/tmp</value>
  </property>
  <property>
    <name>system:user.name</name>
    <value>${user.name}</value>
  </property>

解決如下異常

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

創(chuàng)建warehouse

  $ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
  $ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
  $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse

初始化mysql metastore

create schema metastore_db default character set utf8;
CREATE USER hive@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON metastore_db.* TO hive@'%' identified by 'hive' ;
GRANT ALL PRIVILEGES ON metastore_db.* TO hive@'localhost identified by 'hive';
 flush privileges;-

配置metastore,使用推薦的mysql替換derby

  <!-- 設(shè)置元數(shù)據(jù)存放的數(shù)據(jù)庫地址、名字 -->
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore_db?useSSL=false</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/dbName?ssl=true for postgres database.
    </description>
  </property>
<!--Driver class名字-->
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
<!--連接使用的用戶名-->
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>
  <!--連接使用的密碼-->
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value>
    <description>password to use against metastore database</description>
  </property>

拷貝mysql-connection-version.jar 到hive lib下并初始化metastore_db

注意mysql-connection的版本選擇5.1.34最好，較低版本可能會遇到錯(cuò)誤

$HIVE_HOME/bin/schematool -dbType mysql -initSchema

如果出現(xiàn)上面的錯(cuò)誤,檢查下mysql 用戶權(quán)限，連接之類的配置

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>  

配置日志

hive-log4j.properties

hive.log.dir=/Users/username/Documents/DeveloperTool/hive/local/logs/${user.name} //日志路徑改成指定路徑

啟動

$HIVE_HOME/bin/hiveserver2
hive --service metastore //啟動metatore 提供hcatalog streaming寫入端口 9083默認(rèn)

常見錯(cuò)誤

For direct MetaStore DB connections, we don’t support retries at the client level

create table years (year string, event string) row format delimited fields terminated by '\t';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)

修改mysql-connection版本