99re在线精品只有,伊人久久黄色电影,在线看最新不卡av

一.基本功能:

1.啟動(dòng)hive時(shí)報(bào)錯(cuò)

  java.lang.ExceptionInInitializerError
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:190)
      at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.init(JDBCStatsPublisher.java:265)
      at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:412)
  Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed
      at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:388)
      at java.net.URLClassLoader.defineClass(URLClassLoader.java:417)

解決方案:

  將mysql-connector-java-5.1.6-bin.jar包導(dǎo)入到$HIVE_HOME/lib目錄下

2.啟動(dòng)hive時(shí)報(bào)錯(cuò):

  [ERROR] Terminal initialization failed; falling back to unsupported
  java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
      at jline.TerminalFactory.create(TerminalFactory.java:101)
      at jline.TerminalFactory.get(TerminalFactory.java:158)
  Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
      at jline.console.ConsoleReader.<init>(ConsoleReader.java:230)
      at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)

解決方案:

  將當(dāng)前hive版本的$HIVE_HOME/lib目錄下的jline-2.12.jar包拷貝到$HADOOP_HOME/share/hadoop/yarn/lib目錄下, 并將舊版本的Hive的Jline包從$HADOOP_HOME/etc/hadoop/yarn/lib目錄下刪除

3.啟動(dòng)hive時(shí)報(bào)錯(cuò)

  Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
  Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:172)
  Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at java.net.URI.checkPath(URI.java:1804)
        at java.net.URI.<init>(URI.java:752)
        at org.apache.hadoop.fs.Path.initialize(Path.java:203)
        ... 11 more

解決方案:

  1.查看hive-site.xml配置，會(huì)看到配置值含有"system:java.io.tmpdir"的配置項(xiàng)
  2.新建文件夾${HIVE_HOME}/hive/logs
  3.將含有"system:java.io.tmpdir"的配置項(xiàng)的值修改為${HIVE_HOME}/hive/logs
  即: 新添屬性為
  <property>
      <name>hive.exec.local.scratchdir</name>
      <value>${HIVE_HOME}/logs/HiveJobsLog</value>
      <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
      <name>hive.downloaded.resources.dir</name>
      <value>${HIVE_HOME}/logs/ResourcesLog</value>
      <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
      <name>hive.querylog.location</name>
      <value>${HIVE_HOME}/logs/HiveRunLog</value>
      <description>Location of Hive run time structured log file</description>
  </property>
  <property>
      <name>hive.server2.logging.operation.log.location</name>
      <value>${HIVE_HOME}/logs/OpertitionLog</value>
      <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

4.啟動(dòng)hive時(shí)報(bào)錯(cuò)

  Caused by: java.sql.SQLException: Access denied for user 'root'@'master' (using password: YES)
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812)
      at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3269)

解決方案:

  mysql密碼不正確, 查看hive-site.xml配置與mysql的密碼是否一致

5.操作表數(shù)據(jù)時(shí)(向表中導(dǎo)入數(shù)據(jù)), 報(bào)錯(cuò):

  FAILED: RuntimeException org.apache.hadoop.security.AccessControlException: Permission denied: user=services02, access=EXECUTE, inode="/tmp":services01:supergroup:drwx------

解決方案:

  user=services02與inode="/tmp":services01:supergroup不同時(shí),說(shuō)明hive登錄的主機(jī)與HDFS的active狀態(tài)的主機(jī)不一樣
  應(yīng)把user=services02的主機(jī)變?yōu)镠DFS的active狀態(tài)的主機(jī).

二.擴(kuò)展Parquet功能:

1.創(chuàng)建存儲(chǔ)格式為parquet時(shí),建表語(yǔ)句為: (Hive版本為0.12)

  create table parquet_test(x int, y string) 
  row format serde 'parquet.hive.serde.ParquetHiveSerDe'    
  stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat'                    
  outputformat 'parquet.hive.DeprecatedParquetOutputFormat';

報(bào)錯(cuò):

  FAILED: SemanticException [Error 10055]: Output Format must implement 
  HiveOutputFormat, otherwise it should be either IgnoreKeyTextOutputFormat or 
  SequenceFileOutputFormat

解決方案:

  因?yàn)閜arquet.hive.DeprecatedParquetOutputFormat類(lèi)并沒(méi)有在Hive的CLASSPATH中配置
  單獨(dú)下載parquet-hive-1.2.5.jar包(此類(lèi)屬于$IMPALA_HOME/lib目錄下), 在$HIVE_HOME/lib目錄下建立個(gè)軟鏈就可以了

  cd $HIVE_HOME/lib
  ln -s $/home/hadoop/soft/gz.zip/parquet-hive-1.2.5.jar

2.繼續(xù)提交創(chuàng)表語(yǔ)句: (Hive版本為0.12)

  create table parquet_test(x int, y string) 
  row format serde 'parquet.hive.serde.ParquetHiveSerDe'    
  stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat'                    
  outputformat 'parquet.hive.DeprecatedParquetOutputFormat';

報(bào)錯(cuò):

  Exception in thread "main" java.lang.NoClassDefFoundError: parquet/hadoop/api/WriteSupport
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:247)

解決方案:

  通過(guò)yum下載Parquet
  sodu yum -y install parquet

下載parquet的jar包在/usr/lib/parquet目錄下, 將/usr/lib/parquet目錄下的所有jar(除javadoc.jar和sources.jar外)拷貝到$HIVE_HOME/lib目錄下.
若yum無(wú)法下載parquet資源包, 則這是需要配置yum源, 請(qǐng)自行百度尋找資料

3.繼續(xù)提交創(chuàng)表語(yǔ)句:(Hive版本為0.12)

  create table parquet_test(x int, y string) 
  row format serde 'parquet.hive.serde.ParquetHiveSerDe'    
  stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat'                    
  outputformat 'parquet.hive.DeprecatedParquetOutputFormat';

報(bào)錯(cuò):

  FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

解決方案:

  先啟動(dòng)metastore服務(wù):
  hive --service metastore

4.將存儲(chǔ)格式為textFile的文件數(shù)據(jù)insert到parquet時(shí), 報(bào)錯(cuò):

  Error: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426))
  Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        ... 9 more
  Caused by: java.lang.RuntimeException: Map operator initialization failed
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:134)
        ... 22 more
  Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:386)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)
        ... 22 more
  Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:323)
        ... 34 more
  FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
        MapReduce Jobs Launched: 
        Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
        Total MapReduce CPU Time Spent: 0 msec

解決方案:

  在hive-env.sh中添加: 
  JAVA_HOME=/home/hadoop/soft/jdk1.7.0_67
  HADOOP_HOME=/home/hadoop/soft/hadoop-2.4.1
  HIVE_HOME=/home/hadoop/soft/hive-0.12.0
  export HIVE_CONF_DIR=$HIVE_HOME/conf
  export HIVE_AUX_JARS_PATH=$HIVE_HOME/lib
  export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$HADOOP_HOME/lib:$HIVE_HOME/lib

5.將textFile表的數(shù)據(jù)插入到parquet表中,報(bào)錯(cuò):

  Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
            while processing row {"time":1471928602,"uid":687994,"billid":1004,"archiveid":null,"year":"2016","mouth":"2016-08","day":"2016-08-23","hour":"2016-08-23-04"}
  Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 
  Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.ArrayWritable

解決方案:

配置hive-env.xml文件,添加:

      JAVA_HOME=/home/hadoop/soft/jdk1.7.0_67
      HADOOP_HOME=/home/hadoop/soft/hadoop-2.4.1
      HIVE_HOME=/home/hadoop/soft/hive-0.12.0
      export HIVE_CONF_DIR=$HIVE_HOME/conf
      export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$HADOOP_HOME/lib:$HIVE_HOME/lib

通過(guò)yum下載Parquet

      sodu yum -y install parquet

下載parquet的jar包在/usr/lib/parquet目錄下, 將/usr/lib/parquet目錄下的所有jar(除javadoc.jar和sources.jar外)拷貝到$HIVE_HOME/lib目錄下.
文件壓縮格式問(wèn)題:
在insert插入數(shù)據(jù)之前, 要先設(shè)置壓縮格式, 要三種格式可選, 通常選用SNAPPY:

    CompressionCodecName.UNCOMPRESSED
    CompressionCodecName.SNAPPY
    CompressionCodecName.GZIP

5.在Hive命令行中執(zhí)行set parquet.compression = SNAPPY語(yǔ)句; 然后再進(jìn)行insert數(shù)據(jù)插入操作

6.創(chuàng)建存儲(chǔ)格式為parquet的表時(shí),報(bào)錯(cuò)

  FAILED: SemanticException [Error 10055]: Output Format must implement HiveOutputFormat, otherwise it should be either IgnoreKeyTextOutputFormat or SequenceFileOutputFormat

解決方案:

  查看hive版本, 如果hive是0.13版本以下的, 則創(chuàng)表時(shí)用: stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat' outputformat 'parquet.hive.DeprecatedParquetOutputFormat';
  若hive是0.13版本以上, 則創(chuàng)表時(shí)用: stored as parquet;

三.Hive1.x升級(jí)到2.x異常匯總

1.啟動(dòng)hive時(shí)報(bào)錯(cuò)

  Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
            at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
            at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
  Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
            at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
  Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
            at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
            at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
  Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  Caused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore schema version 1.2.0 Metastore is not upgraded or corrupt)
            at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7768)
            at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)

解決方案:

  (1)刪除HDFS上的hive數(shù)據(jù)與hive數(shù)據(jù)庫(kù)
        hadoop fs -rm -r -f /tmp/hive
        hadoop fs -rm -r -f /user/hive
  (2)刪除MySQL上的hive的元數(shù)據(jù)信息
        mysql -uroot -p 
        drop database hive

隨之而來(lái)了一個(gè)問(wèn)題:

  Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
            at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
            at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
  Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
            at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
            at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
  Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
            at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
            at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
  Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  Caused by: MetaException(message:Version information not found in metastore. )
            at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
            at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)

解決方案:

  初始化hive, 將mysql作為hive的元數(shù)據(jù)庫(kù)
  schematool -dbType mysql -initSchema

2.使用HPL/SQL時(shí)報(bào)錯(cuò):

  java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.ConnectException: Connection refused

解決方案(若方案一仍然報(bào)錯(cuò), 則執(zhí)行方案二):

方案一:
啟動(dòng)hiveServer2服務(wù),

  cd $HIVE_HOME/bin
  ./hiveserver2

方案二:

編輯hplsql-site.xml, 修改以下配置:

  <property>
      <name>hplsql.conn.default</name>
      <value>hive2conn</value>
      <description>The default connection profile</description>
  </property>
  <property>
      <name>hplsql.conn.hive2conn</name>
      <value>org.apache.hive.jdbc.HiveDriver;jdbc:hive2://m1:10000</value>
      <description>HiveServer2 JDBC connection</description>
  </property>

然后啟動(dòng)hiveServer2

    cd $HIVE_HOME/bin
    ./hiveserver2

使用beeline測(cè)試連接

    cd $HIVE_HOME/bin
    ./beeline
    !connect jdbc:hive2://m1:10000

3.在hive2.0中使用hplsql, 對(duì)存儲(chǔ)過(guò)程文件進(jìn)行執(zhí)行時(shí), 報(bào)錯(cuò):

  java.sql.SQLException: 
  Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: centos is not allowed to impersonate hive
            at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:209)
            at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
  Caused by: org.apache.hive.service.cli.HiveSQLException: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: centos is not allowed to impersonate hive
            at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:266)
  Caused by: org.apache.hive.service.cli.HiveSQLException: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: centos is not allowed to impersonate hive
            at org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:336)
            at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:279)
  Caused by: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: centos is not allowed to impersonate hive
            at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
            at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
  Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: centos is not allowed to impersonate hive
            at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
            at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:526)
  Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException:User: centos is not allowed to impersonate hive
            at org.apache.hadoop.ipc.Client.call(Client.java:1470)
            at org.apache.hadoop.ipc.Client.call(Client.java:1401)

解決方案:

暫停HDFS服務(wù)
stop-dfs.sh
修改core-site.xml文件
在$HADOOP_HOME/etc/hadoop/core-site.xml文件中增加以下配置, 注意:Hadoop.proxyuser.centos.hosts配置項(xiàng)名稱(chēng)中centos部分為報(bào)錯(cuò)User:*中的用戶名部分

  <property>
      <name>hadoop.proxyuser.centos.groups</name>
      <value>*</value>
  </property>
  <property>
      <name>hadoop.proxyuser.centos.hosts</name>
      <value>*</value>
  </property>

將core-site.xml文件分發(fā)到其他主機(jī)
啟動(dòng)HDFS服務(wù)
start-dfs.sh

4.在Hive2.0中使用HPL/SQL, 對(duì)存儲(chǔ)過(guò)程文件進(jìn)行執(zhí)行時(shí), 報(bào)錯(cuò):

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Permission denied: user=anonymous, access=EXECUTE, inode="/tmp":centos:supergroup:drwx------
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:208)

解決方案:

1. 修改有問(wèn)題的目錄權(quán)限
    hadoop fs -chmod 755 /tmp

2.在hive-site.xml文件中修改以下配置項(xiàng)
<property>
    <name>hive.scratch.dir.permission</name>
    <value>755</value>
</property>

5.使用HPL/SQL存儲(chǔ)過(guò)程時(shí)報(bào)錯(cuò)

在HPL/SQL2.2.1版本中, 使用from語(yǔ)句時(shí), 總是從配置文件里找hplsql.dual.table配置項(xiàng)的值, 總是報(bào)錯(cuò)找不到dual表, 或找不到select語(yǔ)句中指定cloumn

解決方案:

HPL/SQL官網(wǎng)

  http://www.hplsql.org/doc

注意: 一定得是0.3.17或以上版本的HPL/SQL
下載一個(gè)HPL/SQL 0.3.17的tan.gz文件, 解壓后將hplsql-0.3.17.jar包放入$HIVE_HOME包下, 并改名為hive-hplsql-*.jar格式的包,如:hive-hplsql-0.3.17.jar

五.Hive On Spark作為引擎:

  官方參考配置項(xiàng)
      https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark
  遇坑解決方案指南:
      http://www.cnblogs.com/breg/p/5552342.html
  搭建教程及部分遇坑解決指南
      http://www.cnblogs.com/linbingdong/p/5806329.html

1.啟動(dòng)hive時(shí)報(bào)錯(cuò)

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
        at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:477)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
```

#####解決方案:
```
    啟動(dòng)metastore服務(wù):
    hive --service metastore
```


---
</br>
</br>
#####2.調(diào)用spark引擎計(jì)算時(shí), 報(bào)錯(cuò):
```
  Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
```

######解決方案:
```
  錯(cuò)誤日志中出現(xiàn)Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask 該錯(cuò)誤, 大部分是因?yàn)閟park的安裝包是包含Hive引用包的, 出現(xiàn)此類(lèi)問(wèn)題應(yīng)自己手動(dòng)編譯一個(gè)spark包
```



---
</br>
</br>
#####3.調(diào)用spark引擎計(jì)算時(shí), 報(bào)錯(cuò):
```
2016-12-19T20:19:15,491 ERROR [main] client.SparkClientImpl: Error while waiting for client to connect.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancel client 'dcee57ba-ea77-4e92-bd43-640e8385e2e7'. 
Error: Child process exited before connecting back with error log 
        Warning: Ignoring non-spark config property: hive.spark.client.server.connect.timeout=200000
        Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
        Warning: Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
        Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
        Warning: Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
        16/12/19 20:19:15 INFO client.RemoteDriver: Connecting to: m1:48286
Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
        at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:45)
        at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:134)
Caused by: java.lang.RuntimeException: Cancel client 'dcee57ba-ea77-4e92-bd43-640e8385e2e7'. 
Error: Child process exited before connecting back with error log Warning: Ignoring non-spark config property: hive.spark.client.server.connect.timeout=200000
        Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
        Warning: Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
        Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
        Warning: Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
        16/12/19 20:19:15 INFO client.RemoteDriver: Connecting to: m1:48286
Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS
      at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:45)
      at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:134)
```

######解決方案:
```
參考資料:
    http://www.cnblogs.com/breg/p/5552342.html
```

日志中出現(xiàn)````Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS```該錯(cuò)誤, 大部分是因?yàn)?*spark**的安裝包是包含**Hive**引用包的, 出現(xiàn)此類(lèi)問(wèn)題應(yīng)自己手動(dòng)編譯一個(gè)**spark**包




---
</br>
</br>
#####4.使用編譯好的spark安裝包, 安裝好后, 啟動(dòng)master時(shí)報(bào)錯(cuò):
```
  Spark Command: /home/centos/soft/jdk1.7.0_67/bin/java -cp /home/centos/soft/spark/conf/:/home/centos/soft/spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/home/centos/soft/hadoop/etc/hadoop/:/home/centos/soft/hadoop/etc/hadoop/:/home/centos/soft/hadoop/lib/spark-assembly-1.6.0-hadoop2.6.0.jar -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip m1 --port 7077 --webui-port 8080
  ========================================
  Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
        at java.lang.Class.getMethod0(Class.java:2813)
  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
```

######解決方案:
```
  出現(xiàn)此類(lèi)問(wèn)題, 均是spark源碼編譯時(shí)出錯(cuò), 推薦使用Maven編譯
```



---
</br>
</br>
#####5.使用編譯好的spark安裝包, 安裝好后, 啟動(dòng)master時(shí)報(bào)錯(cuò):
```
  Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
          at java.lang.Class.getDeclaredMethods0(Native Method)
          at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
          at java.lang.Class.getMethod0(Class.java:2813)
  Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
          at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
```


######解決方案:
```
  出現(xiàn)此類(lèi)問(wèn)題, 均是spark源碼編譯時(shí)出錯(cuò), 推薦使用Maven編譯
```



---
</br>
</br>
#####6.Hive On Spark使用hplsql存儲(chǔ)過(guò)程時(shí)報(bào)錯(cuò),
**Beeline**中的報(bào)錯(cuò)信息:
```
  Unhandled exception in HPL/SQL
  java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://m1:10000: null
            at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:209)
            at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
            at java.sql.DriverManager.getConnection(DriverManager.java:571)
  Caused by: org.apache.thrift.transport.TTransportException
            at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
            at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
```


**Hive-log.log**的報(bào)錯(cuò)信息:
```
ERROR [HiveServer2-Handler-Pool: Thread-43] server.TThreadPoolServer: 
Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
```

######解決方案:
將```hive.server2.authentication```配置項(xiàng)改為```NONE```
```
<property>
    <name>hive.server2.authentication</name>
    <value>NONE</value>
</property>
```



---
</br>
</br>
#####7.使用spark-submit --master yarn提交任務(wù)時(shí)沒(méi)問(wèn)題, 以HPL/SQL用存儲(chǔ)過(guò)程提交任務(wù)執(zhí)行Spark任務(wù)時(shí)卻報(bào)錯(cuò):
```
  16/12/26 16:45:01 WARN cluster.ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered
  16/12/26 16:45:16 WARN cluster.ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered
```

######解決方案:

出現(xiàn)這種問(wèn)題的原因有很多種, 我所遇到的問(wèn)題的解決方案是: 將```/etc/hosts```下的```127.0.0.1```注釋, 然后設(shè)置一個(gè)新值
```
  sudo vi /etc/hosts
      #127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
      127.0.0.1   localhost
```






---
</br>
</br>
#####8.報(bào)錯(cuò)信息如下:
```
java.lang.StackOverflowError   at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53)
```

######解決方案：
SQL語(yǔ)句的`where`條件過(guò)長(zhǎng)，字符串棧溢出








---
</br>
</br>
#####9.報(bào)錯(cuò)信息如下:
```
Error: Could not find or load main class org.apache.hive.beeline.BeeLine
```

######解決方案：
重新編譯Hive，并帶上參數(shù)`-Phive-thriftserver`









---
</br>
</br>
#####10.報(bào)錯(cuò)信息如下:
```
check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1
```

######解決方案：
用新版`mysql-connector`








---
</br>
</br>
#####11.報(bào)錯(cuò)信息如下:
```
java.lang.NoSuchMethodError: org.apache.parquet.schema.Types$MessageTypeBuilder.addFields([Lorg/apache/parquet/schema/Type;)Lorg/apache/parquet/schema/Types$BaseGroupBuilder;
```

######解決方案：
版本沖突所致，統(tǒng)一`hive`和`spark`中`parquet`組件版本






---
</br>
</br>
#####12.報(bào)錯(cuò)信息如下:
```
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
```

######解決方案：
不要官網(wǎng)編譯好的spark，自己下載Spark源碼重新編譯，并保證編譯的Spark版本滿足Hive源碼中pom.xml文件中對(duì)spark的一個(gè)大版本要求, 若使用Hive on Spark,則在編譯過(guò)程中不帶Phive有關(guān)的任何參數(shù)










---
</br>
</br>
#####13.報(bào)錯(cuò)信息如下:
```
java.lang.NoSuchFieldError: SPARK_RPC_SERVER_ADDRESS  at org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:45)
```

######解決方案：
不要官網(wǎng)編譯好的spark，自己下載Spark源碼重新編譯，并保證編譯的Spark版本滿足Hive源碼中pom.xml文件中對(duì)spark的一個(gè)大版本要求, 若使用Hive on Spark,則在編譯過(guò)程中不帶Phive有關(guān)的任何參數(shù)









---
</br>
</br>
</br>

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九 欧美,1769亚洲,黄色成人av

搭建Hive所遇過(guò)的坑

一.基本功能:

1.啟動(dòng)hive時(shí)報(bào)錯(cuò)

解決方案:

2.啟動(dòng)hive時(shí)報(bào)錯(cuò):

解決方案:

3.啟動(dòng)hive時(shí)報(bào)錯(cuò)

解決方案:

4.啟動(dòng)hive時(shí)報(bào)錯(cuò)

解決方案:

5.操作表數(shù)據(jù)時(shí)(向表中導(dǎo)入數(shù)據(jù)), 報(bào)錯(cuò):

解決方案:

二.擴(kuò)展Parquet功能:

1.創(chuàng)建存儲(chǔ)格式為parquet時(shí),建表語(yǔ)句為: (Hive版本為0.12)

報(bào)錯(cuò):

解決方案:

2.繼續(xù)提交創(chuàng)表語(yǔ)句: (Hive版本為0.12)

報(bào)錯(cuò):

解決方案:

3.繼續(xù)提交創(chuàng)表語(yǔ)句:(Hive版本為0.12)

報(bào)錯(cuò):

解決方案:

4.將存儲(chǔ)格式為textFile的文件數(shù)據(jù)insert到parquet時(shí), 報(bào)錯(cuò):

解決方案:

5.將textFile表的數(shù)據(jù)插入到parquet表中,報(bào)錯(cuò):

解決方案:

6.創(chuàng)建存儲(chǔ)格式為parquet的表時(shí),報(bào)錯(cuò)

解決方案:

三.Hive1.x升級(jí)到2.x異常匯總

1.啟動(dòng)hive時(shí)報(bào)錯(cuò)

解決方案:

隨之而來(lái)了一個(gè)問(wèn)題:

解決方案:

2.使用HPL/SQL時(shí)報(bào)錯(cuò):

解決方案(若方案一仍然報(bào)錯(cuò), 則執(zhí)行方案二):

3.在hive2.0中使用hplsql, 對(duì)存儲(chǔ)過(guò)程文件進(jìn)行執(zhí)行時(shí), 報(bào)錯(cuò):

解決方案:

4.在Hive2.0中使用HPL/SQL, 對(duì)存儲(chǔ)過(guò)程文件進(jìn)行執(zhí)行時(shí), 報(bào)錯(cuò):

解決方案:

5.使用HPL/SQL存儲(chǔ)過(guò)程時(shí)報(bào)錯(cuò)

解決方案:

五.Hive On Spark作為引擎:

1.啟動(dòng)hive時(shí)報(bào)錯(cuò)

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av