異常說明
SparkSession初始化一個(gè)lazy的HiveClient用于Hive MetaStore連接，一旦使用其內(nèi)部就會(huì)實(shí)例化一個(gè)org.apache.spark.sql.hive.HiveSessionState對(duì)象，執(zhí)行SessionState.start方法時(shí)會(huì)根據(jù)HiveConf里的配置執(zhí)行一系列操作，包括創(chuàng)建一些臨時(shí)的工作目錄等。這些目錄如果沒有正確的權(quán)限會(huì)導(dǎo)致這個(gè)過程失敗，致使連接不上Server.
客戶端異常
hzyaoqin@hadoop980:~/spark-packages/spark-2.1.0-bin-ne-1.3.0-SNAPSHOT$ bin/beeline -u 'jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse'
Connecting to jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse
17/09/06 17:17:14 INFO jdbc.Utils: Supplied authorities: hadoop980.hz.163.org:10001
17/09/06 17:17:14 INFO jdbc.Utils: Resolved authority: hadoop980.hz.163.org:10001
17/09/06 17:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/09/06 17:17:15 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse
Can't overwrite cause with java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2://hadoop980.hz.163.org:10001/de (closed)>
其中異常主體為，
Can't overwrite cause with java.lang.IllegalArgumentException:
Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
， 實(shí)際意為初始化HiveSessionState時(shí)發(fā)生了不可名狀的事情；進(jìn)一步去看服務(wù)端日志；
注：HiveSession級(jí)別的日志沒有做到客戶端這邊，只有Operation級(jí)別的日志可以看哦
服務(wù)端異常
Error opening session:
org.apache.spark.SparkException: Failed Init SparkSession for user[hzyaoqin]java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:135)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.getOrCreateSparkSession(SparkHiveSessionImpl.scala:109)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.open(SparkHiveSessionImpl.scala:186)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerSessionManager.openSession(ThriftServerSessionManager.scala:225)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerCLIService.openSessionWithImpersonation(ThriftServerCLIService.scala:91)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.getSessionHandle(ThriftClientCLIService.scala:252)
at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.OpenSession(ThriftClientCLIService.scala:266)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1120)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:111)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:888)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:124)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:118)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:118)
... 15 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1117)
... 31 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:183)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:90)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:102)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:101)
at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
... 36 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:180)
... 44 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:367)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:271)
at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:65)
... 49 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1722)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3863)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
非常明顯,hzyaoqin用戶沒有用戶hdfs所屬目錄的執(zhí)行權(quán)限；
Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
這個(gè)目錄為hive.exec.strachdir（缺省為/tmp/hive/public）下創(chuàng)建username/sessionId的臨時(shí)目錄，我們多租戶下執(zhí)行都是用hzyaoqin這個(gè)執(zhí)行賬號(hào)去真正執(zhí)行的，這個(gè)目錄如果是hdfs用戶的，必然不是有本次session創(chuàng)建的，應(yīng)該是以前的遺留目錄
解決方案
- 1、刪除/tmp/hive/public/hzyaoqin（如果這目錄沒啥用），重新連接
- 2、重新配置hive.exec.strachdir到另一個(gè)有權(quán)限的目錄，可以在連接串中動(dòng)態(tài)的session級(jí)別指定或classpath中的hive-site.xml進(jìn)行server級(jí)別的全局設(shè)置，前者重新連接，后者重啟server。
