栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

sqoop将mysql数据导入hive时因缺少jar包报的一些错误以及解决方法

sqoop将mysql数据导入hive时因缺少jar包报的一些错误以及解决方法

错误一:缺少log4j相关jar包导致的错误

21/10/28 17:32:27 ERROR tool.importTool: import failed: java.io.IOException: Exception thrown in Hive
	at org.apache.sqoop.hive.Hiveimport.executescript(Hiveimport.java:358)
	at org.apache.sqoop.hive.Hiveimport.importTable(Hiveimport.java:241)
	at org.apache.sqoop.tool.importTool.importTable(importTool.java:537)
	at org.apache.sqoop.tool.importTool.run(importTool.java:628)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.sqoop.hive.Hiveimport.executescript(Hiveimport.java:331)
	... 9 more
Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextFactory
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:657)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
	... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.spi.LoggerContextFactory
	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 16 more

错误二:缺少datanucleus jar包导致的错误

21/10/28 17:46:21 WARN metadata.Hive: Failed to register all functions.
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient
	at org.apache.hadoop.hive.metastore.metaStoreUtils.newInstance(metaStoreUtils.java:1654)
	at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.(RetryingmetaStoreClient.java:80)
	at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:130)
	at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:101)
	at org.apache.hadoop.hive.ql.metadata.Hive.createmetaStoreClient(Hive.java:3367)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
	at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:366)
	at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
	at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
	at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.sqoop.hive.Hiveimport.executescript(Hiveimport.java:331)
	at org.apache.sqoop.hive.Hiveimport.importTable(Hiveimport.java:241)
	at org.apache.sqoop.tool.importTool.importTable(importTool.java:537)
	at org.apache.sqoop.tool.importTool.run(importTool.java:628)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.hadoop.hive.metastore.metaStoreUtils.newInstance(metaStoreUtils.java:1652)
	... 31 more
Caused by: java.lang.NoClassDefFoundError: org/datanucleus/NucleusContext
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:340)
	at org.apache.hadoop.hive.metastore.metaStoreUtils.getClass(metaStoreUtils.java:1620)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:64)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.newRawStore(HivemetaStore.java:599)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.getMS(HivemetaStore.java:564)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.createDefaultDB(HivemetaStore.java:626)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.init(HivemetaStore.java:416)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HivemetaStore.newRetryingHMSHandler(HivemetaStore.java:6490)
	at org.apache.hadoop.hive.metastore.HivemetaStoreClient.(HivemetaStoreClient.java:238)
	at org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient.(SessionHivemetaStoreClient.java:70)
	... 36 more
Caused by: java.lang.ClassNotFoundException: org.datanucleus.NucleusContext
	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 49 more
21/10/28 17:46:21 ERROR tool.importTool: import failed: java.io.IOException: Exception thrown in Hive
	at org.apache.sqoop.hive.Hiveimport.executescript(Hiveimport.java:358)
	at org.apache.sqoop.hive.Hiveimport.importTable(Hiveimport.java:241)
	at org.apache.sqoop.tool.importTool.importTable(importTool.java:537)
	at org.apache.sqoop.tool.importTool.run(importTool.java:628)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.sqoop.hive.Hiveimport.executescript(Hiveimport.java:331)
	... 9 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
	at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
	... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
	at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:366)
	at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
	at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
	... 17 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient
	at org.apache.hadoop.hive.metastore.metaStoreUtils.newInstance(metaStoreUtils.java:1654)
	at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.(RetryingmetaStoreClient.java:80)
	at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:130)
	at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:101)
	at org.apache.hadoop.hive.ql.metadata.Hive.createmetaStoreClient(Hive.java:3367)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
	... 22 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.hadoop.hive.metastore.metaStoreUtils.newInstance(metaStoreUtils.java:1652)
	... 31 more
Caused by: java.lang.NoClassDefFoundError: org/datanucleus/NucleusContext
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:340)
	at org.apache.hadoop.hive.metastore.metaStoreUtils.getClass(metaStoreUtils.java:1620)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:64)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.newRawStore(HivemetaStore.java:599)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.getMS(HivemetaStore.java:564)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.createDefaultDB(HivemetaStore.java:626)
	at org.apache.hadoop.hive.metastore.HivemetaStore$HMSHandler.init(HivemetaStore.java:416)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HivemetaStore.newRetryingHMSHandler(HivemetaStore.java:6490)
	at org.apache.hadoop.hive.metastore.HivemetaStoreClient.(HivemetaStoreClient.java:238)
	at org.apache.hadoop.hive.ql.metadata.SessionHivemetaStoreClient.(SessionHivemetaStoreClient.java:70)
	... 36 more
Caused by: java.lang.ClassNotFoundException: org.datanucleus.NucleusContext
	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 49 more

错误三:还因为缺少hive的jar包导致报错,xshell中日志被顶没了,错误代码就不放了

针对错误一、二、三的解决方法:
将/usr/hive/apache-hive-2.1.1-bin/lib目录下的包全部复制到/usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib当中,已经存在的跳过即可,如图所示:

错误四:jdk中/jre/lib/security/java.policy文件配置问题

21/10/28 17:49:45 ERROR bonecp.BoneCP: Unable to start/stop JMX
java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:585)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
	at com.jolbox.bonecp.BoneCP.registerUnregisterJMX(BoneCP.java:528)
	at com.jolbox.bonecp.BoneCP.(BoneCP.java:500)

解决方法:在该文件末尾加入以下代码:

permission javax.management.MBeanTrustPermission "register";

如图所示

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/389491.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号