栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

Hadoop如何加载第三方包(hadoop打包jar包)

Hadoop如何加载第三方包(hadoop打包jar包)

报错背景:

本地idea编写Hadoop程序,对nginx日志进行分析,引入nl.bitwalker.useragentutils.UserAgent包解析nginx的浏览器信息获取浏览器系统名称和浏览器名称,

本地测试可以完全ok,然后打包到Hadoop集群运行,报错找不到nl.bitwalker.useragentutils.UserAgent包
java.lang.Exception: java.lang.NoClassDefFoundError: nl/bitwalker/useragentutils/UserAgent
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.NoClassDefFoundError: nl/bitwalker/useragentutils/UserAgent
	at cn.awz.log.LogMapper.getOperatingSystem(LogMapper.java:84)
	at cn.awz.log.LogMapper.getBrowser(LogMapper.java:93)
	at cn.awz.log.LogMapper.parse(LogMapper.java:49)
	at cn.awz.log.LogMapper.map(LogMapper.java:25)
	at cn.awz.log.LogMapper.map(LogMapper.java:16)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: nl.bitwalker.useragentutils.UserAgent
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more
原因Hadoop默认不支持第三方包,

查了好多文档和博客依然没解决,无意间看见一篇博客(后来找不到啦)叫我在环境变量里面配置
HADOOP_CLASSPATH完美解决

解决在maven仓库中找到该包,

上传到主节点所在的Linux中,配置环境变量,
export HADOOP_CLASSPATH=~/UserAgentUtils-1.2.4.jar
激活环境变量
source /etc/profile

运行架包

hadoop jar code-1.0-SNAPSHOT.jar cn/awz/log/LogDriver /flume/upload/20220317/17 /nginxLogs


成功

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/771727.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号