本地idea编写Hadoop程序,对nginx日志进行分析,引入nl.bitwalker.useragentutils.UserAgent包解析nginx的浏览器信息获取浏览器系统名称和浏览器名称,
本地测试可以完全ok,然后打包到Hadoop集群运行,报错找不到nl.bitwalker.useragentutils.UserAgent包java.lang.Exception: java.lang.NoClassDefFoundError: nl/bitwalker/useragentutils/UserAgent at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) Caused by: java.lang.NoClassDefFoundError: nl/bitwalker/useragentutils/UserAgent at cn.awz.log.LogMapper.getOperatingSystem(LogMapper.java:84) at cn.awz.log.LogMapper.getBrowser(LogMapper.java:93) at cn.awz.log.LogMapper.parse(LogMapper.java:49) at cn.awz.log.LogMapper.map(LogMapper.java:25) at cn.awz.log.LogMapper.map(LogMapper.java:16) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: nl.bitwalker.useragentutils.UserAgent at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 14 more原因Hadoop默认不支持第三方包,
查了好多文档和博客依然没解决,无意间看见一篇博客(后来找不到啦)叫我在环境变量里面配置
HADOOP_CLASSPATH完美解决
export HADOOP_CLASSPATH=~/UserAgentUtils-1.2.4.jar激活环境变量
source /etc/profile
运行架包
hadoop jar code-1.0-SNAPSHOT.jar cn/awz/log/LogDriver /flume/upload/20220317/17 /nginxLogs
成功



