栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

2021-11-16Hadoop集群问题记录

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

2021-11-16Hadoop集群问题记录

项目场景:运行WordCount单词记数案例 问题描述:在运行wordcount单词计数程序时报错:错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

2021-11-16 10:06:25,707 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.150.129:8032
2021-11-16 10:06:26,291 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1637026326069_0009
2021-11-16 10:06:26,536 INFO input.FileInputFormat: Total input files to process : 1
2021-11-16 10:06:26,794 INFO mapreduce.JobSubmitter: number of splits:1
2021-11-16 10:06:26,973 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1637026326069_0009
2021-11-16 10:06:26,974 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-11-16 10:06:27,165 INFO conf.Configuration: resource-types.xml not found
2021-11-16 10:06:27,166 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
2021-11-16 10:06:27,231 INFO impl.YarnClientImpl: Submitted application application_1637026326069_0009
2021-11-16 10:06:27,272 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1637026326069_0009/
2021-11-16 10:06:27,273 INFO mapreduce.Job: Running job: job_1637026326069_0009
2021-11-16 10:06:30,308 INFO mapreduce.Job: Job job_1637026326069_0009 running in uber mode : false
2021-11-16 10:06:30,308 INFO mapreduce.Job: map 0% reduce 0%
2021-11-16 10:06:30,320 INFO mapreduce.Job: Job job_1637026326069_0009 failed with state FAILED due to: Application application_1637026326069_0009 failed 2 times due to AM Container for appattempt_1637026326069_0009_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-11-16 10:06:29.931]Exception from container-launch.
Container id: container_1637026326069_0009_02_000001
Exit code: 1

[2021-11-16 10:06:29.934]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

[2021-11-16 10:06:29.937]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1637026326069_0009 Then click on links to logs of each attempt.
. Failing the application.
2021-11-16 10:06:30,337 INFO mapreduce.Job: Counters: 0
[root@master mapreduce]# hadoop jar hadoop-mapreduce-examples-3.1.4.jar wordcount /a.txt /output11
2021-11-16 10:07:17,595 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.150.129:8032
2021-11-16 10:07:18,235 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1637026326069_0010
2021-11-16 10:07:18,456 INFO input.FileInputFormat: Total input files to process : 1
2021-11-16 10:07:18,611 INFO mapreduce.JobSubmitter: number of splits:1
2021-11-16 10:07:18,867 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1637026326069_0010
2021-11-16 10:07:18,868 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-11-16 10:07:19,012 INFO conf.Configuration: resource-types.xml not found
2021-11-16 10:07:19,012 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
2021-11-16 10:07:19,076 INFO impl.YarnClientImpl: Submitted application application_1637026326069_0010
2021-11-16 10:07:19,120 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1637026326069_0010/
2021-11-16 10:07:19,120 INFO mapreduce.Job: Running job: job_1637026326069_0010
2021-11-16 10:07:20,145 INFO mapreduce.Job: Job job_1637026326069_0010 running in uber mode : false
2021-11-16 10:07:20,146 INFO mapreduce.Job: map 0% reduce 0%
2021-11-16 10:07:20,164 INFO mapreduce.Job: Job job_1637026326069_0010 failed with state FAILED due to: Application application_1637026326069_0010 failed 2 times due to AM Container for appattempt_1637026326069_0010_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-11-16 10:07:19.816]Exception from container-launch.
Container id: container_1637026326069_0010_02_000001
Exit code: 1

原因分析:yarn-site.xml的yarn.application.classpath 的问题

1.可能是yarn-site.xml这个文件的配置问题,这个文件中没有配置 yarn.application.classpath 2.或者这个文件配置有误


解决方案:
cd /usr/local/hadoop-3.1.4/etc/hadoop
vim yarn-site.xml

如果没有,就添加 yarn.application.classpath 这个配置文件


yarn.application.classpath 
 
/usr/local/hadoop-3.1.4/etc/hadoop:/usr/local/hadoop-3.1.4/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/common/*:/usr/local/hadoop-3.1.4/share/hadoop/hdfs:/usr/local/hadoop-3.1.4/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.4/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.4/share/hadoop/yarn:/usr/local/hadoop-3.1.4/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/yarn/*
 

检查这个配置文件

如果有 就检查一下是否有误,比如hadoop-3.1.4中间有个“-”,还有就是需要注意,这一串代码是连起来的,并不是分开的
注意:集群的每个节点都需要修改
解决完成后就重启yarn或者hadoop集群
重启后再次运行相关MapReduce项目 问题已解决!

记录学习遇到的问题,安装Hadoop的方式不一样,解决方法可能也会因此改变,此方法仅供参考

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/531818.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号