大数据平台架构实战（二）IntelliJ IDEA搭建hadoop

工具类开发，随便找，IntelliJ IDEA搭建Hadoop开发环境（下）_Captain.Y.的博客-CSDN博客

由于我们的开发环境和hadoop运行环境都在同一台机器，直接用maven工具就可以了

如果是其他情形 ，尽量保证jar包的依赖完整性。

pom文件

 
        
            org.apache.hadoop
            hadoop-client
            2.10.2
        


        
        
            junit
            junit
            4.12

        
        
            commons-httpclient
            commons-httpclient
            3.1
        
    

    
        1.8
        1.8
    

打包完成后，就可以在hadoop平台直接运行了

 bin/hadoop jar /Users/xxxxxx/Project/hadoop_base/target/hadoop_base-1.0-SNAPSHOT.jar org.xxxx.algorithm.tools.WordCount /input /output/wc

22/06/01 15:26:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/01 15:26:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
22/06/01 15:26:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
22/06/01 15:26:33 INFO input.FileInputFormat: Total input files to process : 29
22/06/01 15:26:34 INFO mapreduce.JobSubmitter: number of splits:29
22/06/01 15:26:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1654067025193_0002
22/06/01 15:26:34 INFO conf.Configuration: resource-types.xml not found
22/06/01 15:26:34 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
22/06/01 15:26:34 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
22/06/01 15:26:34 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
22/06/01 15:26:34 INFO impl.YarnClientImpl: Submitted application application_1654067025193_0002
22/06/01 15:26:34 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1654067025193_0002/
22/06/01 15:26:34 INFO mapreduce.Job: Running job: job_1654067025193_0002
22/06/01 15:26:36 INFO mapreduce.Job: Job job_1654067025193_0002 running in uber mode : false
22/06/01 15:26:36 INFO mapreduce.Job:  map 0% reduce 0%
22/06/01 15:26:36 INFO mapreduce.Job: Job job_1654067025193_0002 failed with state FAILED due to: Application application_1654067025193_0002 failed 2 times due to AM Container for appattempt_1654067025193_0002_000002 exited with  exitCode: 127
Failing this attempt.Diagnostics: [2022-06-01 15:26:36.146]Exception from container-launch.
Container id: container_1654067025193_0002_02_000001
Exit code: 127

[2022-06-01 15:26:36.147]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory


[2022-06-01 15:26:36.147]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: /bin/java: No such file or directory


For more detailed output, check the application tracking page: http://localhost:8088/cluster/app/application_1654067025193_0002 Then click on links to logs of each attempt.
. Failing the application.
22/06/01 15:26:36 INFO mapreduce.Job: Counters: 0

 解决方法：

修改 etc/hadoop/hadoop-env.sh 

export JAVA_HOME=/path/to/java/home