栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

搭建map-reduce开发环境

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

搭建map-reduce开发环境

## Launch IntelliJ

## create Java project - for example "wordcount"
select ‘Create New Project’
    -> "Java project"
    -> select 1.8 JDK in "Project SDK"
    -> Next -> Next
    -> enter "wordcount" in "Project name"
    -> enter "~/work/wordcount" in "Project location"
    -> Finish

## create Java Class
select File -> New -> "Java Class" -> enter "WordCount" -> Create
then a new file generated under src/

## add the Hadoop module dependencies
select File -> "Project Structure" -> Modules -> Dependencies
    -> "+" -> "JARS or directories"
Browse to the Hadoop installation, select jar files under following directorys:
    share/hadoop/common
    share/hadoop/common/lib
    share/hadoop/mapreduce
    share/hadoop/yarn

## configure
select Run -> "Edit Configuration" -> Application -> "+"
    -> enter "WordCount" in "Main class"
    -> enter "input output" as input / output path in "Program arguments"
    -> Apply -> OK
put the input text under "~/work/mapreduce/input"

## coding and run / debug
Note:
    in IntelliJ, WordCount runs with Hadoop standalone mode
    don't forget to delete "~/work/mapreduce/output" before running

## create a jar file
$ cd out/production/mapreduce
$ jar cvf ../../../wordcount.jar *.class

## run with Hadoop pseudo-distributed mode
$ hdfs dfs -rm -f -r output
$ hadoop jar wordcount.jar WordCount input output
$ hdfs dfs -cat output/*

reference: https://tokluo.wordpress.com/2016/01/31/using-intellij-to-write-your-application/

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/946635.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号