栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

在Oozie中配置 map-reduce workflow

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

在Oozie中配置 map-reduce workflow

## prepare wordcount case
$ cd ~/work/oozie-5.2.1
$ tree oozie
oozie
├── apps
│   └── wordcount
│       ├── job.properties
│       ├── job.properties.orig
│       ├── lib
│       │   └── wordcount.jar
│       ├── workflow.xml
│       └── workflow.xml.orig
└── data
    └── wordcount
        └── input
           └── in.txt

$ cat oozie/apps/wordcount/job.properties

nameNode=hdfs://localhost:9000
resourceManager=localhost:8032
queueName=default
oozieRoot=user/${user.name}/oozie

oozie.wf.application.path=${nameNode}/${oozieRoot}/apps/wordcount/workflow.xml
inputDir=data/wordcount/input
outputDir=data/wordcount/output

$ cat oozie/apps/wordcount/workflow.xml


    
    
        
            ${resourceManager}
            ${nameNode}
            
                
            
            
                
                    mapred.mapper.new-api
                    true
                
                
                    mapred.reducer.new-api
                    true
                
                
                    mapreduce.job.queuename
                    ${queueName}
                
                
                    mapreduce.job.map.class
                    WordCount$TokenizerMapper
                
                
                    mapreduce.job.reduce.class
                    WordCount$IntSumReducer
                
                
                    mapreduce.job.output.key.class
                    org.apache.hadoop.io.Text
                
                
                    mapreduce.job.output.value.class
                    org.apache.hadoop.io.IntWritable
                
                
                    mapreduce.input.fileinputformat.inputdir
					${nameNode}/${oozieRoot}/${inputDir}
                
                
                    mapreduce.output.fileoutputformat.outputdir
                    ${nameNode}/${oozieRoot}/${outputDir}
                
            
        
        
        
    
    
        Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
    
    

Note: all key / value pairs in workflow.xml can be found from
http://localhost:8088/cluster -> Application's History -> Configuration

$ cat oozie/data/wordcount/input/in.txt

hadoop
mapreduce
hdfs
hbase
hdfs
spark
hive
hcatalog
sqoop
flume
kafka
oozie
hue

## upload case
$ hdfs dfs -put oozie /user/sun_xo/oozie
$ hdfs dfs -rm -f -r /user/sun_xo/oozie/data/wordcount/output

## start oozie
$ cd ~/work/oozie-5.2.1
$ bin/oozied.sh start
$ export OOZIE_URL=http://localhost:11000/oozie
$ bin/oozie job -config oozie/apps/wordcount/job.properties -run
job: 0000000-220523112321053-oozie-sun_-W
$ bin/oozie job -info 0000000-220523112321053-oozie-sun_-W

ob ID : 0000000-220602202911687-oozie-sun_-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : wordcount-wf
App Path      : hdfs://localhost:9000/user/sun_xo/oozie/apps/wordcount/workflow.xml
Status        : SUCCEEDED
Run           : 0
User          : sun_xo
Group         : -
Created       : 2022-06-02 12:30 GMT
Started       : 2022-06-02 12:30 GMT
Last Modified : 2022-06-02 12:30 GMT
Ended         : 2022-06-02 12:30 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code  
------------------------------------------------------------------------------------------------------------------------------------
0000000-220602202911687-oozie-sun_-W@:start:                                  OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------
0000000-220602202911687-oozie-sun_-W@wordcount-node                           OK        application_1653709899071_0004SUCCEEDED  -         
------------------------------------------------------------------------------------------------------------------------------------
0000000-220602202911687-oozie-sun_-W@end                                      OK        -                      OK         -         
------------------------------------------------------------------------------------------------------------------------------------

$ hdfs dfs -cat "/user/sun_xo/oozie/data/wordcount/output/*"

flume	1
hadoop	1
hbase	1
hcatalog	1
hdfs	2
hive	1
hue	1
kafka	1
mapreduce	1
oozie	1
spark	1
sqoop	1

reference: 搭建Ozzie环境
reference: 搭建map-reduce开发环境

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/946630.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号