栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

MapReduce之排序及序列化案例的代码实现

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

MapReduce之排序及序列化案例的代码实现

这里我们要实现的案例是先对学生姓名进行排序(字典排序),如果遇到姓名重名的情况,再对年龄进行排序(升序)。

  • 将原始数据文件上传至HDFS
[root@hadoop01 test_data]# hdfs dfs -mkdir /test_comparation_input
[root@hadoop01 test_data]# hdfs dfs -put test_comparation.txt /test_comparation_input

这里会对原始的5行数据先按照姓名的ascii值进行字典排序,对于重名的Bob,会对第2行和第4行的两个Bob按照年龄进行二次排序。

新建project:

  • 引入pom依赖


    4.0.0

    wyh.test
    test_comparation
    1.0-SNAPSHOT

    
        8
        8
    

    jar
    
        
            org.apache.hadoop
            hadoop-common
            2.7.5
        
        
            org.apache.hadoop
            hadoop-client
            2.7.5
        
        
            org.apache.hadoop
            hadoop-hdfs
            2.7.5
        
        
            org.apache.hadoop
            hadoop-mapreduce-client-core
            2.7.5
        
        
            junit
            junit
            RELEASE
        
    
    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                3.1
                
                    1.8
                    1.8
                    UTF-8
                
            
            
                org.apache.maven.plugins
                maven-shade-plugin
                2.4.3
                
                    
                        package
                        
                            shade
                        
                        
                            true
                        
                    
                
            
        
    

  • 自定义实现排序及序列化的Bean类
package wyh.test.comparation;

import org.apache.hadoop.io.WritableComparable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

public class ComparationBean implements WritableComparable {

    //定义Bean对象中要包含的属性
    private String studentName;
    private int age;

    public String getStudentName() {
        return studentName;
    }

    public void setStudentName(String studentName) {
        this.studentName = studentName;
    }

    @Override
    public String toString() {
        //这里没有使用它原来的格式,我们自己重新定义了toString的格式
        return studentName + "t" + age;
    }

    public int getAge() {
        return age;
    }

    public void setAge(int age) {
        this.age = age;
    }

    //定义排序规则,我们只需要定义比较规则即可,不需要我们关心如何调用
    @Override
    public int compareTo(ComparationBean comparationBean) {
        //先用String类中自带的compareTo()实现字符串studentName的排序
        int compareResult = this.studentName.compareTo(comparationBean.getStudentName());
        //当studentName比较结果是0时,说明重名,此时再比较age的值
        if(compareResult == 0){
            return this.age - comparationBean.getAge();
        }
        return compareResult;
    }

    //该方法用于实现序列化,即将原始数据转换为字节流
    @Override
    public void write(DataOutput dataOutput) throws IOException {
        //将Bean对象中的第一个属性实现序列化,对于字符串的序列化,使用的是writeUTF()
        dataOutput.writeUTF(studentName);
        //将Bean对象中的第二个属性实现序列化,对于int类型的序列化,使用的是writeInt()
        dataOutput.writeInt(age);
    }

    //该方法用于实现反序列化
    @Override
    public void readFields(DataInput dataInput) throws IOException {
        //将反序列化的值赋给成员变量
        this.studentName = dataInput.readUTF();
        this.age = dataInput.readInt();

    }
}
  • 自定义Mapper类
package wyh.test.comparation;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

//这里的K2类型是我们自定义的Bean(因为要实现排序及序列化),这个Bean对象中的属性来自于V1.
public class ComparationMapper extends Mapper {
    @Override
    protected void map(LongWritable key, Text value, Mapper.Context context) throws IOException, InterruptedException {
        //将V1的行数据拆分提取为两个属性,赋给自定义Bean对象
        String[] split = value.toString().split(",");
        ComparationBean comparationBean = new ComparationBean();
        comparationBean.setStudentName(split[0]);
        comparationBean.setAge(Integer.parseInt(split[1]));
        //将K2,V2写入context对象
        context.write(comparationBean, NullWritable.get());
    }
}
  • 自定义Reducer类
package wyh.test.comparation;

import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

//在我们的案例中reduce不需要对K2,V2做进一步的处理,所以这里直接将K2,V2赋值给K3,V3即可
public class ComparationReducer extends Reducer {
    @Override
    protected void reduce(ComparationBean key, Iterable values, Context context) throws IOException, InterruptedException {
        context.write(key, NullWritable.get());
    }
}
  • 自定义JobMain类
package wyh.test.comparation;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.OutputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class ComparationJobMain extends Configured implements Tool {
    @Override
    public int run(String[] strings) throws Exception {
        Job job = Job.getInstance(super.getConf(), "test_comparation_job");
        job.setJarByClass(ComparationJobMain.class);
        job.setInputFormatClass(TextInputFormat.class);
        TextInputFormat.addInputPath(job, new Path("hdfs://192.168.126.132:8020/test_comparation_input"));
        job.setMapperClass(ComparationMapper.class);
        job.setMapOutputKeyClass(ComparationBean.class);
        job.setMapOutputValueClass(NullWritable.class);
        job.setReducerClass(ComparationReducer.class);
        job.setOutputKeyClass(ComparationBean.class);
        job.setOutputValueClass(NullWritable.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        TextOutputFormat.setOutputPath(job, new Path("hdfs://192.168.126.132:8020/test_comparation_output"));
        boolean status = job.waitForCompletion(true);
        return status?0:1;
    }

    public static void main(String[] args) throws Exception {
        Configuration configuration = new Configuration();
        //启动job任务
        int runStatus = ToolRunner.run(configuration, new ComparationJobMain(), args);
        System.exit(runStatus);
    }
}
  • 打包project

clean---package

  • 将jar上传至服务器 

  •  运行jar
[root@hadoop01 test_jar]# hadoop jar test_comparation-1.0-SNAPSHOT.jar wyh.test.comparation.ComparationJobMain
  • 查看HDFS目录树

  •  查看输出结果文件
[root@hadoop01 test_jar]# hdfs dfs -cat /test_comparation_output/part-r-00000

可以看到姓名先做了排序,然后重名的Bob会按照年龄的升序进行二次排序:

 这样就简单地实现了MapReduce中的排序及序列化功能。

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/936433.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号