hadoop序列化的实现的例子
MapReduce是一个分布式运算程序的编程框架,是用户开发“基于Hadoop的数据分析应用”的核心框架。
MapReduce核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的分布式运算程序,并发运行在一个Hadoop集群上。
在本次实验里面设定了三个属性UpFlow,DownFlow,SumFlow,
在计算的时候不能用Writable相加,要加只能long和龙相加,但是此事进行序列化,所以写了flowBean包装三个属性。
本个FlowBean自己实现了Writable接口,现在本身就相当于一个Writable
封装了三个long
但是LongWritable里面只包装了一个long,longWritable是实现了Writable这个接口
public class FlowBean implements Writable {
private long upFlow;//流量使用上线
private long downFlow;//流量使用时。一开始的流量
private long sumFlow;//总计这次用了多少的流量
public FlowBean(long upFlow, long downFlow, long sumFlow) {
this.upFlow = upFlow;
this.downFlow = downFlow;
this.sumFlow = sumFlow;
}
public FlowBean() {
}
public long getUpFlow() {
return upFlow;
}
public void setUpFlow(long upFlow) {
this.upFlow = upFlow;
}
public long getDownFlow() {
return downFlow;
}
public void setDownFlow(long downFlow) {
this.downFlow = downFlow;
}
public long getSumFlow() {
return sumFlow;
}
public void setSumFlow(long sumFlow) {
this.sumFlow = sumFlow;
}
@Override
public void write(DataOutput output) throws IOException{
output.writeLong(upFlow);
output.writeLong(downFlow);
output.writeLong(sumFlow);
}
@Override
public void readFields(DataInput input) throws IOException {
this.upFlow=input.readLong();
this.downFlow=input.readLong();
this.sumFlow=upFlow+downFlow;
}
//s虽然有setget方法,但是我们设置一个可以一次性,传输upFlow和downFlow的方法
//会更加的方便
public void set(long upFlow,long downFlow) {
this.upFlow =upFlow;
this.downFlow =downFlow;
this.sumFlow =upFlow + downFlow;
}
@Override
public String toString() {
return "FlowBean{" +
"upFlow=" + upFlow +
", downFlow=" + downFlow +
", sumFlow=" + sumFlow +
'}';
}
}
Mapper
这个map(LongWritable key, Text value , Mapper
方法的key代表的是我数据传进来的行号
value代表的是这一行的所有数据
package com.atguigu.mapreduce.flow; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import java.io.IOException; public class FlowMapper extends MapperReducer{ //封装这两个对象 private Text phone=new Text();//手机号 private FlowBean flowBean=new FlowBean();//流量 protected void map(LongWritable key, Text value , Mapper .Context context) throws IOException, InterruptedException { //按照庄行服进行切分 String[] fields=value.toString().split("t"); //手机号是我们的第一个字段,数组的0,我们不要,我们需要 phone.set(fields[1]); //f封装流量 //每一行的流量的顺序还不一定,所以只能从数据的最后,向前数 flowBean.set( //上行流量在倒数第三位 Long.parseLong(fields[fields.length-3]), //下行流量在倒数第二位 Long.parseLong(fields[fields.length-2]) ); //手机号和流量输出 // context 整个job的替身,相当于this,就是将整个job抽象为一个对象,用来在程序中获取job相关信息的吗 // 一个容器产生一个context //Context是map里面的内部类 context.write(phone,flowBean); //此时 context.write(phone,flowBean)对应的是Mapper //里面的后两个泛型Text和FlowBean } }
package com.atguigu.mapreduce.flow; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; import java.awt.*; import java.io.IOException; public class FlowReducer extends ReducerDriver{ //将sumUpFlow和 sumDownFlow封装成一个新的flowBean private FlowBean result = new FlowBean(); @Override protected void reduce(Text key,Iterable values, Reducer .Context context ) throws IOException, InterruptedException { long sumUpFlow=0; long sumDownFlow=0; for (FlowBean value: values ) { sumDownFlow+=value.getDownFlow(); sumUpFlow+=value.getUpFlow(); } result.set(sumUpFlow,sumDownFlow); context.write(key,result); } }
此处有两个输出,但是只有一个真正的reduce输出
map要输出给reduce
map是将文件输出位(value,1)
而reduc把相同word的1加一起,输出 这个value写了多少次
所以都要输出
package com.atguigu.mapreduce.flow;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
public class FlowDriver {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
//获取job对象
Job job=Job.getInstance(new Configuration());
//设置mapper和reducer
//3 关联Mapper和Reducer
job.setMapperClass(FlowMapper.class);
job.setReducerClass(FlowReducer.class);
//设置输出类型
//4 设置Map端输出KV类型
job.setMapOutputKeyClass(Text.class);//map的Key输出的是手机号
job.setMapOutputValueClass(FlowBean.class);//Value输出的是flowBean
//5 设置程序最终输出的KV类型
job.setOutputKeyClass(Text.class);//reduce的Key输出的也是手机号
job.setOutputValueClass(FlowBean.class);
//输入输出路径
FileInputFormat.setInputPaths(job,new Path("D:\Atguigu\BigData\input\phone.txt"));
FileOutputFormat.setOutputPath(job,new Path("D:\Atguigu\BigData\output"));
Boolean b=job.waitForCompletion(true);
System.exit(b?0:1);
}
}



