MapReduce之单词统计

提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档

• 一、搭建环境
• 二、编写WordCount程序
• • 1.在IDEA中引入所需的jar包，IDEA支持文件夹方式引入
  • 2.代码编写
  • 3.代码打包
  • 4.程序调试
• 总结

window下安装jdk和idea
VM安装Linux
配置JDK：JDK是Java运行环境，基于Java编写的应用程序都需要Java运行环境，显然Hadoop是基于Java开发的，所以也需要配置JDK环境。
上传并解压jdk压缩包至Linux服务器
配置jdk全局环境变量
安装Hadoop：上传并解压jdk压缩包至Linux服务器；配置Hadoop全局环境变量
伪分布式配置：配置核心文件hadoop-env.sh、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml

1.编写Mapper函数
2.编写Reducer函数
3.编写Main函数主入口

Mapper代码如下：

import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper extends Mapper {

   @Override
   protected void map(LongWritable key, Text value, Context context)
         throws IOException, InterruptedException {
      
      String data= value.toString();
      //分词
      String[] words = data.split(" ");
      
      //输出每个单词
      for(String w:words){
         context.write(new Text(w), new LongWritable(1));
      }
   }

}

Main函数主入口如下：

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCountMain {

   public static void main(String[] args) throws Exception {
      //创建一个job = map + reduce
      Configuration conf = new Configuration();
      
      //创建一个Job
      Job job = Job.getInstance(conf);
      //指定任务的入口
      job.setJarByClass(WordCountMain.class);
      
      //指定job的mapper
      job.setMapperClass(WordCountMapper.class);
      job.setMapOutputKeyClass(Text.class);
      job.setMapOutputValueClass(LongWritable.class);
      
      //指定job的reducer
      job.setReducerClass(WordCountReducer.class);
      job.setOutputKeyClass(Text.class);
      job.setOutputValueClass(LongWritable.class);
      
      //指定任务的输入和输出
      FileInputFormat.setInputPaths(job, new Path(args[0]));
      FileOutputFormat.setOutputPath(job, new Path(args[1]));       
      
      //提交任务
      job.waitForCompletion(true);
   }

}







Reducer代码如下：



```c
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends Reducer{

   @Override
   protected void reduce(Text k3, Iterable v3,Context context) throws IOException, InterruptedException {
      //v3: 是一个集合，每个元素就是v2
      long total = 0;
      for(LongWritable l:v3){
         total = total + l.get();
      }
      
      //输出
      context.write(k3, new LongWritable(total));
   }

}