肝到半夜3点,上亿行数据,秒级查询

当前有一个大约10G的txt文本文件,数据格式如下

uid        phone
10位数字  11位数字

需求是要根据uid做到秒级的查询,显示其对应的phone
​

将所有的数据按照id做一个排序
然后将排序好的数据做二分查找(理论上可以1秒以内)
​

普通单机排序肯定不行了,跑的话得跑老长时间
所以选择最近刚刚学的hadoop

在排好序的基础上进行二分查询
这种算法就算是全地球的数据加起来也能秒出
但是遇到的问题是,如何做到文件的随机读取呢?
经过一番百度与前辈的指导找到了java随机读取文件的api RandomAccessFile

将下面的代码放到集群上跑个半小时就完成了排序

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

import java.io.IOException;


public class Sort {
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        //参数解析器(这个一定要放在job创建之前!!!!)
        GenericOptionsParser optionsParser = new GenericOptionsParser(conf,args);
        String[] remainingArgs = optionsParser.getRemainingArgs();
        Job job = Job.getInstance(conf,"long类型数据排序");

        job.setJarByClass(Sort.class);
        job.setMapperClass(SortMapper.class);
        job.setReducerClass(SortReducer.class);

        job.setOutputKeyClass(LongWritable.class);
        job.setOutputValueClass(LongWritable.class);

        String[] input = remainingArgs[0].split(",");
        for(String filePath:input){
            FileInputFormat.addInputPath(job,new Path(filePath));
        }
        //指定输出结果文件路径
        FileOutputFormat.setOutputPath(job,new Path(remainingArgs[1]));
        //指定job执行模式(等待任务执行完成后，提交任务的客户端才会退出!)
        System.exit(job.waitForCompletion(true) ? 0 : 1);

    }
    public static class SortMapper extends Mapper {
        private LongWritable resKey = new LongWritable();
        private LongWritable resValue = new LongWritable();
        private long key;
        private long val;
        @Override
        protected void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            String[] split = value.toString().split("t");
            //没切分成功的也直接跳过
            if(split.length!=2){
                return;
            }

            //对长度不正确的直接跳过
            if(split[0].length()!=11||split[1].length()!=10){
                System.out.println("-------长度不正确的脏数据---------");
                System.out.println("脏数据key--->"+split[1]);
                System.out.println("脏数据value--->"+split[0]);
                return;
            }

            try {
                resKey.set(Long.parseLong(split[1]));
                resValue.set(Long.parseLong(split[0]));
            } catch (NumberFormatException e) {
                System.out.println("-------非数字的脏数据直接跳过---------");
                System.out.println("脏数据key--->"+split[1]);
                System.out.println("脏数据value--->"+split[0]);
                return;
            }
            context.write(resKey,resValue);
        }
    }
    public static class SortReducer extends Reducer {
        @Override
        protected void reduce(LongWritable key, Iterable values, Context context) throws IOException, InterruptedException {
            for(LongWritable val :values){
                context.write(key,val);
            }
        }
    }
}

这里仅放了文件随机读写的代码,这个方法是最核心的,二分法百度一大堆

	//这个方法是文件随机读取的核心
    public static String readByRowNum(long rowNum,int rowByte,File fin) {
        byte[] buf = new byte[0];
        try {
            RandomAccessFile ras = new RandomAccessFile(fin,"r");
            buf = new byte[rowByte];
            long totalOffset =  rowByte*(rowNum-1);
            //跳过一定的字节
            ras.seek(totalOffset);
            //从跳过字节开始的地方开始读取
            ras.read(buf,0,rowByte);
        } catch (IOException e) {
            return null;
        }
        return new String(buf);
    }

做这个小项目的最大收获就是了解到了java的文件随机读写在处理大文件的时候可以很方便
有任何技术问题的大家可以私信我,一起进步.