栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

Hive TaskFactory 生成 Task

Hive TaskFactory 生成 Task

TaskFactory#get(T, HiveConf)

本静态方法生成根据 woker 和 conf,得到 Task。首先根据 worker 的 class,得到一个 Task,并且设置 worker。

public static  Task get(T work, HiveConf conf) {
    @SuppressWarnings("unchecked")
    Task ret = get((Class) work.getClass());
    ret.setWork(work);
    if (null != conf) {
      ret.setConf(conf);
    }
    return ret;
  }
TaskFactory#get(Class)

本静态方法生成根据 Task,遍历 taskvec,如果 workerClass 和 Worker 的class 相等,则拿到 taskClass,进行创建实例。

@VisibleForTesting
  static  Task get(Class workClass) {

    for (TaskTuple t : taskvec) {
      if (t.workClass == workClass) {
        try {
          Task ret = (Task) t.taskClass.newInstance();
          ret.setId("Stage-" + Integer.toString(getAndIncrementId()));
          return ret;
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
      }
    }

    throw new RuntimeException("No task for work class " + workClass.getName());
  }
taskvec

存储了所有 Worker class 和 Task Class 的映射。

public static ArrayList> taskvec;
  static {
    taskvec = new ArrayList>();
    taskvec.add(new TaskTuple(MoveWork.class, MoveTask.class));
    taskvec.add(new TaskTuple(FetchWork.class, FetchTask.class));
    taskvec.add(new TaskTuple(CopyWork.class, CopyTask.class));
    taskvec.add(new TaskTuple(ReplCopyWork.class, ReplCopyTask.class));
    taskvec.add(new TaskTuple(DDLWork.class, DDLTask.class));
    taskvec.add(new TaskTuple(
        MaterializedViewDesc.class,
        MaterializedViewTask.class));
    taskvec.add(new TaskTuple(FunctionWork.class,
        FunctionTask.class));
    taskvec
        .add(new TaskTuple(ExplainWork.class, ExplainTask.class));
    taskvec
        .add(new TaskTuple(ExplainSQRewriteWork.class, ExplainSQRewriteTask.class));
    taskvec.add(new TaskTuple(ConditionalWork.class,
        ConditionalTask.class));
    taskvec.add(new TaskTuple(MapredWork.class,
                                          MapRedTask.class));

    taskvec.add(new TaskTuple(MapredLocalWork.class,
        MapredLocalTask.class));
    taskvec.add(new TaskTuple(StatsWork.class, StatsTask.class));
    taskvec.add(new TaskTuple(ColumnStatsUpdateWork.class, ColumnStatsUpdateTask.class));
    taskvec.add(new TaskTuple(MergeFileWork.class,
        MergeFileTask.class));
    taskvec.add(new TaskTuple(DependencyCollectionWork.class,
        DependencyCollectionTask.class));
    taskvec.add(new TaskTuple(TezWork.class, TezTask.class));
    taskvec.add(new TaskTuple(SparkWork.class, SparkTask.class));
    taskvec.add(new TaskTuple<>(ReplDumpWork.class, ReplDumpTask.class));
    taskvec.add(new TaskTuple<>(ReplLoadWork.class, ReplLoadTask.class));
    taskvec.add(new TaskTuple<>(ReplStateLogWork.class, ReplStateLogTask.class));
    taskvec.add(new TaskTuple(ExportWork.class, ExportTask.class));
    taskvec.add(new TaskTuple(ReplTxnWork.class, ReplTxnTask.class));
  }
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/335811.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号