栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

Yarn 容量调度器多队列提交案例

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

Yarn 容量调度器多队列提交案例

文章目录
  • Yarn 容量调度器多队列提交案例
    • 需求
    • 配置多队列的容量调度器
      • 1 修改如下配置
        • SecureCRT的上传和下载
      • 2 上传到集群并分发
      • 3 重启Yarn或yarn rmadmin -refreshQueues
      • 4 向Hive队列提交任务
        • 提交方式-打jar包的方式
    • 任务优先级
      • 任务优先级的使用

Yarn 容量调度器多队列提交案例

默认只有一个default队列,不能满足生产要求。一般按照业务模块如登录注册、购物车等创建队列。

需求

需求1:default队列占总内存的40%,最大资源容量占总资源60%(本身占40%可以再借用20%),hive队列占总内存的60%,最大资源容量占总资源80%
需求2:配置队列优先级

配置多队列的容量调度器

在/opt/module/hadoop-3.1.3/etc/hadoop下的capacity-scheduler.xml中配置

1 修改如下配置

直接配置不好配,我们先下载

[ranan@hadoop102 hadoop]$ sz capacity-scheduler.xml

修改如下配置


    yarn.scheduler.capacity.root.queues
	
    default,hive
    
      The queues at the this level (root is the root queue).
    



    yarn.scheduler.capacity.root.default.capacity
	
    40
    Default queue target capacity.




    yarn.scheduler.capacity.root.hive.capacity
	
    60
    Default queue target capacity.




    yarn.scheduler.capacity.root.hive.user-limit-factor
    1
    
      hive queue user limit a percentage from 0.0 to 1.0.
    




    yarn.scheduler.capacity.root.default.maximum-capacity
    60
    
      The maximum capacity of the default queue. 
    




    yarn.scheduler.capacity.root.hive.maximum-capacity
    80
    
      The maximum capacity of the hive queue. 
    




   yarn.scheduler.capacity.root.hive.state
   RUNNING
   
   The state of the hive queue. State can be one of RUNNING or STOPPED.
   



 
   yarn.scheduler.capacity.root.hive.acl_submit_applications
   *
   
   The ACL of who can submit jobs to the hive queue.
   




   yarn.scheduler.capacity.root.default.acl_administer_queue
   *
   
   The ACL of who can administer jobs on the hive queue.
   



   
	yarn.scheduler.capacity.root.hive.acl_application_max_priority
    *
    
      The ACL of who can submit applications with configured priority.
      For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
    
  



   
     yarn.scheduler.capacity.root.hive.maximum-application-lifetime
     
     -1
     
        Maximum lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        This will be a hard time limit for all applications in this
        queue. If positive value is configured then any application submitted
        to this queue will be killed after exceeds the configured lifetime.
        User can also specify lifetime per application basis in
        application submission context. But user lifetime will be
        overridden if it exceeds queue maximum lifetime. It is point-in-time
        configuration.
        Note : Configuring too low value will result in killing application
        sooner. This feature is applicable only for leaf queue.
     
   


   
     yarn.scheduler.capacity.root.hive.default-application-lifetime
     
     -1
     
        Default lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        If the user has not submitted application with lifetime value then this
        value will be taken. It is point-in-time configuration.
        Note : Default lifetime can't exceed maximum lifetime. This feature is
        applicable only for leaf queue.
     
   

补充:
容量调度器所有的队列从根目录开始?

SecureCRT的上传和下载

SecureCRT下载sz(send发送)
下载一个文件:sz filename
下载多个文件:sz filename1 filename2
下载dir目录下的所有文件,不包含dir下的文件夹:sz dir/*
rz(received)上传

2 上传到集群并分发
[ranan@hadoop102 hadoop]$ rz
[ranan@hadoop102 hadoop]$ xsync capacity-scheduler.xml
3 重启Yarn或yarn rmadmin -refreshQueues

重启Yarn或者执行yarn rmadmin -refreshQueues 更新yarn队列相关配置

[ranan@hadoop102 hadoop]$ yarn rmadmin -refreshQueues

4 向Hive队列提交任务

知识点:-D 表示运行时改变参数值

-D mapreduce.job.queuename=hive

[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -D mapreduce.job.queuename=hive /input /output

提交到了hive队列,默认是default队列

提交方式-打jar包的方式

如果是自己写的程序,可以再打包的配置信息Driver中声明提交到哪个队列

public class WcDrvier {
	public static void main(String[] args) throws IOException,
		ClassNotFoundException, InterruptedException {
		Configuration conf = new Configuration();
		conf.set("mapreduce.job.queuename","hive");
		//1. 获取一个 Job 实例
		Job job = Job.getInstance(conf);
		....
			
		//6. 提交 Job
		boolean b = job.waitForCompletion(true);
		System.exit(b ? 0 : 1);
}
}
任务优先级

容量调度器,在资源紧张时,优先级高的任务将优先获取资源。
默认情况,所有任务优先级为0,如果需要使用任务优先级,需要做相关的配置。

任务优先级的使用

在/opt/module/hadoop-3.1.3/etc/hadoop下的yarn-site.xml中配置

1.修改 yarn-site.xml 文件,增加以下参数


yarn.cluster.max-application-priority

5

2.分发配置,并重启 Yarn

[ranan@hadoop102 hadoop]$ xsync yarn-site.xml
//仅重启Yarn
[ranan@hadoop102 hadoop-3.1.3]$ sbin/stop-yarn.sh
[ranan@hadoop102 hadoop-3.1.3]$ sbin/start-yarn.sh

3.模拟资源紧张环境, 可连续提交以下任务,直到新提交的任务申请不到资源为止。

//求pi 执行了2000000次
[ranan@hadoop102 hadoop-3.1.3]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 5 2000000

4.再次重新提交优先级高的任务,让优先级高的任务限制性

-D mapreduce.job.priority=5

[ranan@hadoop102 hadoop-3.1.3]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi -D mapreduce.job.priority=5 5 2000000

5.如果优先级高的任务已经提交到集群上了,也可以通过以下命令修改正在执行的任务的优先级。

yarn application -appID -updatePriority 优先级

[ranan@hadoop102 hadoop-3.1.3]$ yarn application -appID application_1611133087930_0009 -updatePriority 5

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/946313.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号