- Yarn 容量调度器多队列提交案例
- 需求
- 配置多队列的容量调度器
- 1 修改如下配置
- SecureCRT的上传和下载
- 2 上传到集群并分发
- 3 重启Yarn或yarn rmadmin -refreshQueues
- 4 向Hive队列提交任务
- 提交方式-打jar包的方式
- 任务优先级
- 任务优先级的使用
默认只有一个default队列,不能满足生产要求。一般按照业务模块如登录注册、购物车等创建队列。
需求需求1:default队列占总内存的40%,最大资源容量占总资源60%(本身占40%可以再借用20%),hive队列占总内存的60%,最大资源容量占总资源80%
需求2:配置队列优先级
在/opt/module/hadoop-3.1.3/etc/hadoop下的capacity-scheduler.xml中配置
1 修改如下配置直接配置不好配,我们先下载
[ranan@hadoop102 hadoop]$ sz capacity-scheduler.xml
修改如下配置
yarn.scheduler.capacity.root.queues default,hive The queues at the this level (root is the root queue). yarn.scheduler.capacity.root.default.capacity 40 Default queue target capacity. yarn.scheduler.capacity.root.hive.capacity 60 Default queue target capacity. yarn.scheduler.capacity.root.hive.user-limit-factor 1 hive queue user limit a percentage from 0.0 to 1.0. yarn.scheduler.capacity.root.default.maximum-capacity 60 The maximum capacity of the default queue. yarn.scheduler.capacity.root.hive.maximum-capacity 80 The maximum capacity of the hive queue. yarn.scheduler.capacity.root.hive.state RUNNING The state of the hive queue. State can be one of RUNNING or STOPPED. yarn.scheduler.capacity.root.hive.acl_submit_applications * The ACL of who can submit jobs to the hive queue. yarn.scheduler.capacity.root.default.acl_administer_queue * The ACL of who can administer jobs on the hive queue. yarn.scheduler.capacity.root.hive.acl_application_max_priority * The ACL of who can submit applications with configured priority. For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}] yarn.scheduler.capacity.root.hive.maximum-application-lifetime -1 Maximum lifetime of an application which is submitted to a queue in seconds. Any value less than or equal to zero will be considered as disabled. This will be a hard time limit for all applications in this queue. If positive value is configured then any application submitted to this queue will be killed after exceeds the configured lifetime. User can also specify lifetime per application basis in application submission context. But user lifetime will be overridden if it exceeds queue maximum lifetime. It is point-in-time configuration. Note : Configuring too low value will result in killing application sooner. This feature is applicable only for leaf queue. yarn.scheduler.capacity.root.hive.default-application-lifetime -1 Default lifetime of an application which is submitted to a queue in seconds. Any value less than or equal to zero will be considered as disabled. If the user has not submitted application with lifetime value then this value will be taken. It is point-in-time configuration. Note : Default lifetime can't exceed maximum lifetime. This feature is applicable only for leaf queue.
补充:
容量调度器所有的队列从根目录开始?
SecureCRT下载sz(send发送)
下载一个文件:sz filename
下载多个文件:sz filename1 filename2
下载dir目录下的所有文件,不包含dir下的文件夹:sz dir/*
rz(received)上传
[ranan@hadoop102 hadoop]$ rz [ranan@hadoop102 hadoop]$ xsync capacity-scheduler.xml3 重启Yarn或yarn rmadmin -refreshQueues
重启Yarn或者执行yarn rmadmin -refreshQueues 更新yarn队列相关配置
[ranan@hadoop102 hadoop]$ yarn rmadmin -refreshQueues4 向Hive队列提交任务
知识点:-D 表示运行时改变参数值
-D mapreduce.job.queuename=hive
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -D mapreduce.job.queuename=hive /input /output
提交到了hive队列,默认是default队列
如果是自己写的程序,可以再打包的配置信息Driver中声明提交到哪个队列
public class WcDrvier {
public static void main(String[] args) throws IOException,
ClassNotFoundException, InterruptedException {
Configuration conf = new Configuration();
conf.set("mapreduce.job.queuename","hive");
//1. 获取一个 Job 实例
Job job = Job.getInstance(conf);
....
//6. 提交 Job
boolean b = job.waitForCompletion(true);
System.exit(b ? 0 : 1);
}
}
任务优先级
容量调度器,在资源紧张时,优先级高的任务将优先获取资源。
默认情况,所有任务优先级为0,如果需要使用任务优先级,需要做相关的配置。
在/opt/module/hadoop-3.1.3/etc/hadoop下的yarn-site.xml中配置
1.修改 yarn-site.xml 文件,增加以下参数
yarn.cluster.max-application-priority 5
2.分发配置,并重启 Yarn
[ranan@hadoop102 hadoop]$ xsync yarn-site.xml //仅重启Yarn [ranan@hadoop102 hadoop-3.1.3]$ sbin/stop-yarn.sh [ranan@hadoop102 hadoop-3.1.3]$ sbin/start-yarn.sh
3.模拟资源紧张环境, 可连续提交以下任务,直到新提交的任务申请不到资源为止。
//求pi 执行了2000000次 [ranan@hadoop102 hadoop-3.1.3]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi 5 2000000
4.再次重新提交优先级高的任务,让优先级高的任务限制性
-D mapreduce.job.priority=5
[ranan@hadoop102 hadoop-3.1.3]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar pi -D mapreduce.job.priority=5 5 2000000
5.如果优先级高的任务已经提交到集群上了,也可以通过以下命令修改正在执行的任务的优先级。
yarn application -appID -updatePriority 优先级
[ranan@hadoop102 hadoop-3.1.3]$ yarn application -appID application_1611133087930_0009 -updatePriority 5



