栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

Hadoop HA《hadoop-Yarn》

Hadoop HA《hadoop-Yarn》

1.Hadoop-Yarn

可参考我的文章《HadoopHA 搭建》

2.修改环境变量

cd /opt//hadoop-3.1.2/etc/hadoopvi hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_111
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_ZKFC_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
修改配置文件

vi mapred-site.xml


  mapreduce.framework.name
  yarn



  mapreduce.jobhistory.address
  10.1.1.58:10020



  mapreduce.jobhistory.webapp.address
  10.1.1.58:19888



mapreduce.jobhistory.done-dir
/history/done



mapreudce.jobhistory.intermediate.done-dir
/history/done/done_intermediate


  mapreduce.application.classpath
  
 /opt/test/hadoop-3.1.2/etc/hadoop,
 /opt/test/hadoop-3.1.2/share/hadoop/common/*,
 /opt/test/hadoop-3.1.2/share/hadoop/common/lib/*,
 /opt/test/hadoop-3.1.2/share/hadoop/hdfs/*,
 /opt/test/hadoop-3.1.2/share/hadoop/hdfs/lib/*,
 /opt/test/hadoop-3.1.2/share/hadoop/mapreduce/*,
 /opt/test/hadoop-3.1.2/share/hadoop/mapreduce/lib/*,
 /opt/test/hadoop-3.1.2/share/hadoop/yarn/*,
 /opt/test/hadoop-3.1.2/share/hadoop/yarn/lib/*
  


vi yarn-site.xml


  yarn.resourcemanager.ha.enabled
  true



  yarn.resourcemanager.cluster-id
  yarn-test



  yarn.resourcemanager.ha.rm-ids
  rm1,rm2



  yarn.resourcemanager.hostname.rm1
  10.1.1.58


  yarn.resourcemanager.hostname.rm2
  10.1.1.195


  yarn.resourcemanager.webapp.address.rm1
  10.1.1.58:8088


  yarn.resourcemanager.webapp.address.rm2
  10.1.1.195:8088



  yarn.resourcemanager.zk-address
  10.1.1.201:2181,10.1.1.158:2181,10.1.1.185:2181


  yarn.nodemanager.aux-services
  mapreduce_shuffle



  yarn.log-aggregation-enable
  true


  yarn.log-aggregation.retain-seconds
  86400



  yarn.resourcemanager.recovery.enabled
  true



  yarn.resourcemanager.store.class
  org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore



  yarn.nodemanager.vmem-check-enabled
  false


  yarn.nodemanager.vmem-pmem-ratio  3


分发
scp yarn-site.xml mapred-site.xml root@10.1.1.152:`pwd`
scp yarn-site.xml mapred-site.xml root@10.1.1.195:`pwd`
启动测试



zkServer.sh start

start-dfs.sh

start-yarn.sh

启动命令(会产生两条警告信息):
mr-jobhistory-daemon.sh start historyserver
可以使用命令启动:
mapred --daemon start historyserver
关闭命令:
 mr-jobhistory-daemon.sh stop historyserver
测试

http://10.1.1.195:8088

成功

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/706698.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号