Hadoop高可用配置（HA）

一、组件部署架构

tunabook102：NameNode、ResourceManager、ZooKeeper、ZKFC
tunabook103：NameNode、ZKFC、ZooKeeper
tunabook104：DataNode、QJM、ZooKeeper、NodeManager
tunabook105：DataNode、QJM、ZooKeeper、NodeManager
tunabook106：DataNode、QJM、ZooKeeper、NodeManager

二、配置文件（tunabook102tunabook103tunabook104tunabook105tunabook106）

core-site.xml


    
	fs.defaultFS
	hdfs://cluster1
    
    
	hadoop.security.authorization
	false
    
    
	ha.zookeeper.quorum
	
	    tunabook102:2181,tunabook103:2181,tunabook:104:2181,tunabook:105,tunabook:106

hadoop-env.sh

export JAVA_HOME=/export/server/jdk1.8.0_65
export HADOOP_MAPRED_HOME=/export/server/hadoop-3.3.0
export HDFS_JOURNALNODE_USER=root
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root 
export HDFS_ZKFC_USER=root

hdfs-site.xml


	
		dfs.replication
		3
	
	
		dfs.permissions
		false
	
	
		dfs.namenode.name.dir
		/export/hdfs/namenode
	
	
		dfs.datanode.data.dir
		/export/hdfs/datanode
	
	
		dfs.nameservices
		cluster1
	
	
		dfs.ha.namenodes.cluster1
		nn1,nn2	
	
	
		dfs.namenode.rpc-address.cluster1.nn1
		tunabook102:9000
	
	
		dfs.namenode.rpc-address.cluster1.nn2
		tunabook103:9000
	
	
		dfs.namenode.http-address.cluster1.nn1
		tunabook102:9870
	
	
		dfs.namenode.http-address.cluster1.nn2
		tunabook103:9870
	
	
		dfs.namenode.shared.edits.dir
		qjournal://tunabook104:8485;tunabook105:8485;tunabook106:8485/cluster1
	
	
		dfs.journalnode.edits.dir
		/export/qjmdata
	
	
		dfs.client.failover.proxy.provider.cluster1
		org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
	
	
   		dfs.ha.automatic-failover.enabled
		true
	
	
		dfs.ha.fencing.methods
		sshfence

mapred-site.xml


	
		mapreduce.framework.name
		yarn
	
	
		mapreduce.application.classpath
		$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
		$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

workers
```
tunabook104
tunabook105
tunabook106
```

yarn-site.xml


	
		yarn.nodemanager.aux-services
		mapreduce_shuffle
	
	
		yarn.nodemanager.env-whitelist
		
			JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,
			CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME

zoo.cfg

dataDir=/export/data/zkdata

server.1=tunabook102:2888:3888
server.2=tunabook103:2888:3888
server.3=tunabook104:2888:3888
server.4=tunabook105:2888:3888
server.5=tunabook106:2888:3888

三、启动集群

启动ZooKeeper（tunabook102tunabook103tunabook104tunabook105tunabook106）
```
zkServer.sh start
```
初始化HA HDFS所使用的数据(tunabook102）
```
hdfs zkfc -formatZK
```
启动QJM（tunabook104tunabook105tunabook106）
```
hdfs --daemon start journalnode
```
格式化HFDS （tunabook102）
```
hdfs namenode -format
```
启动namenode（tunabook102）
```
hdfs --daemon start namenode
```
同步初始化结果（tunabook103）
```
hdfs namenode -bootstrapStandby
```
关闭namenode（tunabook102）
```
hdfs --daemon stop namenode
```
开启hadoop服务
```
start-all.sh
```

四、运行结果

Hadoop高可用配置（HA）

大数据系统相关栏目本月热门文章