记录Hadoop Ha + Hbase+ Spark高可用集群的搭建，主要包括每个组件的配置信息，以及启动步骤。
在ubuntu18.04环境下，集群可以正常使用，运行。
2.Ling-Ha集群架构信息| 节点 | Nn | Rm | DFSZK | Dn | Nm | Jn | Zoo | spark | Hm | Hr |
|---|---|---|---|---|---|---|---|---|---|---|
| node1 | √ | √ | √ | √ | √ | |||||
| node2 | √ | √ | √ | √ | ||||||
| node3 | √ | √ | √ | √ | √ | √ | ||||
| node4 | √ | √ | √ | √ | √ | √ | ||||
| node5 | √ | √ | √ | √ | √ | √ | ||||
| node6 | √ | √ | √ | √ |
| 集群 | 版本号 | 端口 |
|---|---|---|
| Hadoop | 3.2.2 | 9870 |
| Yarn | 3.2.2 | 8088 |
| MapReduce JobHistory Server | 3.2.2 | 19888 |
| Spark-master | 3.1.2 | 8080 |
| Spark-histoory | 3.1.2 | 4000 |
| hbase | 2.2.7 | 16010 |
| Zookeeper | 3.4.6 | 2181 |
| 缩写 | 全称 | 作用 |
|---|---|---|
| Nm | Namenode | 元数据节点 |
| Rm | ResourceManager | yarn资源管理节点 |
| DFSZK | DFSZKFailoverController | zookeeper监控节点,Ha配置 |
| Dn | Datanode | 数据节点 |
| Nm | NodeManager | yarn单节点管理,与Rm通信 |
| Jn | JournalNode | 同步NameNode之间数据,Ha配置 |
| Zoo | Zookeeper | zookeeper集群 |
| Hm | HMaster | Hbase主节点 |
| Hr | HRegionServer | Hbase从节点 |
3. Hadoop-Ha配置文件
- core-site.xml
fs.defaultFS hdfs://Ling-Ha hadoop.tmp.dir file:/usr/local/hadoop/tmp Abase for other temporary directories. ha.zookeeper.quorum node3:2181,node4:2181,node5:2181 ha.zookeeper.session-timeout.ms 10000
- hdfs-site.xml
dfs.nameservices Ling-Ha dfs.ha.namenodes.Ling-Ha nn1,nn2 dfs.namenode.rpc-address.Ling-Ha.nn1 node1:9000 dfs.namenode.http-address.Ling-Ha.nn1 node1:9870 dfs.namenode.rpc-address.Ling-Ha.nn2 node2:9000 dfs.namenode.http-address.Ling-Ha.nn2 node2:9870 dfs.namenode.shared.edits.dir qjournal://node3:8485;node4:8485;node5:8485/Ling-Ha dfs.journalnode.edits.dir /usr/local/hadoop/journal dfs.ha.automatic-failover.enabled true dfs.client.failover.proxy.provider.Ling-Ha org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence shell(true) dfs.ha.fencing.ssh.private-key-files /home/node1/.ssh/id_rsa dfs.ha.fencing.ssh.connect-timeout 60000 dfs.replication 3 dfs.namenode.name.dir file:/usr/local/hadoop/tmp/dfs/name dfs.datanode.data.dir file:/usr/local/hadoop/tmp/dfs/data dfs.permissions false
- yarn-site.xml
yarn.resourcemanager.cluster-id Ling-yarn yarn.resourcemanager.ha.enabled true yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 node1 yarn.resourcemanager.hostname.rm2 node2 yarn.resourcemanager.webapp.address.rm1 node1:8088 yarn.resourcemanager.webapp.address.rm2 node2:8088 yarn.resourcemanager.zk-address node3:2181,node4:2181,node5:2181 yarn.resourcemanager.ha.automatic-failover.enabled true yarn.resourcemanager.ha.automatic-failover.zk-base-path /yarn-leader-election yarn.resourcemanager.recovery.enabled true yarn.resourcemanager.store.class org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore yarn.nodemanager.aux-services mapreduce_shuffle
- mapred-site.xml
mapreduce.framework.name yarn mapreduce.jobhistory.address node1:10020 mapreduce.jobhistory.webapp.address node1:19888 yarn.app.mapreduce.am.env HADOOP_MAPRED_HOME=/usr/local/hadoop mapreduce.map.env HADOOP_MAPRED_HOME=/usr/local/hadoop mapreduce.reduce.env HADOOP_MAPRED_HOME=/usr/local/hadoop
- workers
node3 node4 node5 node6
4.Hadoop-Ha初始化启动顺序
- 启动zookeep集群
- 依次启动所有journalnode
#sbin目录下 ./hadoop-daemon.sh start journalnode
- 格式化主节点namenode,然后启动
hdfs namenode -format ./hadoop-daemon.sh start namenode
- 在从节点 同步主节点namenode到备份namenode
hdfs namenode -bootstrapStandby
- 在主节点namenode格式化zk
hdfs zkfc -formatZK
- 停止主节点namenode和所有节点的journalnode
#sbin目录下 ./hadoop-daemon.sh stop namenode ./hadoop-daemon.sh stop journalnode
- 执行脚本,启动所有节点
start-dfs.sh
5.Hadoop Ha模式下 配置spark和日志服务器
- 配置spark-env.sh
#!/usr/bin/env bash export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath) export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop export SPARK_MASTER_IP="填写主节点ip" #"配置日志服务器" export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=4000 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://Ling-Ha/sparkLog"
- 配置spark-default.conf
spark.eventLog.enabled true spark.eventLog.dir hdfs://Ling-Ha/sparkLog spark.eventLog.compress true spark.files file:///usr/local/spark/conf/hdfs-site.xml,file:///usr/local/spark/conf/core-site.xml
- 配置workers
node1 node2 node3 node4 node5 node6
- spark.eventLog.dir 路径需修改成hdfs-site.xml 中dfs.nameservices配置的值,且不需要指定端口
- 将hadoop的core-site.xml和hdfs-site.xml，放到spark的conf目录下，让spark能找到Hadoop的配置
- 修改hbase-env.sh
#java export JAVA_HOME=/usr/loca/jvm/hbase #hbase export Hbase_CLASSPATH=/usr/local/hbase/conf #不使用自带zookeeper export Hbase_MANAGES_ZK=false #不包含hadoop包 export Hbase_DISABLE_HADOOP_CLASSPATH_LOOKUP="true"
- 修改hbase-site.xml
hbase.cluster.distributed true hbase.rootdir hdfs://Ling-Ha/hbase hbase.zookeeper.property.dataDir /usr/local/hbase/zookeeper hbase.zookeeper.quorum node3:2181,node4:2181,node5:2181
- 修改regionservers
#添加节点名称 node3 node4 node5 node6
- 将hadoop的core-site.xml和hdfs-site.xml，放到Hbase的conf目录下，让Hbase能找到Hadoop的配置
- 配置Hbase Ha模式 在conf目录下创建backup-masters文件,输入备份节点名称
