Hadoop集群的安装

Hadoop 集群的搭建

上传、解压、更名、配置环境变量
-1. 上传和解压两个软件包
[root@qianfeng01 ~]# tar -zxvf jdk-8u221-linux-x64.tar.gz -C
/usr/local/
[root@qianfeng01 ~]# tar -zxvf hadoop-2.7.6.tar.gz -C
/usr/local/
-2. 进⼊local⾥，给两个软件更名
[root@qianfeng01 ~]# cd /usr/local/
[root@qianfeng01 local]# mv 1.8.0_221/ jdk
[root@qianfeng01 local]# mv hadoop-2.7.6/ hadoop
-3. 配置环境变量
[hadoop@qianfeng01 local]# vi /etc/profile
.....省略...........
#java environment
export JAVA_HOME=/usr/local/jdk
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
#hadoop environment
export HADOOP_HOME=/usr/local/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

配置core-site.xml

[root@qianfeng01 ~]# cd $HADOOP_HOME/etc/hadoop/
[root@qianfeng01 hadoop]# vi core-site.xml

 
 
 fs.defaultFS
 hdfs://qianfeng01:8020
 
 
 
 hadoop.tmp.dir
 /usr/local/hadoop/tmp
 

配置hdfs-site.xml

[root@qianfeng01 hadoop]# vi hdfs-site.xml

 
 
 dfs.namenode.name.dir
 file://${hadoop.tmp.dir}/dfs/name
 
 
 
 dfs.datanode.data.dir
 file://${hadoop.tmp.dir}/dfs/data
 
 
 
 dfs.replication
 3
 
 
 
 dfs.blocksize
 134217728
 
 
 
 dfs.namenode.secondary.http-address
 qianfeng02:50090
 
 
 
 dfs.namenode.http-address
 qianfeng01:50070
  

配置mapred-site.xml

 [root@qianfeng01 hadoop]# cp mapred-site.xml.template 
mapred-site.xml
[root@qianfeng01 hadoop]# vi mapred-site.xml

 
 
 mapreduce.framework.name
 yarn
 
 
 
 mapreduce.jobhistory.address
 qianfeng01:10020
 
 
 
 mapreduce.jobhistory.webapp.address
 qianfeng01:19888
 

配置yarn-site.xml

[root@qianfeng01 hadoop]# vi yarn-site.xml

 
 
 yarn.nodemanager.aux-services
 mapreduce_shuffle
 
 
 
 yarn.resourcemanager.hostname
 qianfeng01
 
 
 
 
 yarn.nodemanager.auxservices.mapreduce_shuffle.class
 org.apache.hadoop.mapred.ShuffleHandler
 
 
 
 yarn.resourcemanager.address
 qianfeng01:8032
 
 
 
 yarn.resourcemanager.scheduler.address
 qianfeng01:8030
 
 
 
 yarn.resourcemanager.resourcetracker.address
 qianfeng01:8031
 
 
 
 yarn.resourcemanager.admin.address
 qianfeng01:8033
 
 
 
 yarn.resourcemanager.webapp.address
 qianfeng01:8088
 

配置hadoop-env.sh

[root@qianfeng01 hadoop]# vi hadoop-env.sh
.........
# The java implementation to use.
export JAVA_HOME=/usr/local/jdk
.........

配置slaves⽂件

# 此⽂件⽤于指定datanode守护进程所在的机器节点主机名
[root@qianfeng01 hadoop]# vi slaves
qianfeng01
qianfeng02
qianfeng03

配置yarn-env.sh⽂件

[root@qianfeng01 hadoop]# vi yarn-env.sh
.........
# some Java parameters
export JAVA_HOME=/usr/local/jdk
if [ "$JAVA_HOME" != "" ]; then
 #echo "run java in $JAVA_HOME"
 JAVA_HOME=$JAVA_HOME
fi
.........

--1. 同步hadoop到slave节点上
[root@qianfeng01 ~]# cd /usr/local
[root@qianfeng01 local]# scp -r ./hadoop qianfeng02:/usr/local/
[root@qianfeng01 local]# scp -r ./hadoop qianfeng03:/usr/local/
--2. 同步/etc/profile到slave节点上
[root@qianfeng01 local]# scp /etc/profile qianfeng02:/etc/
[root@qianfeng01 local]# scp /etc/profile qianfeng03:/etc/
--3. 如果slave节点上的jdk也没有安装，别忘记同步jdk。
--4. 检查是否同步了/etc/hosts⽂件

启动：第一次启动要格式化（namenode） hdfs namenode -format

1. 启动脚本
 -- start-dfs.sh :⽤于启动hdfs集群的脚本
 -- start-yarn.sh :⽤于启动yarn守护进程
 -- start-all.sh :⽤于启动hdfs和yarn
2. 关闭脚本
 -- stop-dfs.sh :⽤于关闭hdfs集群的脚本
 -- stop-yarn.sh :⽤于关闭yarn守护进程
 -- stop-all.sh :⽤于关闭hdfs和yarn
3. 单个守护进程脚本
 -- hadoop-daemons.sh :⽤于单独启动或关闭hdfs的某⼀个守护进
程的脚本
 -- hadoop-daemon.sh :⽤于单独启动或关闭hdfs的某⼀个守护进
程的脚本reg:
 hadoop-daemon.sh [start|stop]
[namenode|datanode|secondarynamenode

-- yarn-daemons.sh :⽤于单独启动或关闭hdfs的某⼀个守护进程的
脚本
 -- yarn-daemon.sh :⽤于单独启动或关闭hdfs的某⼀个守护进
程的脚本
 reg:
 yarn-daemon.sh [start|stop]
[resourcemanager|nodemanager]]

**启动hdfs

 使⽤start-dfs.sh，启动 hdfs
 jsp查看进程
 WebUI
 http://192.168.10.101:50070

完成