s
hadoop完全分布式搭建 一、前期准备 新建一台虚拟机 1、基础配置配置网络（静态ip）
vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet PROXY_METHOD=none BROWSER_onLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=e03b9c84-1624-432a-935a-c8c67420d34b DEVICE=ens33 onBOOT=yes IPADDR=192.168.1.10 NETMASK=255.255.255.0 GATEWAY=192.168.1.2 DNS1=144.144.144.144 DNS2=8.8.8.8
重启网络使网络生效
systemctl restart network
关闭防火墙
systemctl stop firewalld
查看防火墙状态
systemctl status firewalld
关闭selinux
vim /etc/selinux/config
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1fCzRYui-1633676973350)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007204417782.png)]
创建hadoop用户
2、克隆三个虚拟机（2_master、2_slave1、2_slave2）关机克隆
更改三台虚拟机的静态ip
修改主机名
hostnamectl set-hostname master hostnamectl set-hostname slave1 hostnamectl set-hostname slave2
添加主机名的映射
vim /etc/hosts
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-iSb0I0Ha-1633676973352)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007211918746.png)]
三台主机互相免密登录
[root@master ~]# ssh-keygen -t rsa -P '' [root@master ~]# ssh-copy-id -i .ssh/id_rsa.pub root@master [root@master ~]# ssh-copy-id -i .ssh/id_rsa.pub root@slave1 [root@master ~]# ssh-copy-id -i .ssh/id_rsa.pub root@slave2
查看密钥
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-keuKnaRg-1633676973353)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007212851393.png)]
二、配置hadoop 1、安装Hadoop解压hadoop压缩包和jdk压缩包
[root@master ~]# tar zxvf hadoop-2.7.1.tar.gz -C /usr/local/src [root@master ~]# tar zxvf jdk-8u231-linux-x64.tar.gz -C /usr/local/src
修改Hadoop和jdk所在文件夹名
[root@master ~]# mv /usr/local/src/hadoop-2.7.1/ /usr/local/src/hadoop [root@master ~]# mv /usr/local/src/jdk1.8.0_231/ /usr/local/src/jdk1.8
配置hadoop和jdk的环境变量
[root@master ~]# vim /etc/profile
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-SNPky7rK-1633676973354)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007220710475.png)]
使环境变量生效
[root@master hadoop]# source /etc/profile2、修改配置文件
配置hadoop-env.sh文件，目的是告诉Hadoop系统jdk的安装目录
[root@master hadoop]# vim etc/hadoop/hadoop-env.sh
25行注释掉
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-biNtpykK-1633676973355)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007221801459.png)]
切换路径更改配置文件/usr/local/src/hadoop/etc/hadoop
[root@master hadoop]# vim hdfs-site.xml
dfs.namenode.name.dir file:/usr/local/src/hadoop/dfs/name dfs.namenode.data.dir file:/usr/local/src/hadoop/dfs/data dfs.replication 3
[root@master hadoop]# vim core-site.xml
fs.defaultFS hdfs://192.168.1.14:9000 io.file.buffer.size 131072 hadoop.tmp.dir file:/usr/local/src/hadoop/tmp
在“/usr/local/src/hadoop/etc/hadoop”目录下有一个mapred-site.xml.template，需要修改文件名成，把它重命名为mapred-site.xml然后修改mapred-site.xml文件配置
[root@master hadoop]# cd /usr/local/src/hadoop/etc/hadoop/ [root@master hadoop]# cp mapred-site.xml.template mapred-site.xml [root@master hadoop]# vim mapred-site.xml
mapreduce.framework.name yarn mapreduce.jobhistory.address master:10020 mapreduce.jobhistory.webapp.address master:19888
[root@master hadoop]# vim yarn-site.xml
3、hadoop其他相关配置文件yarn.resourcenmanager.address master:8032 yarn.resourcemanager.scheduler.address master:8030 yarn.resourcemanager.resource-tracker.address master:8031 yarn.resourcemanager.admin.address master:8033 yarn.resourcemanager.webapp.address master:8088 yarn.resourcemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler
修改masters配置文件
[root@master hadoop]# vim masters
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-INj44DfN-1633676973356)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007230517211.png)]
修改slaves配置文件
[root@master hadoop]# vim slaves
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JmVFCdax-1633676973356)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007230738545.png)]
新建目录
[root@master hadoop]# mkdir /usr/local/src/hadoop/tmp [root@master hadoop]# mkdir /usr/local/src/hadoop/dfs/name -p [root@master hadoop]# mkdir /usr/local/src/hadoop/dfs/data -p同步配置文件到slave 节点
hadoop用户之间相互免密
[root@master hadoop]# su hadoop [hadoop@master hadoop]$ cd [hadoop@master ~]$ ssh-keygen -t rsa -P '' [hadoop@master ~]$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@master [hadoop@master ~]$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@slave1 [hadoop@master ~]$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@slave2
登出Hadoop用户
[hadoop@master ~]$ exit
（1）将master上的Hadoop安装文件同步到slave1、slave2
[root@master ~]# scp -r /usr/local/src/ root@slave1:/usr/local/ [root@master ~]# scp -r /usr/local/src/ root@slave2:/usr/local/
（2）在每个slave节点上配置Hadoop和jdk的环境变量
[root@slave1 ~]# vim /etc/profile
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-PYLUemt7-1633676973357)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211007234833935.png)]
（3）是每个slave节点上的环境变量生效
[root@slave1 ~]# source /etc/profile
（4）在每个master 和slave节点上修改/usr/local/src/hadoop目录的权限
[root@slave1 ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/
（5）在每个slave节点上切换用户到hadoop用户
[root@slave1 ~]# su - hadoop三、Hadoop集群运行
配置Hadoop格式化
[hadoop@master hadoop]$ cd /usr/local/src/hadoop/ [hadoop@master hadoop]$ bin/hdfs namenode -format
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-UIGx5wzn-1633676973358)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211008003101036.png)]
status 0 表示成功
启动NameNode、DataNode、SecondaryNameNode进程
[hadoop@master hadoop]$ start-all.sh
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-9CYQhgO6-1633676973358)(C:UsersLenovoAppDataRoamingTyporatypora-user-imagesimage-20211008003425386.png)]
ll
[root@slave1 ~]# su - hadoop
## 三、Hadoop集群运行 配置Hadoop格式化 ```shell [hadoop@master hadoop]$ cd /usr/local/src/hadoop/ [hadoop@master hadoop]$ bin/hdfs namenode -format
[外链图片转存中…(img-UIGx5wzn-1633676973358)]
status 0 表示成功
启动NameNode、DataNode、SecondaryNameNode进程
[hadoop@master hadoop]$ start-all.sh
[外链图片转存中…(img-9CYQhgO6-1633676973358)]
