Hadoop集群搭建
创建三台虚拟机
在所有节点上设置
#vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.30.21 ceph01
192.168.30.22 ceph02
192.168.30.23 ceph03
所有节点
生成密钥文件
#ssh-keygen -t rsa \一直按enter
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:68+bD/U2lCCZOE+Z1MbfQ4HOKun2c8ACk+hGMPTdNFs root@ceph01
The key's randomart image is:
+---[RSA 2048]----+
| .. o.E ...|
| o. . oo+=+. . |
| o...+oB.+... |
| o + + . +.o.|
| o S + o o .|
| o = = o |
| . o + . + |
| . + + o . |
| o.*++ |
+----[SHA256]-----+
所有节点中执行
#ssh-copy-id ceph01
#ssh-copy-id ceph02
#ssh-copy-id ceph03
在所有节点中创建文件夹
[root@ceph01 ~]# mkdir -p /export/data
[root@ceph01 ~]# mkdir -p /export/servers
[root@ceph01 ~]# mkdir -p /export/sofrware
在software中传入jdk和hadoop的安装包并解压jdk的安装包
重命名jdk文件夹
root@ceph01 sofrware]# ls
hadoop-2.7.4.tar.gz jdk1.8.0_341 jdk-8u341-linux-x64.tar.gz
[root@ceph01 sofrware]# mv jdk1.8.0_341 jdk
[root@ceph01 sofrware]# ls
hadoop-2.7.4.tar.gz jdk jdk-8u341-linux-x64.tar.gz
在环境变量中加入如下内容
[root@ceph01 sofrware]#vim /etc/profile
#Jdk
export JAVA_HOME=/export/sofrware/jdk
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
重启环境变量
[root@ceph01 ~]# source /etc/profile
[root@ceph01 ~]# java -version
java version "1.8.0_341"
Java(TM) SE Runtime Environment (build 1.8.0_341-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.341-b10, mixed mode)
即代表jdk安装配置完成
解压hadoop安装包
[root@ceph01 sofrware]# tar -zxvf hadoop-2.7.4.tar.gz
配置环境变量
[root@ceph01 sofrware]# vim /etc/profile
#hadoop
export HADOOP_HOME=/export/sofrware/hadoop-2.7.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[root@ceph01 sofrware]# source /etc/profile
[root@ceph01 sofrware]# hadoop version
Hadoop 2.7.4
Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r cd915e1e8d9d0131462a0b7301586c175728a282
Compiled by kshvachk on 2017-08-01T00:29Z
Compiled with protoc 2.5.0
From source with checksum 50b0468318b4ce9bd24dc467b7ce1148
This command was run using /export/sofrware/hadoop-2.7.4/share/hadoop/common/hadoop-common-2.7.4.jar
即代表hadoop安装完成
主节点
进入配置目录
[root@ceph01 ~]# cd /export/sofrware/hadoop-2.7.4/etc/hadoop/
[root@ceph01 hadoop]# vim hadoop-env.sh
export JAVA_HOME=/export/sofrware/jdk
[root@ceph01 hadoop]# vim core-site.xml
[root@ceph01 hadoop]# vim hdfs-site.xml
[root@ceph01 hadoop]# vim mapred-site.xml
~
[root@ceph01 hadoop]# vim yarn-site.xml
[root@ceph01 hadoop]# vim slaves
ceph01
ceph02
ceph03
在主节点配置完毕之后将文件传给其余节点
[root@ceph01 hadoop]# scp -r /export/ ceph02:/
[root@ceph01 hadoop]# scp -r /export/ ceph03:/
在主节点进行格式化，若出现下面successfully formatted则表示成功
[root@ceph01 ~]# hdfs namenode -format
22/07/21 14:38:45 INFO common.Storage: Storage directory /export/sofrware/hadoop-2.7.4/tmp/dfs/name has been successfully formatted.
在所有节点启动hadoop服务
[root@ceph01 ~]# start-all.sh
查看服务启动情况
[root@ceph01 ~]# jps
2225 DataNode
3586 Jps
2132 NameNode
2678 NodeManager
2573 ResourceManager
在windows中修改C:WindowsSystem32driversetc下的hosts文件，在文件的最后加入节点的ip地址以及节点的名字
192.168.30.21 ceph01
192.168.30.22 ceph02
192.168.30.23 ceph03
查看hdfs集群状态，打开谷歌浏览器，输入ceph01:50070
查看yarn集群状态,打开谷歌浏览器，输入ceph01:8088
至此，hadoop集群已经搭建完成
