contOS配置hadoop
1.vi /etc/sysconfig/network-scripts/ifcfg-ens33进入配置网络编辑分配IP
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=ens33
UUID=af99d9a2-66ad-4f73-a71a-063a1badb02d
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.47.10
GATEWAY=192.168.47.2
NETMASK=255.255.255.0
DNS1=192.168.47.2
- 解压文件的命令tar -zxvf /root/mzr/hadoop-2.8.4.tar.gz -C /usr/local
2.修改虚拟机名称和ip映射
- 修改机器名称vi /etc/hostname，修改完重启虚拟机使用reboot
- 修改ip映射：vi /etc/hosts
192.168.245.111 6110master
192.168.245.112 6110slave0
192.168.245.113 6110slave1
3.配置 jdk
- 配置jdk解压到/usr/local目录，进入配置文件vi /etc/profile配置完成之后重启环境source /etc/profile
#jdk的配置文件 java env
export JAVA_HOME=/usr/local/jdk1.8.0_151
export JRE_HOME=/usr/local/jdk1.8.0_151/jre
export PATH=$PATH:/usr/local/jdk1.8.0_151/bin
export CLASSPATH=./:/usr/local/jdk1.8.0_151/lib:/usr/local/jdk1.8.0_151/jre/lib
- 检验是否配置成功的命令java,java -version，javac,
# 配置成功的标识
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)
4.配置hadoop
- 把文件解压至/usr/local目录下
- 配置环境，使用命令进入vi /etc/profile配置
# 配置hadoop环境 Hadoop env
export HADOOP_HOME=/usr/local/hadoop-2.8.4/
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
- 重启环境使用命令source /etc/profile
- 检验Hadoop是否安装成功使用命令which hadoop,hadoop version
# 成功标识
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /usr/local/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar
5.Hadoop 分布式配置
1.需要配置的文件名称，进入目录cd /usr/local/hadoop-3.3.1/etc/hadoop/
# 目录内的文件
[root@6274master hadoop]# ls
capacity-scheduler.xml hadoop-policy.xml kms-acls.xml mapred-queues.xml.template workers
configuration.xsl hadoop-user-functions.sh.example kms-env.sh mapred-site.xml yarn-env.cmd
container-executor.cfg hdfs-rbf-site.xml kms-log4j.properties shellprofile.d yarn-env.sh
core-site.xml hdfs-site.xml kms-site.xml slaves yarnservice-log4j.properties
hadoop-env.cmd httpfs-env.sh log4j.properties ssl-client.xml.example yarn-site.xml
hadoop-env.sh httpfs-log4j.properties mapred-env.cmd ssl-server.xml.example
hadoop-metrics2.properties httpfs-site.xml mapred-env.sh user_ec_policies.xml.template
工作过程：
1、通过安装、克隆方式逐步配置好3台Centos7 64位操作系统的虚拟机；
2、安装好jdk；
3、安装配置SSH；
4、配置hadoop-env.sh；
5、配置hadoop-core-site.xm；
6、配置hadoop-hdfs-site.xm；
7、配置hadoop-mapred-site.xml；
8、配置hadoop-yarn-site.xml；
9、配置slave；
10、分发到所有虚拟机上；
11、测试并运行程序。
2.配置 vi hadoop-env.sh
export JAVA_HOME=/usr/local/ jdk-15.0.2
export HADOOP_CONF_DIR=/usr/local/hadoop-3.2.2/etc/hadoop/
3、配置hadoop-core-site.xml
fs.defaultFS
hdfs://6274master:9000
io.file.buffer.size
4096
hadoop.tmp.dir
/home/bigdata/tmp
4、配置 hadoop-hdfs-site.xml
dfs.replication
3
dfs.block.size
134217728
dfs.namenode.name.dir
file:///home/hadoopdata/dfs/name
dfs.datanode.data.dir
/home/hadoopdata/dfs/data
fs.checkpoint.dir
/home/hadoopdata/checkpoint/dfs/6274name
dfs.http.address
6274master:50070
dfs.secondary.http.address
6274master:50090
dfs.webhdfs.enabled
true
dfs.permissions
false
5、配置hadoop-mapred-site.xml
mapreduce.framework.name
yarn
true
mapreduce.jobhistory.address
6274master:10020
mpareduce.jobhistory.webapp.address
6274master:19888
6、配置 hadoop-yarn-site.xml
yarn.resourcemanager.hostname
6274master
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resourcemanager.address
6274master:8032
yarn.resourcemanager.scheduler.address
6274master:8030
yarn.resourcemanager.resource-tracker.address
6274master:8031
yarn.resourcemanager.admin.address
6274master:8033
yarn.resourcemanager.webapp.address
6274master:8088
7、配置vi slave
6274master
6274slave0
6274slave1
8.完成分发任务
vi /etc/hosts
192.168.245.111 6110master
192.168.245.112 6110slave0
192.168.245.113 6110slave1
6110slave0: rm -rf /usr/local/hadoop-2.8.4/
6110slave1: rm -rf /usr/local/hadoop-2.8.4/
- 完成分发工作,首先ping 虚拟机名称是否ping的通，不通则不能分发任务：
6110master: scp -r /usr/local/hadoop-2.8.4/ 6110slave0:/usr/local/
6110master: scp -r /usr/local/hadoop-2.8.4/ 6110slave1:/usr/local/
9.启动之前要先在namenode服务器上格式化，只需一次。
hadoop namenode –format
启动三种方式：
start-all.sh
start-dfs.sh
start-yarn.sh
hadoop-daemon.sh start namenode
hadoop-daemons.sh start datanode
yarn-daemon.sh start namenode
yarn-daemons.sh start datanode
mr-jobhistory-daemon.sh start historyserver
- 1、 查看进程是否启动了：jps 查看对应模块的web
http://192.168.47.10:50070
http://192.168.47.10:8088
Hdfs dfs –ls /
Hdfs dfs –put ./*** /
yarn jar $HADOOP_HOME/share/hadoop/mapreduce/Hadoop-mapreduce-examples-2.7.1.jar wordcount /*** /out/01
hdfs dfs –ls /out/01
hdfs dfs –cat /out/01/****
