Hadoop集群分布式安装

先创建三台linux的虚拟机先创建三台linux的虚拟机+先删除linux自带的，在安装自己环江所需要的jdk

命令：rpm -qa | grep -i java | xargs -n1 rpm -e --nodeps

rpm===>Linux管理包的工具
qa===>使用询问模式查询所有套件
grep -i===>输入参数,忽略大小写
xargs -n1===>命令传递参数,一次执行一个命令
pm -e===>删除指定的套件
nodeps===>不验证套件档的相互关联性

集群规划
主机 角色
node1 NN DN RM NM
node2 SNN DN NM
node3 DN NM

基础环境
3台机器都需要操作
# 主机名
cat /etc/hostname

# hosts映射
vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.88.151 node1.itcast.cn node1
192.168.88.152 node2.itcast.cn node2
192.168.88.153 node3.itcast.cn node3

# JDK 1.8安装  上传 jdk-8u241-linux-x64.tar.gz到/export/server/目录下
cd /export/server/
tar zxvf jdk-8u241-linux-x64.tar.gz

	#配置环境变量
	vim /etc/profile
	
	export JAVA_HOME=/export/server/jdk1.8.0_241
	export PATH=$PATH:$JAVA_HOME/bin
	export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
	
	#重新加载环境变量文件
	source /etc/profile

# 集群时间同步
ntpdate ntp5.aliyun.com（只要是新建虚拟机时选择的时区相同，并且联网，这条语句并不重要）

# 防火墙关闭
firewall-cmd --state	#查看防火墙状态
systemctl stop firewalld.service  #停止firewalld服务
systemctl disable firewalld.service  #开机禁用firewalld服务

# ssh免密登录（只需要配置node1至node1、node2、node3即可）

	#node1生成公钥私钥 (一路回车)
	ssh-keygen  
	
	#node1配置免密登录到node1 node2 node3
	ssh-copy-id node1
	ssh-copy-id node2
	ssh-copy-id node3

上传Hadoop安装包到node1 /export/server

hadoop-3.3.0-Centos7-64-with-snappy.tar.gz
tar zxvf hadoop-3.3.0-Centos7-64-with-snappy.tar.gz

修改配置文件(配置文件路径 hadoop-3.3.0/etc/hadoop)

hadoop-env.sh

#文件最后添加
export JAVA_HOME=/export/server/jdk1.8.0_241

    export HDFS_NAMENODE_USER=root
    export HDFS_DATANODE_USER=root
    export HDFS_SECONDARYNAMENODE_USER=root
    export YARN_RESOURCEMANAGER_USER=root
    export YARN_NODEMANAGER_USER=root

core-site.xml

  
    
        fs.defaultFS
        hdfs://node1:8020
    
    
    
    
        hadoop.tmp.dir
        /export/data/hadoop-3.3.0
    
    
    
    
        hadoop.http.staticuser.user
        root
    
    
    
    
        hadoop.proxyuser.root.hosts
        *
    
    
    
        hadoop.proxyuser.root.groups
        *
    
    
    
    
        fs.trash.interval
        1440
    

hdfs-site.xml

 
 
     dfs.namenode.secondary.http-address
     node2:9868
 

mapred-site.xml

  
  
    mapreduce.framework.name
    yarn
  
  
  
  
    mapreduce.jobhistory.address
    node1:10020
  
   
  
  
    mapreduce.jobhistory.webapp.address
    node1:19888
  
  
  
    yarn.app.mapreduce.am.env
    HADOOP_MAPRED_HOME=${HADOOP_HOME}
  
  
  
    mapreduce.map.env
    HADOOP_MAPRED_HOME=${HADOOP_HOME}
  
  
  
    mapreduce.reduce.env
    HADOOP_MAPRED_HOME=${HADOOP_HOME}
  

yarn-site.xml

    
    	yarn.resourcemanager.hostname
    	node1
    
    
    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    
    
    
    
        yarn.nodemanager.pmem-check-enabled
        false
    
    
    
    
        yarn.nodemanager.vmem-check-enabled
        false
    
    
    
    
      yarn.log-aggregation-enable
      true
    
    
    
    
        yarn.log.server.url
        http://node1:19888/jobhistory/logs
    
    
    
    
      yarn.log-aggregation.retain-seconds
      604800
    

workers

node1[.itcast.cn]
node2[.itcast.cn]
node3[.itcast.cn]

分发同步hadoop安装包

cd /export/server

  scp -r hadoop-3.3.0 root@node2:$PWD
  scp -r hadoop-3.3.0 root@node3:$PWD

将hadoop添加到环境变量（3台机器）
vim /etc/profile

export HADOOP_HOME=/export/server/hadoop-3.3.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile


#别忘了scp给其他两台机器哦

Hadoop集群启动

（首次启动）格式化namenode

hdfs namenode -format

脚本一键启动

```[root@node1 ~]# start-dfs.sh 
Starting namenodes on [node1]
Last login: Thu Nov  5 10:44:10 CST 2020 on pts/0
Starting datanodes
Last login: Thu Nov  5 10:45:02 CST 2020 on pts/0
Starting secondary namenodes [node2]
Last login: Thu Nov  5 10:45:04 CST 2020 on pts/0

[root@node1 ~]# start-yarn.sh 
Starting resourcemanager
Last login: Thu Nov  5 10:45:08 CST 2020 on pts/0
Starting nodemanagers
Last login: Thu Nov  5 10:45:44 CST 2020 on pts/0

Web UI页面

HDFS集群：http://node1:9870/YARN集群：http://node1:8088/

同时需要添加虚拟机在本地主机的主机映射，以便可以通过别名访问web ui界面

错误1:运行hadoop3官方自带mr示例出错。

错误信息

 
 Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
 
   yarn.app.mapreduce.am.env
   HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
 
 
   mapreduce.map.env
   HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
 
 
   mapreduce.reduce.env
   HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
 

解决 mapred-site.xml,增加以下配置

   yarn.app.mapreduce.am.env
   HADOOP_MAPRED_HOME=${HADOOP_HOME}
 
 
   mapreduce.map.env
   HADOOP_MAPRED_HOME=${HADOOP_HOME}
 
 
   mapreduce.reduce.env
   HADOOP_MAPRED_HOME=${HADOOP_HOME}