hadoop伪分布式和完全分布式的区别(伪分布式hadoop安装)

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

修改hostname

vim /etc/hostname

配置映射

vim /etc/hosts

配置静态ip

vim /etc/sysconfig/network-scripts/ifcfg-ens33 

配置免密登录

ssh hadoop100
cd .ssh
ssh-keygen -t rsa
ssh-copy-id hadoop100

安装jdk和hadoop

其他博客有介绍如何安装，这里就不过多介绍

配置环境变量

#java
export JAVA_HOME=/export/servers/jdk
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

#hadoop
export HADOOP_HOME=/export/servers/hadoop
export PATH=$PATH:$HADOOP_HOME/bin

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

刷新环境变量

source /etc/profile

测试jdk和hadoop是否安装完成

[root@hadoop100 profile.d]# java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
[root@hadoop100 profile.d]# hadoop version
Hadoop 3.1.3
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579
Compiled by ztang on 2019-09-12T02:47Z
Compiled with protoc 2.5.0
From source with checksum ec785077c385118ac91aadde5ec9799
This command was run using /export/servers/hadoop/share/hadoop/common/hadoop-common-3.1.3.jar

修改hadoop的配置文件
修改core-site.xml

    
        fs.defaultFS
        hdfs://hadoop100:8020
    

    
    
        hadoop.tmp.dir
        /export/servers/hadoop/data
    

    
    
        hadoop.http.staticuser.user
        root
    

修改yarn-site.xml

    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    

    
    
        yarn.resourcemanager.hostname
        hadoop100
    

    
    
        yarn.nodemanager.env-whitelist
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    

	
	
	    yarn.log-aggregation-enable
	    true
	
	
	  
	    yarn.log.server.url  
	    http://hadoop100:19888/jobhistory/logs
	
	
	
	    yarn.log-aggregation.retain-seconds
	    604800
	

配置hdfs-site.xml

	
        dfs.namenode.http-address
        hadoop100:9870
    
	
    
        dfs.namenode.secondary.http-address
        hadoop100:9868
    

配置mapred-site.xml

	
	
	    mapreduce.framework.name
	    yarn
	
	
	
	    mapreduce.jobhistory.address
	    hadoop100:10020
	
	
	
	    mapreduce.jobhistory.webapp.address
	    hadoop100:19888
	

配置workers

hadoop100

格式化namenode

hadoop namenode -format

启动hdfs

sbin/start-dfs.sh

启动yarn

sbin/start-yarn.sh

启动历史服务器

mapred --daemon start historyserver