centos7安装hadoop(hadoop完全分布式安装)

注意hadoop只支持jdk7和jdk8，不支持jdk11

java -version
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)

生成密钥对，可一路回车，之后会在对应用户下生成一个ssh文件夹

ssh-keygen -t rsa

进入.ssh目录下

ssh-copy-id -i ~/.ssh/id_rsa.pub root@localhost

通过ftp等工具将安装包上传到服务器上，并解压到/usr/local/目录

cd /usr/local/
wget https://dlcdn.apache.org/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz
tar -zxvf hadoop-2.10.1.tar.gz
mv hadoop-2.10.1/ hadoop

配置环境

vim /etc/profile

加入以下配置

# hadoop环境配置
export HADOOP_HOME=/usr/local/hadoop
export PATH=.:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

使配置文件生效

source /etc/profile

通过hadoop version测试

hadoop version
Hadoop 2.10.1
Subversion https://github.com/apache/hadoop -r 1827467c9a56f133025f28557bfc2c562d78e816
Compiled by centos on 2020-09-14T13:17Z
Compiled with protoc 2.5.0

修改hadoop-2.10.1/etc/hadoop/下的配置文件

cd hadoop-2.10.1/etc/hadoop/

1.修改hadoop-env.sh中的java路径

vi hadoop-env.sh

 2.hdfs-site.xml中添加配置

            
        dfs.replication
        1
    

3.core-site.xml中添加配置

    
        fs.defaultFS
        hdfs://localhost:9000
    

4.复制mapred-site.xml并添加配置

cp mapred-site.xml.template mapred-site.xml             //复制YARN配置文件模板
vi mapred-site.xml

    
        mapreduce.framework.name
        yarn
    

5.配置yarn-site.xml

        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
        
                yarn.nodemanager.aux-services.mapreduce.shuffle.class
                org.apache.hadoop.mapred.ShuffleHandler
        

格式化hadoop

hadoop namenode -format

start-dfs.sh 
start-yarn.sh

或者

start-all.sh 

通过jps可以查看已经开启的进程

 通过对应的地址端口号（默认为8088和50070）可以查看ui界面

stop-all.sh