完全分布式hadoop3.1.3集群搭建

一、搭建集群

1.1 架构信息1.2 配置文件1.3 修改自定义配置文件1.4 分发配置文件 二、启动hadoop集群

各组件和机器间的对应关系如下表

原则上是NameNode和SecondaryNameNode不在一台机器上；而且ResourceManager消耗资源比较高，放在资源相对充足的hadoop101上。

3台机器上分别安装hadoop3.1.3，注意：首先要安装JDK3台机器间配置免密登录(方便分发文件)，创建一个分发文件的脚本xsync。这一步可以省略。

默认配置文件

core-default.xmlhdfs-default.xmlyarn-default.xmlmapred-default.xml

官方建议不要在默认配置文件中修改，可以复制他的内容到自定义配置文件中，然后修改值。

自定义配置文件自定义的配置文件都在hadoop-3.1.3/etc/hadoop/目录下

core-site.xmlhdfs-site.xmlyarn-site.xmlmapred-site.xml 1.3 修改自定义配置文件

core-site.xml

        
          fs.defaultFS
          hdfs://hadoop100:8020
          The name of the default file system.  A URI whose
          scheme and authority determine the FileSystem implementation.  The
          uri's scheme determines the config property (fs.SCHEME.impl) naming
          the FileSystem implementation class.  The uri's authority is used to
          determine the host, port, etc. for a filesystem.
        
        
          hadoop.tmp.dir
          /usr/local/hadoop-3.1.3/data
          A base for other temporary directories.
        
        

修改如下：

2. hdfs-site.xml

        
          dfs.namenode.http-address
          hadoop100:9870
          
            The address and the base port where the dfs namenode web ui will listen on.
          
        
        
          dfs.namenode.secondary.http-address
          hadoop102:9868
          
            The secondary namenode http server address and port.
          
        

3. yarn-site.xml

  
    A comma separated list of services where service name should only
      contain a-zA-Z0-9_ and can not start with numbers
    yarn.nodemanager.aux-services
    mapreduce_shuffle
    
  
  
    The hostname of the RM.
    yarn.resourcemanager.hostname
    hadoop101
  
  
    Environment variables that containers may override rather than use NodeManager's default.
    yarn.nodemanager.env-whitelist
    JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
  

4. mapred-site.xml

        
          mapreduce.framework.name
          yarn
          The runtime framework for executing MapReduce jobs.
          Can be one of local, classic or yarn.
          
        

分发配置文件到另外两台机器上这里使用上节配置的分发脚本xsync，如果不使用xsync，则可以手动复制4个文件到另外两台机器上。

cd /usr/local/hadoop-3.1.3/etc/hadoop/
xsync core-site.xml hdfs-site.xml yarn-site.xml mapred-site.xml

。。。后续明天再写