大数据工程实践v1.0版本 | （一）环境搭建 (1) | 统一配置hadoop3.1.3完全分布式集群三个节点

文章目录

1. 运行环境
2. 各节点角色分配
3. 集群配置清单
4. 具体配置
- 4.1 hadoop-env.sh
- 4.2 hdfs-site.xml
- 4.3 mapred-site.xml
- 4.4 yarn-site.xml
- 4.5 yarn-site.xml
- 4.6 core-site.xml
- 4.7 workers

1. 运行环境 2. 各节点角色分配

服务名称	子服务	服务器	服务器	服务器
		hadoop101	hadoop102	hadoop103
HDFS	NameNode	√
	DataNode	√	√	√
	SecondaryNameNode			√
Yarn	NodeManager	√	√	√
Yarn	ResourceManager		√

3. 集群配置清单

以下配置文件均在hadoop根目录下的 /etc/hadoop/ 内

配置文件	作用描述
hadoop-env.sh	配置hadoop运行所需的JDK环境变量
hdfs-site.xml	配置HDFS中NameNode和DataNode的保存目录，以及后者的存储块副本数
mapred-site.xml	配置mapreduce的运行框架，默认为local
yarn-site.xml	指定yarn的ResourceManager节点，设置yarn运行的日志保存目录及保存时间
core-site.xml	设置hadoop缓存目录以及默认的HDFS地址
workers	设置集群里所有节点的主机名

4. 具体配置

4.1 hadoop-env.sh

export JAVA_HOME=/opt/java/jdk8

4.2 hdfs-site.xml


    
            dfs.namenode.name.dir
            file:/opt/module/hadoop/namenode_dir
    

    
            dfs.datanode.data.dir
            file:/opt/module/hadoop/datanode_dir
    
    
    
            dfs.namenode.http-address
            hadoop101:9870
    
    
    
            dfs.namenode.secondary.http-address
            hadoop103:9868
    
    
            dfs.replication
            3

4.3 mapred-site.xml


    
        mapreduce.framework.name
        yarn
    
    
        mapreduce.jobhistory.address
        hadoop101:10020
    
    
        mapreduce.jobhistory.webapp.address
        hadoop101:19888

4.4 yarn-site.xml


    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
     
        
        yarn.resourcemanager.hostname
        hadoop102
     
    
      	yarn.nodemanager.env-whitelist 			      
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    
    
    
        yarn.log-aggregation-enable
        true
    

    
    
        yarn.log.server.url
        http://hadoop101:19888/jobhistory/logs
    

    
    
        yarn.log-aggregation.retain-seconds
        1209600

4.5 yarn-site.xml


    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
     
        
        yarn.resourcemanager.hostname
        hadoop102
     
    
      	yarn.nodemanager.env-whitelist 			      
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    
    
    
        yarn.log-aggregation-enable
        true
    

    
    
        yarn.log.server.url
        http://hadoop101:19888/jobhistory/logs
    

    
    
        yarn.log-aggregation.retain-seconds
        1209600

4.6 core-site.xml


    
            hadoop.tmp.dir
            file:/opt/module/hadoop/tmp
            Abase for other temporary directories.
     

     
            fs.defaultFS
            hdfs://hadoop101:9000

4.7 workers

hadoop101
hadoop102
hadoop103

大数据工程实践v1.0版本 | （一）环境搭建 (1) | 统一配置hadoop3.1.3完全分布式集群三个节点

大数据系统相关栏目本月热门文章