MAC PRO OS 安装 Hadoop 3.3.1 [防止踩坑]

先安装好Homebrew

苹果电脑 常规安装脚本（推荐 完全体 几分钟安装完成）：
/bin/zsh -c "$(curl -fsSL https://gitee.com/cunkai/HomebrewCN/raw/master/Homebrew.sh)

苹果电脑 极速安装脚本（精简版 几秒钟安装完成）：
/bin/zsh -c "$(curl -fsSL https://gitee.com/cunkai/HomebrewCN/raw/master/Homebrew.sh)" speed

其次装好JAVA

brew install java

安装maven和gmp

brew install maven
brew install gmp

设置ssh免密登录[后续执行start-dfs.sh需要SSH验证 为了简单起见不设密码]

ssh-keygen -t rsa一直回车后
cd ~/.ssh

将公钥写入~/.ssh/authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

打开共享

进行测试
ssh localhost

安装Hadoop

brew install hadoop
为了保证网络通畅，请科学上网！用时大概2-3小时…

找到自己的JAVA_HOME路径

我的在usr/libexec/java_home
执行/usr/libexec/java_home即可得到路径
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home
如有像我的Internet Plug-Ins中间有空格的请注意，加一斜杠转义
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home

配置Hadoop文件

1.修改存储hadoop元数据文档
vim /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop/core-site.xml

  
     hadoop.tmp.dir
     file:/opt/homebrew/Cellar/hadoop/3.3.1/libexec/tmp
  
  
     fs.defaultFS
     hdfs://localhost:8020
  

2.修改hdfs-site.xml文档
vim /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop/hdfs-site.xml

    
         dfs.replication
         1
    
    
         dfs.namenode.name.dir
         file:/opt/homebrew/Cellar/hadoop/3.3.1/libexec/tmp/dfs/name
    
    
         dfs.datanode.data.dir
         file:/opt/homebrew/Cellar/hadoop/3.3.1/libexec/tmp/dfs/data
    

3.设置Hadoop环境变量
vim ~/.bash_profile

export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.3.1/libexec
export HADOOP_COMMON_HOME=$HADOOP_HOME
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:/opt/homebrew/Cellar/scala/bin

执行source ~/.bash_profile更新环境变量。

4.配置yarn文件yarn-site.xml
vim /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop/yarn-site.xml

  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
  
  
    yarn.nodemanager.env-whitelist
    JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
  
  
    yarn.resourcemanager.hostname
    localhost
  
  
    yarn.nodemanager.resource.memory-mb
    20480
  
  
    yarn.scheduler.minimum-allocation-mb
    2048
  
  
    yarn.nodemanager.vmem-pmem-ratio
    2.1
  

4.配置yarn文件yarn-site.xml
vim /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop/yarn-site.xml

  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
  
  
    yarn.nodemanager.env-whitelist
    JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
  
  
    yarn.resourcemanager.hostname
    localhost
  
  
    yarn.nodemanager.resource.memory-mb
    20480
  
  
    yarn.scheduler.minimum-allocation-mb
    2048
  
  
    yarn.nodemanager.vmem-pmem-ratio
    2.1
  

5.配置yarn文件mapred-site.xml
vim /opt/homebrew/Cellar/hadoop/3.3.1/libexec/etc/hadoop/mapred-site.xml

	 
	  mapreduce.framework.name
	  yarn
	

MAC⚠️还需配置如下：
6.cp lib/native库 至 libexec目录
这样作为java为执行语言的hadoop就有了编译器
否则在start-dfs的时候会出现报错
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable。

解决办法：

1. 先下载lib/native
2. 然后copy这个文件夹至/usr/local/Cellar/hadoop/3.1.1/libexec
3. 配置 /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hadoop-env.sh

export JAVA_HOME=你的java_home路径（见上面：找自己的JAVA_HOME路径的方式）
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native" 

start Hadoop

依次执行以start namenode
cd /opt/homebrew/Cellar/hadoop/3.3.1/bin
/hdfs namenode -format

start:
cd /opt/homebrew/Cellar/hadoop/3.3.1/sbin
./start-dfs.sh

网页查看是否成功
http://localhost:9870/dfshealth.html#tab-overview

stop:
./stop-all.sh

start yarn

依次执行
cd /opt/homebrew/Cellar/hadoop/3.3.1/sbin
./start-yarn.sh

网页查看是否成功
http://localhost:8088/cluster

文件查看与管理

如果想要具体的可视化工具可安装HUE

详见：https://zhuanlan.zhihu.com/p/135331195

其它

在此感谢如下文章：
Mac安装Hadoop
https://blog.csdn.net/weixin_44864748/article/details/121066700
Mac的Hadoop开发环境搭建
https://www.jianshu.com/p/3f1e0a93c679
Warning: Permanently added the ECDSA host key
https://blog.csdn.net/qq_16069927/article/details/88963530
Mac修改环境变量路径中带空格的处理
https://blog.csdn.net/taoerchun/article/details/110938758
Mac安装Hadoop
https://blog.csdn.net/vbirdbest/article/details/88189753

如需转载请把以上感谢的文章加上