Spark官网下载
https://spark.apache.org/downloads.html
Hdoop官网下载
https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz
JDK下载地址
https://www.oracle.com/java/technologies/downloads/#java8
百度云链接
链接:https://pan.baidu.com/s/1SbvMo7FeeZ50YiEERMsI4g
提取码:qwer
环境:Centos7.6(腾讯云)
-
Jdk安装
# 卸载本机openjdk环境 yum list installed | grep java yum -y remove java-1.7/8.0-openjdk* # 安装JDK1.8 tar -zxvf jdk-8u333-linux-x64.tar.gz # --> /opt/software/jdk1.8.0_333 # 添加配置 vim /etc/profile export JAVA_HOME=/opt/software/jdk1.8.0_333 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH # 使配置生效 source /etc/profile -
Hdoop安装
# 解压 tar -zxvf hadoop-3.2.0.tar.gz # --> /opt/software/hadoop-3.2.0 cd hadoop-3.2.0 ll ''' drwxr-xr-x 2 1001 1002 4096 Jan 8 2019 bin # 可执行二进制文件 hdfs的客户端操作命令 drwxr-xr-x 3 1001 1002 4096 Jan 8 2019 etc # hadoop配置文件 drwxr-xr-x 2 1001 1002 4096 Jan 8 2019 include drwxr-xr-x 3 1001 1002 4096 Jan 8 2019 lib drwxr-xr-x 4 1001 1002 4096 Jan 8 2019 libexec -rw-rw-r-- 1 1001 1002 150569 Oct 19 2018 LICENSE.txt -rw-rw-r-- 1 1001 1002 22125 Oct 19 2018 NOTICE.txt -rw-rw-r-- 1 1001 1002 1361 Oct 19 2018 README.txt drwxr-xr-x 3 1001 1002 4096 Jan 8 2019 sbin # hdfs yarn 组件启停命令 drwxr-xr-x 4 1001 1002 4096 Jan 8 2019 share # doc说明文档、jar包 ''' # 配置环境变量 vim /etc/profile export HADOOP_HOME=/opt/software/hadoop-3.2.0 export PATH=${HADOOP_HOME}/bin:$PATH # 使配置生效 source /etc/profile # 修改hdoop配置信息 # 进入 /opt/software/hadoop-3.2.0/etc/hadoop/ 目录下,修改文件以下配置 # 1.hadoop-env.sh vim ./hadoop-env.sh export JAVA_HOME=/opt/software/jdk1.8.0_333/ # 2.core-site.xml vim ./core-site.xml ''' ''' # 3.hdfs-site.xml 指定副本系数和临时文件存储位置 vim ./hdfs-site.xml '''fs.defaultFS hdfs://hadoop001:8020 hadoop.tmp.dir /tmp/hadoop/tmp ''' # 4.slaves 置所有从属节点的主机名或 IP 地址,由于是单机版本,所以指定本机即可: # 关闭防火墙,不关闭防火墙可能导致无法访问 Hadoop 的 Web UI 界面: systemctl status firewalld # 查看防火墙 systemctl status firewalld # 关闭防火墙 # 启动前,hadoop初始化 cd /opt/software/hadoop-3.2.0/bin ./hdfs namenode -format # 启动hdfs cd /opt/software/hadoop-3.2.0/sbin ./start-dfs.sh # 验证是否启动成功 # 方式1:>> jps # 方式2:查看web UI界面,端口9870(hadoop2 是50070) # 浏览器输入 xx.xx.xx.xx:9870,正常情况下能看到问题3解决后页面,是hadoop服务的overview # Hadoop(YARN)环境搭建 # 进入/opt/software/hadoop-3.2.0/etc/hadoop/目录下,修改以下配置 # 1. mapred-site.xml vim ./mapred-site.xml '''dfs.replication 1 ''' # 2. yarn-site.xml vim ./yarn-site.xml '''mapreduce.framework.name yarn ''' # 启动yarn服务 cd /opt/software/hadoop-3.2.0/sbin ./start-yarn.sh # 验证yarn服务是否启动成功 # 方式一:>> jps # 方式二:查看yarn Web UI界面,端口号为8088yarn.nodemanager.aux-services mapreduce_shuffle yarn Web界面:
hadoop 安装过程中报错,及解决方案
- 启动时,报错
解决:
vim /etc/profile.d/my_env.sh # 添加缺失变量 export HDFS_NAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root # 使环境变量生效 source /etc/profile
- 启动时,报错
解决:
# 设置ssh免密登录(单机部署hdoop,也需要配置ssh) ssh-keygen -t rsa cd ~/.ssh cat id_rsa.pub >> authorized_keys chmod 600 authorized_keys
-
远程访问,xx.xx.xx.xx:9870 拒绝访问
解决:登录腾讯云控制台,开放9870端口访问
-
无法通过hdfs web管理界面上传文件
“Permission denied: user=dr.who, access=WRITE, inode=“/”:root:supergroup:drwxr-xr-x”
解决:
# 第一种方式可以直接修改dfs文件目录权限(待定) vim core-site.xml
hdfs dfs -chmod -R 755 / # 第二种方式,core-site.xml配置里面关闭权限认证(待定) vim core-site.xmlhadoop.http.staticuser.user root # 重启hdfs服务和yarn服务 cd /opt/software/hadoop-3.2.0/sbin ./stop-dfs.sh ./stop-yarn.sh ./start-dfs.sh ./start-yarn.shdfs.permissions.enabled false
- 启动时,报错
-
Spark安装
# 解压 tar -xzvf ./spark-3.1.3-bin-hadoop3.2.tgz # --》/opt/software/spark-3.1.3-bin-hadoop3.2 # 配置环境变量 vim /etc/profile ''' export SPARK_HOME=/opt/software/spark-3.1.3-bin-hadoop3.2 export PATH=${SPARK_HOME}/bin:$PATH ''' # 使配置生效 source /etc/profile # 启动Spark spark-shell --master local[2]



