学习尚硅谷的azkaban课程,教程视频里用的mysql5.x,自己机器装的8.x,导致启动executor连接不上,所以改着试试,让他兼容mysql8.x。参考电商数仓项目(五) azkaban安装、配置和使用
2.具体步骤 2.1 前期环境准备 2.1.1 所有服务器安装git,gcc (root) 编译要用么??yum install -y git gcc*2.1.2 下载安装 gradle 4.6 (/u02存zip包,/u01存软件)
wget https://downloads.gradle-dn.com/distributions/gradle-4.6-bin.zip
unzip gradle-4.6-bin.zip
mv gradle-4.6 /u01/gradle-4.6
cd /u01/gradle-4.6
mkdir repo user_home ## 本地库??
cd init.d
vi init.gradle
[root@hadoop102 init.d]# cat init.gradle
allprojects {
repositories {
maven { url 'file:/u01/gradle-4.6/repo'}
mavenLocal()
maven { name "Alibaba" ; url "https://maven.aliyun.com/repository/public" }
maven { name "Bstek" ; url "http://nexus.bsdn.org/content/groups/public/" }
mavenCentral()
}
buildscript {
repositories {
maven { name "Alibaba" ; url 'https://maven.aliyun.com/repository/public' }
maven { name "Bstek" ; url 'http://nexus.bsdn.org/content/groups/public/' }
maven { name "M2" ; url 'https://plugins.gradle.org/m2/' }
}
}
}
2.1.3 配置gradle环境变量
## 添加到自己的环境变量里 export GRADLE_HOME=/u01/gradle-4.6 export GRADLE_USER_HOME=/u01/gradle-4.6/user_home export PATH=$PATH:$GRADLE_HOME/bin [root@hadoop102 profile.d]# cat my_env.sh #JAVA_HOME export JAVA_HOME=/opt/module/jdk1.8.0_212 export PATH=$PATH:$JAVA_HOME/bin #HADOOP_HOME export HADOOP_HOME=/opt/module/hadoop-3.1.3 export PATH=$PATH:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin #HIVE_HOME export HIVE_HOME=/opt/module/hive-3.1.2 export PATH=$PATH:$HIVE_HOME/bin #KAFKA_HOME export KAFKA_HOME=/opt/module/kafka-3.0.0 export PATH=$PATH:$KAFKA_HOME/bin #GRADLE_HOME export GRADLE_HOME=/u01/gradle-4.6 export GRADLE_USER_HOME=/u01/gradle-4.6/user_home export PATH=$PATH:$GRADLE_HOME/bin2.2 编译azkaban-3.84.4 2.2.1 下载对应源码包
应该是从这里azkaban-3.84.4
wget https://github.com/azkaban/azkaban/archive/refs/tags/3.84.4.tar.gz tar -zxvf 3.84.4.tar.gz cd azkaban-3.84.4/2.2.2 修改对应的mysql版本
1)azkaban-3.84.4目录下的build.gradle文件107行
107 mysqlConnector : 'mysql:mysql-connector-java:5.1.28', ##改为 mysqlConnector : 'mysql:mysql-connector-java:8.0.22',
2)修改mysql驱动程序 MySQLDataSource.java 51行
[root@hadoop102 db]# pwd
/u02/azkaban-3.84.4/azkaban-db/src/main/java/azkaban/db
[root@hadoop102 db]# vim MySQLDataSource.java
51行 改成setDriverClassName("com.mysql.cj.jdbc.Driver");
2.2.3 尝试编译
[root@hadoop102 azkaban-3.84.4]# gradle build -x test
报错 之前在windows编译也报这个错误,说是要安装nodejs 什么的,怕弄脏windows,改在虚拟机里搞
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':azkaban-web-server:nodeSetup'.
> Could not resolve all files for configuration ':azkaban-web-server:detachedConfiguration1'.
> Could not resolve org.nodejs:node:8.10.0.
Required by:
project :azkaban-web-server
> Could not resolve org.nodejs:node:8.10.0.
> Could not get resource 'https://nodejs.org/dist/v8.10.0/ivy.xml'.
> Could not GET 'https://nodejs.org/dist/v8.10.0/ivy.xml'.
> Connect to nodejs.org:443 [nodejs.org/104.20.23.46, nodejs.org/104.20.22.46, nodejs.org/2606:4700:10:0:0:0:6814:162e, nodejs.org/2606:4700:10:0:0:0:6814:172e] failed: 网络不可达 (connect failed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 2m 50s
51 actionable tasks: 51 executed
2.2.3 尝试解决报错
1.不要自动下载npm,手动下载配置后屏蔽自动下载
1)上传文件到u02目录;解压缩到u01目录;修改文件名;配置环境变量;
[root@hadoop102 u02]# tar -zxvf node-v8.10.0-linux-x64.tar.gz -C /u01/ [root@hadoop102 u02]# cd /u01/ [root@hadoop102 u01]# mv node-v8.10.0-linux-x64/ node-v8.10.0 [root@hadoop102 u01]# vim /etc/profile.d/my_env.sh [root@hadoop102 u01]# cat /etc/profile.d/my_env.sh #JAVA_HOME export JAVA_HOME=/opt/module/jdk1.8.0_212 export PATH=$PATH:$JAVA_HOME/bin #HADOOP_HOME export HADOOP_HOME=/opt/module/hadoop-3.1.3 export PATH=$PATH:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin #HIVE_HOME export HIVE_HOME=/opt/module/hive-3.1.2 export PATH=$PATH:$HIVE_HOME/bin #KAFKA_HOME export KAFKA_HOME=/opt/module/kafka-3.0.0 export PATH=$PATH:$KAFKA_HOME/bin #GRADLE_HOME export GRADLE_HOME=/u01/gradle-4.6 export GRADLE_USER_HOME=/u01/gradle-4.6/user_home export PATH=$PATH:$GRADLE_HOME/bin #NODE_HOME export NODE_HOME=/u01/node-v8.10.0 export PATH=$PATH:$NODE_HOME/bin
2)修改web-server下的build.gradle文件,注释掉 node{} 部分,并重新编译试试
[root@hadoop102 u01]# vim /u02/azkaban-3.84.4/azkaban-web-server/build.gradle
node {} 整个注释掉
root@hadoop102 azkaban-3.84.4]# gradle build -x test
- 编译成功了!!!!!!!!!!!!!!!!!!!!!!!
[root@hadoop102 distributions]# pwd /u02/azkaban-3.84.4/azkaban-web-server/build/distributions [root@hadoop102 distributions]# ll 总用量 45496 -rw-r--r--. 1 root root 23232984 6月 1 21:04 azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -rw-r--r--. 1 root root 23348841 6月 1 21:04 azkaban-web-server-0.1.0-SNAPSHOT.zip [root@hadoop102 distributions]# [root@hadoop102 distributions]# pwd /u02/azkaban-3.84.4/azkaban-exec-server/build/distributions [root@hadoop102 distributions]# ll 总用量 37428 -rw-r--r--. 1 root root 19160043 5月 31 23:22 azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -rw-r--r--. 1 root root 19164289 5月 31 23:22 azkaban-exec-server-0.1.0-SNAPSHOT.zip [root@hadoop102 distributions]#2.3.2 删除老的包,导入新的包
切换到atguigu用户由于之前的mysql驱动有问题,删除残留的数据
1)删除所有节点上残留的包
[atguigu@hadoop102 ~]$ cd /opt/software/ [atguigu@hadoop102 software]$ rm -f azkaban-exec-server-3.84.4.tar.gz azkaban-web-server-3.84.4.tar.gz [atguigu@hadoop102 ~]$ cd /opt/module/azkaban-3.84.4/ [atguigu@hadoop102 azkaban-3.84.4]$ rm -rf azkaban-web/ azkaban-exec/ [atguigu@hadoop103 ~]$ cd /opt/module/azkaban-3.84.4/ [atguigu@hadoop103 azkaban-3.84.4]$ rm -rf azkaban-exec/ [atguigu@hadoop104 ~]$ cd /opt/module/azkaban-3.84.4/ [atguigu@hadoop104 azkaban-3.84.4]$ rm -rf azkaban-exec/ xshell直接拖过去 -rw-r--r--. 1 atguigu atguigu 19160043 5月 31 23:22 azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -rw-r--r--. 1 atguigu atguigu 23232984 6月 1 21:04 azkaban-web-server-0.1.0-SNAPSHOT.tar.gz2.3.3 搭建环境
1)解压改名等
[atguigu@hadoop102 software]$ tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C /opt/module/azkaban-3.84.4/ [atguigu@hadoop102 software]$ tar -zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C /opt/module/azkaban-3.84.4/ [atguigu@hadoop102 azkaban-3.84.4]$ mv azkaban-exec-server-0.1.0-SNAPSHOT/ azkaban-exec-3.84.4 [atguigu@hadoop102 azkaban-3.84.4]$ mv azkaban-web-server-0.1.0-SNAPSHOT/ azkaban-web-3.84.4
2)配置mysql 略: 建azkaban库,建azkaban用户,授权,执行建表语句,修改数据包大小,重启服务等
3)配置executor
- 编辑 azkaban.properties :修改时区,webserver,executor 端口 ,mysql 配置
[atguigu@hadoop102 azkaban-3.84.4]$ cd azkaban-exec-3.84.4/conf/ [atguigu@hadoop102 conf]$ vim azkaban.properties #... default.timezone.id=Asia/Shanghai #... azkaban.webserver.url=http://hadoop102:8081 executor.port=12321 #... database.type=mysql mysql.port=3306 mysql.host=hadoop102 mysql.database=azkaban mysql.user=azkaban mysql.password=000000
- 同步到所有节点
[atguigu@hadoop102 azkaban-3.84.4]$ xsync azkaban-exec-3.84.4/
- 在所有节点启动executor注意执行路径 ,都出现executor.port ,启动成功
[atguigu@hadoop102 azkaban-exec-3.84.4]$ bin/start-exec.sh [atguigu@hadoop102 azkaban-exec-3.84.4]$ ll 总用量 48 drwxr-xr-x. 3 atguigu atguigu 4096 5月 31 23:22 bin drwxr-xr-x. 2 atguigu atguigu 4096 6月 1 21:47 conf -rw-rw-r--. 1 atguigu atguigu 5 6月 1 21:52 currentpid drwxrwsr-x. 2 atguigu atguigu 4096 6月 1 21:53 executions -rw-rw-r--. 1 atguigu atguigu 5 6月 1 21:53 executor.port -rw-rw-r--. 1 atguigu atguigu 7747 6月 1 21:53 executorServerLog__2022-06-01+21:52:58.out drwxr-xr-x. 2 atguigu atguigu 4096 5月 31 23:22 lib drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:52 logs drwxr-xr-x. 3 atguigu atguigu 4096 5月 31 23:22 plugins drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:53 projects drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:53 temp --------------------------------------------------------------------- [atguigu@hadoop103 azkaban-3.84.4]$ cd azkaban-exec-3.84.4/ [atguigu@hadoop103 azkaban-exec-3.84.4]$ ll 总用量 48 drwxr-xr-x. 3 atguigu atguigu 4096 5月 31 23:22 bin drwxr-xr-x. 2 atguigu atguigu 4096 6月 1 21:47 conf -rw-rw-r--. 1 atguigu atguigu 5 6月 1 21:53 currentpid drwxrwsr-x. 2 atguigu atguigu 4096 6月 1 21:53 executions -rw-rw-r--. 1 atguigu atguigu 5 6月 1 21:53 executor.port -rw-rw-r--. 1 atguigu atguigu 7747 6月 1 21:53 executorServerLog__2022-06-01+21:53:16.out drwxr-xr-x. 2 atguigu atguigu 4096 5月 31 23:22 lib drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:53 logs drwxr-xr-x. 3 atguigu atguigu 4096 5月 31 23:22 plugins drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:53 projects drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:53 temp --------------------------------------------------------------------- [atguigu@hadoop104 azkaban-exec-3.84.4]$ bin/start-exec.sh [atguigu@hadoop104 azkaban-exec-3.84.4]$ ll 总用量 48 drwxr-xr-x. 3 atguigu atguigu 4096 5月 31 23:22 bin drwxr-xr-x. 2 atguigu atguigu 4096 6月 1 21:47 conf -rw-rw-r--. 1 atguigu atguigu 5 6月 1 21:54 currentpid drwxrwsr-x. 2 atguigu atguigu 4096 6月 1 21:54 executions -rw-rw-r--. 1 atguigu atguigu 5 6月 1 21:54 executor.port -rw-rw-r--. 1 atguigu atguigu 7747 6月 1 21:54 executorServerLog__2022-06-01+21:54:12.out drwxr-xr-x. 2 atguigu atguigu 4096 5月 31 23:22 lib drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:54 logs drwxr-xr-x. 3 atguigu atguigu 4096 5月 31 23:22 plugins drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:54 projects drwxrwxr-x. 2 atguigu atguigu 4096 6月 1 21:54 temp
- 激活executor
mysql> select * from executors;
+----+-----------+-------+--------+
| id | host | port | active |
+----+-----------+-------+--------+
| 1 | hadoop102 | 45951 | 0 |
| 2 | hadoop103 | 32972 | 0 |
| 3 | hadoop104 | 45289 | 0 |
+----+-----------+-------+--------+
3 rows in set (0.00 sec)
[atguigu@hadoop102 azkaban-exec-3.84.4]$ curl -G "hadoop102:12321/executor?action=activate" && echo
{"status":"success"}
[atguigu@hadoop102 azkaban-exec-3.84.4]$ curl -G "hadoop103:12321/executor?action=activate" && echo
{"status":"success"}
[atguigu@hadoop102 azkaban-exec-3.84.4]$ curl -G "hadoop104:12321/executor?action=activate" && echo
{"status":"success"}
[atguigu@hadoop102 azkaban-exec-3.84.4]$
mysql> select * from executors;
+----+-----------+-------+--------+
| id | host | port | active |
+----+-----------+-------+--------+
| 4 | hadoop102 | 12321 | 1 |
| 5 | hadoop103 | 12321 | 1 |
| 6 | hadoop104 | 12321 | 1 |
+----+-----------+-------+--------+
3 rows in set (0.00 sec)
3)配置webServer
- 编辑 azkaban.properties:时区,mysql,内存占用情况修改
[atguigu@hadoop102 azkaban-3.84.4]$ vim azkaban-web-3.84.4/conf/azkaban.properties
- 编辑azkaban-users.xml :添加
[atguigu@hadoop102 azkaban-3.84.4]$ cat azkaban-web-3.84.4/conf/azkaban-users.xml
- 启动webserver : 注意路径
[atguigu@hadoop102 azkaban-web-3.84.4]$ bin/start-web.sh
4.访问 http://hadoop102:8081,并用 atguigu 用户登陆:成功
azkaban-3.84.4重新编译后的executor和web-server,修改mysql驱动为8.x版本



