- maven的安装
- mvn的下载地址
- 方式一:使用git方式安装hudi
- 安装git
- 方式二:下载hudi-0.10.1.src.tgz 方式安装
- 注
- spark结合hudi
- spark-shell启动
- 设置表名
- 插入数据
- 查询数据
- 修改数据
- 修改数据的查询
- 时间点查询
- 删除数据
- 覆盖数据
- hudi 结合Flink Sql Client操作
- 插入数据
一般都是用maven进行编译。但直接解压也可以使用 maven的安装 mvn的下载地址
https://archive.apache.org/dist/maven/maven-3/
https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/
我用的3.5.4版本,开始用的3.2.5编译失败,选择高版本后直接成功
vi /etc/profile
底部添加java和maven相关的环境变量
export JAVA_HOME=/usr/local/jdk_lin export GOPATH=/opt/go export GOROOT=/opt/go export HADOOP_HOME=/opt/hadoop-3.1.4 export MAVEN_HOME=/opt/apache-maven-3.5.4 export SPARK_HOME=/opt/spark export SPARK_CONF_DIR=$SPARK_HOME/conf export PYSPARK_DRIVER_PYTHON=jupyter-notebook export PYSPARK_DRIVER_PYTHON_OPTS="--ip=0.0.0.0 --port=7070" CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$MAVEN_HOME/bin:$HADOOP_HOME/bin:$GOROOT/bin:$SPARK_HOME/bin:$SPARK_HOME/jars
source /etc/profile mvn -v
进去maven的conf目录
vi settings.xml
方式一:使用git方式安装hudi 安装gitnexus-aliyun central Nexus aliyun http://maven.aliyun.com/nexus/content/groups/public
yum install git git --version git clone --branch release-0.10.1 https://gitee.com/apache/Hudi.git
下载成功后,进入hudi的安装目录
vi pom.xml
在repository部分添加如下
nexus-aliyun nexus-aliyun http://maven.aliyun.com/nexus/content/groups/public/ true false
构建hudi,执行了十七分钟:
mvn clean package -DskipTests -Dspark3 -Dscala-2.12方式二:下载hudi-0.10.1.src.tgz 方式安装
hudi下载地址:
https://downloads.apache.org/
下载后
1:配置pom文件。
2.执行编译 mvn clean package -DskipTests -Dspark3 -Dscala-2.12
同上
用了九分钟
(base) [root@yxkj153 Hudi]# ./hudi-cli/hudi-cli.sh注
1.hudi和maven的版本需要注意;
2.解压hudi安装包的时候使用tar -zxvf 不管用,可以是安装包下载不完全,进行重新下载
spark-shell启动,需要指定spark-avro模块,因为默认环境里没有,spark-avro模块版本好需要和spark版本对应。这里使用的是hudi已经编译好的jar包。
(base) [root@yxkj153 target]# /opt/spark-3.1.2-bin-hadoop3.2/bin/spark-shell --jars /opt/hudi-0.10.1/packaging/hudi-spark-bundle/target/hudi-spark3.1.2-bundle_2.12-0.10.1.jar --package org.apache.spark:spark-avro_2.12:3.1.2 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'设置表名
设置表名,基本路径和数据生成器
scala> import org.apache.hudi.QuickstartUtils._ import org.apache.hudi.QuickstartUtils._ scala> import scala.collection.JavaConversions._ import scala.collection.JavaConversions._ scala> import org.apache.spark.sql.SaveMode._ import org.apache.spark.sql.SaveMode._ scala> import org.apache.hudi.DataSourceReadOptions._ import org.apache.hudi.DataSourceReadOptions._ scala> import org.apache.hudi.DataSourceWriteOptions._ import org.apache.hudi.DataSourceWriteOptions._ scala> import org.apache.hudi.config.HoodieWriteConfig._ import org.apache.hudi.config.HoodieWriteConfig._ scala> val tableName = "hudi_trips_cow" tableName: String = hudi_trips_cow scala> val basePath = "file:///tmp/hudi_trips_cow" basePath: String = file:///tmp/hudi_trips_cow scala> val dataGen = new DataGenerator dataGen: org.apache.hudi.QuickstartUtils.DataGenerator = org.apache.hudi.QuickstartUtils$DataGenerator@47effce0插入数据
生成一些新数据,加载到DataFrame中,然后将DataFrame写入Hudi表。
scala> val dataGen = new DataGenerator
dataGen: org.apache.hudi.QuickstartUtils.DataGenerator = org.apache.hudi.QuickstartUtils$DataGenerator@47effce0
scala> val inserts = convertToStringList(dataGen.generateInserts(10))
inserts: java.util.List[String] = [{"ts": 1654951305538, "uuid": "c9571c23-a838-41c0-a3e7-03d0a5b642b8", "rider": "rider-213", "driver": "driver-213", "begin_lat": 0.4726905879569653, "begin_lon": 0.46157858450465483, "end_lat": 0.754803407008858, "end_lon": 0.9671159942018241, "fare": 34.158284716382845, "partitionpath": "americas/brazil/sao_paulo"}, {"ts": 1655083361273, "uuid": "db91da5a-fb40-4600-8907-a0898b3d925c", "rider": "rider-213", "driver": "driver-213", "begin_lat": 0.6100070562136587, "begin_lon": 0.8779402295427752, "end_lat": 0.3407870505929602, "end_lon": 0.5030798142293655, "fare": 43.4923811219014, "partitionpath": "americas/brazil/sao_paulo"}, {"ts": 1654575413398, "uuid": "2f71860d-4269-4140-9b01-9c2003b5faef", "rider": "rider-213", "driver"...
scala> val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
warning: there was one deprecation warning (since 2.12.0)
warning: there was one deprecation warning (since 2.2.0)
warning: there were two deprecation warnings in total; for details, enable `:setting -deprecation' or `:replay -deprecation'
df: org.apache.spark.sql.DataFrame = [begin_lat: double, begin_lon: double ... 8 more fields]
scala> df.write.format("hudi").
| options(getQuickstartWriteConfigs).
| option(PRECOMBINE_FIELD_OPT_KEY, "ts").
| option(RECORDKEY_FIELD_OPT_KEY, "uuid").
| option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
| option(TABLE_NAME, tableName).
| mode(Overwrite).
| save(basePath)
warning: there was one deprecation warning; for details, enable `:setting -deprecation' or `:replay -deprecation'
22/06/13 10:00:20 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
22/06/13 10:00:20 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
scala>
Mode(overwrite)将覆盖重新创建表(如果已存在)。可以检查/tmp/hudi_trps_cow 路径下是否有数据生成。
(base) [root@yxkj153 target]# cd /tmp/hudi_trips_cow/ (base) [root@yxkj153 hudi_trips_cow]# ll total 0 drwxr-xr-x 4 root root 41 Jun 13 10:00 americas drwxr-xr-x 3 root root 19 Jun 13 10:00 asia查询数据
因为测试数据分区是 【区域/国家/城市】,所以load(basePath “////”)
scala> val tripsSnapshotDF = spark.read.format("hudi").load(basePath + "*****/*/*").
| select("uuid","partitionpath").
| sort("partitionpath","uuid").
| show(100, false)
hudi 结合Flink Sql Client操作
flink提交任务还是注意在主节点上提交。 要不然可能会报错
flink sql环境的配置参考:
https://nightlies.apache.org/flink/flink-docs-release-1.12/zh/dev/table/sqlClient.html#environment-files
启动命令
(base) [root@yxkj153 opt]# /opt/flink-1.13.6/bin/sql-client.sh embedded -j /opt/hudi-0.10.1/packaging/hudi-flink-bundle/target/hudi-flink-bundle_2.12-0.10.1.jar插入数据
(1)设置返回结果模式为tableau,让结果直接显示,设置处理模式为批处理
Flink SQL> set execution.result-mode=tableau; Flink SQL> SET execution.type = batch;
(2)创建一张Merge on Read的表,如果不指定默认为copy on write表
提供的connector
Available factory identifiers are: blackhole datagen filesystem hudi print
CREATE TABLE t1 ( uuid VARCHAr(20), name VARCHAr(10), age INT, ts TIMESTAMP(3), `partition` VARCHAr(20) )PARTITIONED BY (`partition`) WITH ( 'connector' = 'hudi', 'path' = 'hdfs://mycluster/flink-hudi/t1', 'table.type' = 'MERGE_ON_READ');
(3)插入数据
INSERT INTO t1 VALUES ('id1','Danny',23,TIMESTAMP '1970-01-01 00:00:01','par1'), ('id2','Stephen',33,TIMESTAMP '1970-01-01 00:00:02','par1'), ('id3','Julian',53,TIMESTAMP '1970-01-01 00:00:03','par2'), ('id4','Fabian',31,TIMESTAMP '1970-01-01 00:00:04','par2'), ('id5','Sophia',18,TIMESTAMP '1970-01-01 00:00:05','par3'), ('id6','Emma',20,TIMESTAMP '1970-01-01 00:00:06','par3'), ('id7','Bob',44,TIMESTAMP '1970-01-01 00:00:07','par4'), ('id8','Han',56,TIMESTAMP '1970-01-01 00:00:08','par4');
(4)查看hdfs ui提示成功,查看对应9870页面,hdfs路径下已有数据产生
flink的ui显示正在运行flink任务
下图是执行
Flink SQL> select *from t1;
flink ui展示的图
flink的hive部分官网介绍:
https://nightlies.apache.org/flink/flink-docs-release-1.12/zh/dev/table/connectors/hive/
flink的sql官网介绍
https://nightlies.apache.org/flink/flink-docs-release-1.12/zh/dev/table/sqlClient.html


![[Hudi]hudi的编译及hudi&spark和hudi&flink的简单使用 [Hudi]hudi的编译及hudi&spark和hudi&flink的简单使用](http://www.mshxw.com/aiimages/31/969657.png)
