栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

[Hudi]hudi的编译及hudi&spark和hudi&flink的简单使用

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

[Hudi]hudi的编译及hudi&spark和hudi&flink的简单使用

目录
  • maven的安装
    • mvn的下载地址
  • 方式一:使用git方式安装hudi
    • 安装git
  • 方式二:下载hudi-0.10.1.src.tgz 方式安装
  • 注
  • spark结合hudi
    • spark-shell启动
    • 设置表名
    • 插入数据
    • 查询数据
    • 修改数据
    • 修改数据的查询
    • 时间点查询
    • 删除数据
    • 覆盖数据
  • hudi 结合Flink Sql Client操作
    • 插入数据

一般都是用maven进行编译。但直接解压也可以使用

maven的安装 mvn的下载地址

https://archive.apache.org/dist/maven/maven-3/


https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/
我用的3.5.4版本,开始用的3.2.5编译失败,选择高版本后直接成功

vi /etc/profile

底部添加java和maven相关的环境变量

export JAVA_HOME=/usr/local/jdk_lin
export GOPATH=/opt/go
export GOROOT=/opt/go
export HADOOP_HOME=/opt/hadoop-3.1.4
export MAVEN_HOME=/opt/apache-maven-3.5.4

export SPARK_HOME=/opt/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PYSPARK_DRIVER_PYTHON=jupyter-notebook
export PYSPARK_DRIVER_PYTHON_OPTS="--ip=0.0.0.0 --port=7070"

CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$MAVEN_HOME/bin:$HADOOP_HOME/bin:$GOROOT/bin:$SPARK_HOME/bin:$SPARK_HOME/jars
source /etc/profile
mvn -v

进去maven的conf目录

vi settings.xml


        nexus-aliyun
        central
        Nexus aliyun
        http://maven.aliyun.com/nexus/content/groups/public

方式一:使用git方式安装hudi 安装git
yum install git

git --version

git clone --branch release-0.10.1 https://gitee.com/apache/Hudi.git

下载成功后,进入hudi的安装目录

vi pom.xml

在repository部分添加如下

 
        nexus-aliyun
        nexus-aliyun
        http://maven.aliyun.com/nexus/content/groups/public/
        
            true
        
        
            false
        
 

构建hudi,执行了十七分钟:

mvn clean package -DskipTests -Dspark3 -Dscala-2.12

方式二:下载hudi-0.10.1.src.tgz 方式安装

hudi下载地址:
https://downloads.apache.org/

下载后
1:配置pom文件。
2.执行编译 mvn clean package -DskipTests -Dspark3 -Dscala-2.12

同上
用了九分钟

(base) [root@yxkj153 Hudi]# ./hudi-cli/hudi-cli.sh

注

1.hudi和maven的版本需要注意;
2.解压hudi安装包的时候使用tar -zxvf 不管用,可以是安装包下载不完全,进行重新下载

spark结合hudi spark-shell启动

spark-shell启动,需要指定spark-avro模块,因为默认环境里没有,spark-avro模块版本好需要和spark版本对应。这里使用的是hudi已经编译好的jar包。

(base) [root@yxkj153 target]# /opt/spark-3.1.2-bin-hadoop3.2/bin/spark-shell --jars /opt/hudi-0.10.1/packaging/hudi-spark-bundle/target/hudi-spark3.1.2-bundle_2.12-0.10.1.jar --package org.apache.spark:spark-avro_2.12:3.1.2  --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'

设置表名

设置表名,基本路径和数据生成器

scala> import org.apache.hudi.QuickstartUtils._
import org.apache.hudi.QuickstartUtils._

scala> import scala.collection.JavaConversions._
import scala.collection.JavaConversions._

scala> import org.apache.spark.sql.SaveMode._
import org.apache.spark.sql.SaveMode._

scala> import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceReadOptions._

scala> import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.DataSourceWriteOptions._

scala> import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.hudi.config.HoodieWriteConfig._

scala> val tableName = "hudi_trips_cow"
tableName: String = hudi_trips_cow

scala> val basePath = "file:///tmp/hudi_trips_cow"
basePath: String = file:///tmp/hudi_trips_cow

scala>  val dataGen = new DataGenerator
dataGen: org.apache.hudi.QuickstartUtils.DataGenerator = org.apache.hudi.QuickstartUtils$DataGenerator@47effce0

插入数据

生成一些新数据,加载到DataFrame中,然后将DataFrame写入Hudi表。

scala>  val dataGen = new DataGenerator
dataGen: org.apache.hudi.QuickstartUtils.DataGenerator = org.apache.hudi.QuickstartUtils$DataGenerator@47effce0

scala> val inserts = convertToStringList(dataGen.generateInserts(10))
inserts: java.util.List[String] = [{"ts": 1654951305538, "uuid": "c9571c23-a838-41c0-a3e7-03d0a5b642b8", "rider": "rider-213", "driver": "driver-213", "begin_lat": 0.4726905879569653, "begin_lon": 0.46157858450465483, "end_lat": 0.754803407008858, "end_lon": 0.9671159942018241, "fare": 34.158284716382845, "partitionpath": "americas/brazil/sao_paulo"}, {"ts": 1655083361273, "uuid": "db91da5a-fb40-4600-8907-a0898b3d925c", "rider": "rider-213", "driver": "driver-213", "begin_lat": 0.6100070562136587, "begin_lon": 0.8779402295427752, "end_lat": 0.3407870505929602, "end_lon": 0.5030798142293655, "fare": 43.4923811219014, "partitionpath": "americas/brazil/sao_paulo"}, {"ts": 1654575413398, "uuid": "2f71860d-4269-4140-9b01-9c2003b5faef", "rider": "rider-213", "driver"...

scala> val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
warning: there was one deprecation warning (since 2.12.0)
warning: there was one deprecation warning (since 2.2.0)
warning: there were two deprecation warnings in total; for details, enable `:setting -deprecation' or `:replay -deprecation'
df: org.apache.spark.sql.DataFrame = [begin_lat: double, begin_lon: double ... 8 more fields]

scala>  df.write.format("hudi").
     |         options(getQuickstartWriteConfigs).
     |         option(PRECOMBINE_FIELD_OPT_KEY, "ts").
     |         option(RECORDKEY_FIELD_OPT_KEY, "uuid").
     |         option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
     |         option(TABLE_NAME, tableName).
     |         mode(Overwrite).
     |         save(basePath)
warning: there was one deprecation warning; for details, enable `:setting -deprecation' or `:replay -deprecation'
22/06/13 10:00:20 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
22/06/13 10:00:20 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file

scala>

Mode(overwrite)将覆盖重新创建表(如果已存在)。可以检查/tmp/hudi_trps_cow 路径下是否有数据生成。

(base) [root@yxkj153 target]# cd /tmp/hudi_trips_cow/
(base) [root@yxkj153 hudi_trips_cow]# ll
total 0
drwxr-xr-x 4 root root 41 Jun 13 10:00 americas
drwxr-xr-x 3 root root 19 Jun 13 10:00 asia
查询数据

因为测试数据分区是 【区域/国家/城市】,所以load(basePath “////”)

scala> val tripsSnapshotDF = spark.read.format("hudi").load(basePath + "*****/*/*").
     |   select("uuid","partitionpath").
     |   sort("partitionpath","uuid").
     |   show(100, false)

hudi 结合Flink Sql Client操作

flink提交任务还是注意在主节点上提交。 要不然可能会报错
flink sql环境的配置参考:
https://nightlies.apache.org/flink/flink-docs-release-1.12/zh/dev/table/sqlClient.html#environment-files

启动命令

(base) [root@yxkj153 opt]# /opt/flink-1.13.6/bin/sql-client.sh embedded -j  /opt/hudi-0.10.1/packaging/hudi-flink-bundle/target/hudi-flink-bundle_2.12-0.10.1.jar
插入数据

(1)设置返回结果模式为tableau,让结果直接显示,设置处理模式为批处理

Flink SQL> set execution.result-mode=tableau;
Flink SQL> SET execution.type = batch;

(2)创建一张Merge on Read的表,如果不指定默认为copy on write表
提供的connector

Available factory identifiers are:

blackhole
datagen
filesystem
hudi
print

CREATE TABLE t1
( uuid VARCHAr(20), 
name VARCHAr(10), 
age INT, ts TIMESTAMP(3), 
`partition` VARCHAr(20)
)PARTITIONED BY (`partition`)
WITH ( 
'connector' = 'hudi', 
'path' = 'hdfs://mycluster/flink-hudi/t1', 
'table.type' = 'MERGE_ON_READ');

(3)插入数据

INSERT INTO t1 VALUES ('id1','Danny',23,TIMESTAMP '1970-01-01 00:00:01','par1'), ('id2','Stephen',33,TIMESTAMP '1970-01-01 00:00:02','par1'), ('id3','Julian',53,TIMESTAMP '1970-01-01 00:00:03','par2'), ('id4','Fabian',31,TIMESTAMP '1970-01-01 00:00:04','par2'), ('id5','Sophia',18,TIMESTAMP '1970-01-01 00:00:05','par3'), ('id6','Emma',20,TIMESTAMP '1970-01-01 00:00:06','par3'), ('id7','Bob',44,TIMESTAMP '1970-01-01 00:00:07','par4'), ('id8','Han',56,TIMESTAMP '1970-01-01 00:00:08','par4');

(4)查看hdfs ui提示成功,查看对应9870页面,hdfs路径下已有数据产生


flink的ui显示正在运行flink任务
下图是执行

Flink SQL> select *from t1;

flink ui展示的图

flink的hive部分官网介绍:
https://nightlies.apache.org/flink/flink-docs-release-1.12/zh/dev/table/connectors/hive/

flink的sql官网介绍
https://nightlies.apache.org/flink/flink-docs-release-1.12/zh/dev/table/sqlClient.html

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/969657.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号