目录
1.新建项目
2.依赖引入
3.编写函数
4.打包
5.测试
6.参考链接
1.新建项目
idea->New->Project,选择Maven和jdk-> Next,修改Name
2.依赖引入
使用udf,需要引入org.apache.hive.hive-exec包,我这边需要处理json数据,加了个fastjson包
1.8 1.8 UTF-8 1.8 org.apache.hive hive-execprovided 3.1.2 com.alibaba fastjson2.0.6 org.apache.maven.plugins maven-shade-plugin2.4.3 package shade *:* META-INF public ObjectInspector initialize(ObjectInspector[] objectInspectors) throws UDFArgumentException { if (objectInspectors.length != 2){ throw new UDFArgumentException("The number of parameters is not 2"); } return PrimitiveObjectInspectorFactory.javaStringObjectInspector; } public String evaluate(DeferredObject[] deferredObjects) throws HiveException { if (deferredObjects[0].get() == null || deferredObjects[0].get().toString().length() == 0) { if (deferredObjects[1].get() != null && deferredObjects[1].get().toString().length() != 0) { return deferredObjects[1].get().toString(); } else { // 数据异常 即关联失败 返回null return null; } } else if (deferredObjects[0].get() != null && deferredObjects[0].get().toString().length() != 0) { if (deferredObjects[1].get() == null || deferredObjects[1].get().toString().length() == 0) { return deferredObjects[0].get().toString(); } else { // 解析参数1为json对象 String return_json_str = deferredObjects[0].get().toString(); JSONObject jsonObject = JSON.parseObject(return_json_str); // 转换参数2为map String add_ob_str = deferredObjects[1].get().toString(); Map map = JSON.parseObject(add_ob_str, Map.class); for (Object obj : map.keySet()) { // 排除属性 if (obj.equals("rowkey") || obj.equals("timestamp")) { continue; } // 添加至json jsonObject.put(obj.toString(), map.get(obj).toString()); } return JSON.toJSONString(jsonObject); } } } return null; } public String getDisplayString(String[] strings) { return null; } public static void main(String[] args) throws HiveException { AddJsonObject addJsonObject = new AddJsonObject(); DeferredObject[] deferredObjects = new DeferredObject[2]; // deferredObjects[0] = new DeferredJavaObject("{"a":"b"}"); // deferredObjects[1] = new DeferredJavaObject("{"timestamp":"axjffe"}"); deferredObjects[0] = new DeferredJavaObject(""); deferredObjects[1] = new DeferredJavaObject(""); Object return_obj = addJsonObject.evaluate(deferredObjects); System.out.println("返回字符串为: " + return_obj.toString()); } }
4.打包
进入项目的target目录,找到jar包并上传到hive所在服务器,
5.测试
服务器进入spark-sql客户端,执行以下命令:
-- 添加jar包 该路径为服务器本地路径,非hdfs路径 add jar /home/zhangsan/hiveudf-1.0-SNAPSHOT.jar; -- 创建临时函数 create temporary function add_json_object as "com.myudf.AddJsonObject"; select add_json_object(column1,column2) from table_test;
6.参考链接
IDEA maven开发hive的udf详细过程(附图片详解)_Lens5935的博客-CSDN博客_maven配置hive



