栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

使用apache的HttpClient上传文件到Hive数据仓库

使用apache的HttpClient上传文件到Hive数据仓库

        上个月,项目中需要实现上传文件到Hvie数据仓库,一开始使用shell语句上传,在本地是OK的。但是部署到客户环境就报错,所以改用HttpClient组件上传。

        主要是用来记录一下,不做过多文字解释。
        里面注释也有不少,应该够用了。

public Map httpPost(String fileName, String path) throws HttpException, IOException {
		Map ret = new HashMap<>();
		path = path.replace("\", "/");
		String suffix = fileName.substring(fileName.lastIndexOf(".") + 1);
		String hiveAddress = System.getProperty("hive.address");
		// 仓库名称
		String name = System.getProperty("hive.name");
		// 仓库 组名称
		String groupName = System.getProperty("hive.groupName");
		
		PostMethod postMethod = new PostMethod(hiveAddress + "/service/rest/v1/components?repository=" + name);
		HttpClient httpclient = new HttpClient();
		// 登录验证,用户名密码
		UsernamePasswordCredentials creds = new UsernamePasswordCredentials(System.getProperty("hive.username"), System.getProperty("hive.password"));
		httpclient.getState().setCredentials(AuthScope.ANY, creds);
		httpclient.getParams().setAuthenticationPreemptive(true);
		postMethod.setDoAuthentication(true);
		//附件
		File file = new File(path + fileName);
		FilePart filePart = new FilePart("maven2.asset1", file);
		filePart.setCharSet("UTF-8");
		// formData参数
		StringPart groupId = new StringPart("maven2.groupId", groupName, "UTF-8");
		StringPart artifactId = new StringPart("maven2.artifactId", fileName, "UTF-8");
		StringPart version = new StringPart("maven2.version", "0.0.1", "UTF-8");
		StringPart extension = new StringPart("maven2.asset1.extension", suffix, "UTF-8");
		postMethod.addRequestHeader("accept", "application/json");
		try {
			// 参数&附件
			org.apache.commons.httpclient.methods.multipart.Part[] parts = { groupId, artifactId, extension, version, filePart };
			// 设置请求体
			postMethod.setRequestEntity(new MultipartRequestEntity(parts, postMethod.getParams()));
			// 设置最大的连接超时时间(如果超过5秒没有连接成功,则报错)
			httpclient.getHttpConnectionManager().getParams().setConnectionTimeout(5000);
			// 执行http请求
			int statusCode = httpclient.executeMethod(postMethod);
			log.info("execute httpclient return code = " + statusCode);
			//200 204
			if (statusCode != HttpStatus.SC_OK && statusCode != HttpStatus.SC_NO_CONTENT) {
				log.info("httpPost respone body = " + new String(postMethod.getResponseBody(), "utf-8"));
			}
			ret.put("code", "1");
			return ret;
		} catch (FileNotFoundException e1) {
			ret.put("code", "0");
			ret.put("msg", "上传文件至Hive数据仓库出现错误,请联系管理员");
			log.info("上传文件至Hive数据仓库出现错误",e1);
			return ret;
		} catch(ConnectTimeoutException e1 ){
			ret.put("code", "0");
			ret.put("msg", "连接Hive数据仓库出现错误,请联系管理员");
			log.info("连接Hive数据仓库出现错误",e1);
			return ret;
		}finally {
			// 释放连接
			postMethod.releaseConnection();
		}
	}

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/671923.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号