栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

requests httpclients使用json传递多层参数爬取数据

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

requests httpclients使用json传递多层参数爬取数据

1、背景

测试爬取广州工程行业评分数据,参数如下图。是两层的,如果用普通的键值对方式传参。会报“参数不符合规范,不能进行查询”错误。需要将参数先转成json再传递。

2、python requests方法
#!/usr/bin/env python3
#coding:utf-8

import json
import requests
def main():
    url = "http://qycx.gzcc.gov.cn:8081/eval/evalGateWay/query"
    headers = {
               "Content-Type": "application/json" # 增加该请求头
            }
    post_data1 = {
        "search": {
        "categoryCode": "101",
        "dateDate": "2022-06-16",
        "enterpriseName": "",
        "subjectTypeId": "f39465d9f92118343b93656d71437ada",
        "totalRankEnd": "",
        "totalRankStark": "",
        "totalScoreEnd": "",
        "totalScoreStark": ""
        }
    }
	# 使用json.dumps将字段转换成json格式即可
    rs = requests.post(url=url,data=json.dumps(post_data1), headers=headers)
    print(rs.headers, rs.status_code, rs.text)

if __name__ == '__main__':
    main()
3、java httpclients方法
import com.google.gson.Gson; # 导入Gson包
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

public class EnterpriseCredit {
    public static void main(String[] args) throws IOException {
        // 1、创建HttpClients对象
        CloseableHttpClient httpclient = HttpClients.createDefault();

        //2、创建httpPost对象,设置请求url地址
        HttpPost httpPost = new HttpPost("http://qycx.gzcc.gov.cn:8081/eval/evalGateWay/query");
		//设置Content-Type请求头
        httpPost.setHeader("Content-Type", "application/json;charset=utf-8");

        //声明HashMap,封装表单参数search的value
        HashMap searchValueMap = new HashMap<>();
        searchValueMap.put("categoryCode", "101");
        searchValueMap.put("dateDate", "2022-06-15");
        searchValueMap.put("enterpriseName", null);
        searchValueMap.put("subjectTypeId", "f39465d9f92118343b93656d71437ada");
        searchValueMap.put("totalRankEnd", null);
        searchValueMap.put("totalRankStark", null);
        searchValueMap.put("totalScoreEnd", null);
        searchValueMap.put("totalScoreStark", null);
        //申明一个Gson
        Gson gson = new Gson();
        //申明HashMap,封装请求表单参数
        HashMap searchMap = new HashMap<>();
        searchMap.put("search", searchValueMap);
        //将表单参数转换成json再转换成字符串
        String searchJson = gson.toJson(searchMap).toString();
        //创建jsonEntity,供httpPost使用
        StringEntity jsonEntity = new StringEntity(searchJson);

        //设置表单的Entity对象到Post请求中
        httpPost.setEntity(jsonEntity);

        //3、使用HttpClient发起请求,获取response
        CloseableHttpResponse response = httpclient.execute(httpPost);
        //4、解析响应
        if (response.getStatusLine().getStatusCode() == 200){
            String content = EntityUtils.toString(response.getEntity(), "utf8");
            System.out.println(content);
        }
        response.close();
        httpclient.close();
    }
}

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/978359.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号