记录一次爬虫题库

记录一次爬虫题库前言

想做个刷题的微信小程序,所以需要做一个题库
废话少说,上爬虫

爬取

进入刷题网站时加载时间略长

可以确实是vue开发的前端
请求头

accept-encoding: gzip, deflate, br
accept-language: zh-CN,zh;q=0.9
content-length: 2
content-type: application/json;charset=UTF-8
cookie: x-token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJjc3d4X3VzZXJfaWQiOiI2OTUwMzAwMTY3MjQxMjAzNzE1IiwibG9naW5fdGltZSI6MTY1NzI0NjI4NDQwMiwiaXNzIjoi6ICD55-l6YCaIiwic2VydmljZS1jb2RlIjoiY3N0ayIsImNzdGtfdXNlcl9pZCI6IjY5NTAzMDAxNjcyNDEyMDM3MTUiLCJleHAiOjE2NTg5NzQyODQ0MDIsImlhdCI6MTY1NzI0NjI4NDQwMn0.XtLBW1Fl4OMSMvKhTPRu4RNj-czvfLgKETNHwLjjyvo
origin: 手动打码
referer: 手动打码
sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
service-code: cstk
timestamp: 1657335133950
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
version: 3.0.1
web-authorize-flag: hmzJ/IPcepp9QKtustWPqw==
x-authorize-token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJjc3d4X3VzZXJfaWQiOiI2OTUwMzAwMTY3MjQxMjAzNzE1IiwibG9naW5fdGltZSI6MTY1NzI0NjI4NDQwMiwiaXNzIjoi6ICD55-l6YCaIiwic2VydmljZS1jb2RlIjoiY3N0ayIsImNzdGtfdXNlcl9pZCI6IjY5NTAzMDAxNjcyNDEyMDM3MTUiLCJleHAiOjE2NTg5NzQyODQ0MDIsImlhdCI6MTY1NzI0NjI4NDQwMn0.XtLBW1Fl4OMSMvKhTPRu4RNj-czvfLgKETNHwLjjyvo

requests复现即可
成功爬取subject shapter等数据

重头戏

爬取题目的时候发现并没有题目
题目list中的一个

[{
categoryId: "20191218160022-dca2f545-4e2a-4532-878a-cb791bcdd57a"
analysis: "循环冗余校验用于检错，不能自动纠正错误和重新生成数据。通常的做法可以丢弃，自动请求重发和报告上层协议，关键在于使用CRC的这个协议是什么性质。如果是无连接的则丢弃，面向连接可靠的则请求重发"
analysisFilePath: null
answer: "[{"option":"A","content":" 自动纠错 ","rightFlag":"0"},{"option":"B","content":" 报告上层协议 ","rightFlag":"0"},{"option":"C","content":" 重新生成数据 ","rightFlag":"0"},{"option":"D","content":" 自动请求重发 ","rightFlag":"1"}]"
answerList: null
answertype: null
categoryId: "20191218160022-dca2f545-4e2a-4532-878a-cb791bcdd57a"
chapterId: "1604386088961-f3dedccd-aa5c-408d-9b24-55090ec96241"
collectFlag: 0
fraction: 1
id: "20201103145848-d02188e6-7ff1-45b2-9175-b7072bebd7af"
logId: "61544539661743947776"
questionType: {type: "0", name: "单项选择题"}
questionTypeId: null
rightFlag: "0"
seqVal: 1
stem: "iH7ZkmCDBacVeDdbDd/EXGQX/wFx9v341Sb1Jk6vWLD67+XKUKlhdMhRFGHOFDkiUrhhKsPLhZybuQB++DSorgbvQcPkm6bxSP1dY4UY8zvqFSOxEPseI3fZQWUA1je4ecwHyJg9R/QnAA3V3VJj7JKfaHZA3Hi+tLkzWEV3h5Nf4eUfaAON4kvVOQkI6TMevcAwLUVFRtmWAmfFuZWDzw=="
stemFilePath: null
userAnswer: "A"
}]

这就奇了怪了,api返回json中没看到题目而前端可以看到题目
猜测是缓存在本地
在Sources上翻了一圈没看到
猜测是不是遇到加密了
在js中搜索stem
果不其然
部分加密函数

 _.forEach((function(q) {
                            n.includes(q.questionType.name) || n.push(q.questionType.name);
                            var data = q.stem
                              , e = crypto_js__WEBPACK_IMPORTED_MODULE_15___default.a.enc.Utf8.parse("手动打码")
                              , t = crypto_js__WEBPACK_IMPORTED_MODULE_15___default.a.AES.decrypt(data, e, {
                                mode: crypto_js__WEBPACK_IMPORTED_MODULE_15___default.a.mode.ECB,
                                padding: crypto_js__WEBPACK_IMPORTED_MODULE_15___default.a.pad.Pkcs7
                            }).toString(crypto_js__WEBPACK_IMPORTED_MODULE_15___default.a.enc.Utf8);
                            q.stem = t
                        }

原来stem字段是经过AES加密过的
私钥,加密模式都给出了
尝试解密

AES
model:ECB
padding:pkcs7
key:手动打码
output:base64

"=="结尾很明显是base64输出

iH7ZkmCDBacVeDdbDd/EXGQX/wFx9v341Sb1Jk6vWLD67+XKUKlhdMhRFGHOFDkiUrhhKsPLhZybuQB++DSorgbvQcPkm6bxSP1dY4UY8zvqFSOxEPseI3fZQWUA1je4ecwHyJg9R/QnAA3V3VJj7JKfaHZA3Hi+tLkzWEV3h5Nf4eUfaAON4kvVOQkI6TMevcAwLUVFRtmWAmfFuZWDzw==

输出

循环冗余校验标准CRC-16的生成多项式为G(x)=x^16+x^15+x^2+1，它产生的校验码是16位，接收端发现错误后采取的措施是()。

漂亮
写个脚本对json进行清洗
直接复制json后使用navicat直接导入数据库
查询一下

+-----------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| stem                                                                                                                                                      | analysis                                                                                                                                                                                                                                                                             |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 循环冗余校验标准CRC-16的生成多项式为G(x)=x^16+x^15+x^2+1，它产生的校验码是16位，接收端发现错误后采取的措施是()。                                                 | 循环冗余校验用于检错，不能自动纠正错误和重新生成数据。通常的做法可以丢弃，自动请求重发和报告上层协议，关键在于使用CRC的这个协议是什么性质。如果是无连接的则丢弃，面向连接可靠的则请求重发                                                                                            |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

nice

后记

经典的前端信息不安全
即使解密过程由前端完成,还是能被爬
(如果能够加一点代码混淆可能难度会提高,但理论上还是可以爬取)
解密由后端完成那更不用说了,直接是明文传输
安全二字常记心头

记录一次爬虫题库

Python相关栏目本月热门文章