栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Python

python爬虫,对粉丝夺宝的一次数据分析

Python 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

python爬虫,对粉丝夺宝的一次数据分析

新浪微博不久前出了一个类似网易一元多宝的产品。许多人质疑,那些中大奖的人都是新浪请的托,具体是不是我也不知道,所以呢,我觉得有必要找到所有中奖用户。

首先,看这个URL:http://1.weibo.com/profilehis?winner=1&uid=2860976304,它返回这个uid=2860976304的用户的中奖情况。

除此之外,我还找到一个接口,返回也是用户的中奖情况,比较好的是,它的返回是json格式,这样处理起来就简单多了。

POST http://1.weibo.com/aj/page/Profileother HTTP/1.1
Host: 1.weibo.com
Proxy-Connection: keep-alive
Content-Length: 26
Cache-Control: max-age=0
Origin: http://burp
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,**;q=0.8
Referer: http://burp/show/1
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.8
cookie: SINAGLOBAL=5042382967658.341.1454320477597; un=bjmiaoyin2006@yahoo.com.cn; wvr=6; UOR=,,login.sina.com.cn; SCF=AkD5go8FLF4mKVF2-hrc9BU_XIeLwRymEqgOVOtZEk07uzYB0zYwQpPnpt99rxQJBciji219PdUo5s7_BoUBbmA.; SUB=_2A251E2q4DeTxGeVL71cR8i7JwzuIHXVWadtwrDV8PUNbmtAKLVfzkW-AMtvP5SgHzA-5Bi3jRGlhOL_Kdw..; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WW8O9jiA56-HHn0H2.s1mvB5JpX5KMhUgL.FoefSh-7eo5f1hM2dJLoIXnLxKqL1-BL12-LxK-L12qLB-zLxK-L1h-LB.BLxK-LBo5LBo2LxK-L1-zL1-zLxKqL1-BL12-LxK-L12qLB-zLxK-L1h-LB.Bt; SUHB=0gHiV0NSuPLx_o; ALF=1509445224; SSOLoginState=1477909224; _s_tentry=-; Apache=8995918969370.725.1477909230337; ULV=1477909230350:605:126:1:8995918969370.725.1477909230337:1477672544934


这个接口返回的是参加这个pid=42077的所有用户信息 ,包括uid,ip地址,地理位置,参加的时间等。
有了这个,配合爬虫,遍历所有pid,就能得到全部参加粉丝夺宝的uid,这样利用上面的那个接口,就可以找到中奖的用户了。
写两个python的脚本配合一下。
第一个find_uid.py

#! /usr/bin/env python 
# coding=utf-8
# author=ntwu

import requests
import json
import sys
import time 
import threadpool   as tp

headers_fake = {
    "Host":"1.weibo.com",
    "Accept":"application/json",
    "X-Requested-With":"XMLHttpRequest",
    "Accept-Language":"zh-cn",
    "Accept-Encoding":"gzip, deflate",
    "Content-Type":"application/x-www-form-urlencoded",
    "Origin":"http://1.weibo.com",
    'Connection': 'close',
    "User-Agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A456 Safari/602.1",
    "Referer":"http://1.weibo.com/goodsunravel?pid=14875",
    "cookie":"ULV=1476417103515:6:4:4:5995306923612.945.1476417103443:1476368007922; _s_tentry=-; Apache=5995306923612.945.1476417103443; UOR=widget.weibo.com,; ALF=1476447308; SUB=_2A2563U8cDeTxGeVL71cR8i7JwzuIHXVWPlFUrDV8PUJbkNANLXHdkW15ppLDnd47HE2FVJkLel1wrmK7mA..; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WW8O9jiA56-HHn0H2.s1mvB5JpX5oz75NHD95Q0SKBfehz7SKnNWs4Dqc_zi--ciKL2iKy8i--fiKysi-8Fi--fiKnfi-i2i--fi-z7i-zpi--fiKLFiKLFi--ciKL2iKy8i--fiKysi-8Fi--fiKnfi-i2; SCF=Ar1U69gcKJHekMMzG5YnjaDnjG9TWgynF18HHlmDTXKIOHWqdHZYTqfhPhFaH7D1JUVf_uiSD153weX0aorAyhM.; SUHB=0M2F_WriiNbkqY; SINAGLOBAL=6988002809230.238.1461669767154",

}

url = "http://1.weibo.com/aj/goods/goodsactors"

pids = []

for i in range(14847,54847):
    pids.append(i)
code_status=""

time_start = time.time()

reload(sys)
sys.setdefaultencoding('utf-8')
def start(test):

    while True :
 pid = pids.pop()
 audiData = {
 "pid":pid,
 "page":1,
 "key":0,

 }

 while True:
     r =requests.post(url,headers=headers_fake,data=audiData,)
     all_data = json.loads(r.content)
     audiData['page']+=1
     if all_data['data'] == []:
  break
     else:
  for d  in all_data['data']:
      try:
   burp_success = open('duobao_account_uid_all.txt', 'a+')
   burp_success.write(d['uid']+"n")

   burp_success.close()
   time_end = time.time()

      except Exception,e:
   print all_data
   pass

args = [
    ['http://xxx.com', 'test'],
]
pool = tp.ThreadPool(200)
reqs = tp.makeRequests(start, args)
[pool.putRequest(req) for req in reqs]
pool.wait()

大约二十分钟后,

至此,就可以查看这些用户的中奖情况了。看下面这个脚本,find_won.py

#! /usr/bin/env python 
# coding=utf-8
# author=ljs

import codecs
import requests
import json
import sys
import threadpool   as tp
import time 

headers_fake = {
    'Host': '1.weibo.com',
    'Accept': 'application/json',
    'X-Requested-With': 'XMLHttpRequest',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'zh-cn',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Origin': 'http://1.weibo.com',
    'Content-Length': '30',
    'Connection': 'close',
    'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Mobile/14A456 Weibo (iPhone8,2__weibo__6.10.2__iphone__os10.0.2)',
    'Referer': 'http://1.weibo.com/profilehis?uid=1764571925',
    'cookie': '_s_tentry=-; Apache=7433541909010.6455.1477650674104; SINAGLOBAL=7433541909010.6455.1477650674104; ULV=1477650674175:1:1:1:7433541909010.6455.1477650674104:; SUB=_2A2569oNjDeThGeVL71cR8i7JwzuIHXVWazcrrDV8PUJbitANLWjdkWuBBHv6s4H45nFLilyDjupLYZMaCg..; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WW8O9jiA56-HHn0H2.s1mvB5NHD95Q0SKBfehz7SKnNWs4DqcjMi--NiK.Xi-2Ri--ciKnRi-zNe0-XSK5Eeh-RS7tt; SCF=AgXi0Twa0slZFI74Y0Pve7kDAPZKPPBjXl2tcaDxP29Frab512QavT429OPislnVrg..; SUHB=0QS53ljQ_62EeR',
    }
url = "http://1.weibo.com/aj/page/Profileother"
f_user = open('duobao_account_uid_34499.txt', 'r')
time_start = time.time()
reload(sys)
sys.setdefaultencoding('utf-8')
def start(test):
    flag =0
    for user in f_user.readlines():
 flag +=1
 postdata = {
 'uid':user[:-1],
 'type':'won',
 'page':'1',
 }
 requests.adapters.DEFAULT_RETRIES = 5
 r= requests.post(url,data=postdata,headers=headers_fake,timeout=5)
 all_data = json.loads(r.content)
 d = all_data['data']
 if d['list'] != []:
     luck = open('luck.txt','a+')
     success = codecs.open('won2.json' ,'a+',encoding='utf-8')
     line = json.dumps(d['list']) + "n"
     success.write(line.decode('unicode_escape'))
     success.write(user+"n")
     success.close()
     luck.write(user)
     luck.close()
     print("%s::%s"%(user,flag))

args = [
    ['http://xxx.com', 'test'],
]
pool = tp.ThreadPool(200)
reqs = tp.makeRequests(start, args)
[pool.putRequest(req) for req in reqs]
pool.wait()

这是这些用户的中奖情况,

以及中过奖用户的uid:

所以我觉得这真的是个运气游戏???
over!!

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/225904.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号