【无标题】爬取网页时，报错出现404或者418问题

出现该错误其实就是，该网页发现这是一个爬虫程序被识别出来，阻止爬取数据。

解决方法：

1）：添加headers代码段。

    headers={
        'User-Agent':'Mozilla / 5.0(Windows NT 10.0;WOW64) AppleWebKit / 537.36(KHTML, likeGecko) Chrome / 86.0.4240.198 Safari /537.36'
    }

如果仍然出现错误，可能是访问次数太多而被网页识别，需要在网页中找到

Cookie 属性并添加到 headers中

 headers={
        "Cookie":r'bid=VABWdOG3BkA; __gads=ID=bf48cfe5ad9d9b1e-22c4decb60d30092:T=1653376675:RT=1653376675:S=ALNI_MZ3tr9MHp7bM0zUAX6c5nA3rsD1og; dbcl2="241813841:thsTjwJikfg"; push_noty_num=0; push_doumail_num=0; ll="118249"; _vwo_uuid_v2=DB03541594719B9B979B8ED3B8B859669|64f62eb1f394df36dbb536f5aedeb5da; __gpi=UID=000005b1b4c0a84e:T=1653376675:RT=1655024145:S=ALNI_MYs_2CnRxoTQ5u1Lg1sWDL2ypG38Q; __utmv=30149280.24181; ck=2BJN; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1655039749%2C%22https%3A%2F%2Fwww.so.com%2Flink%3Fm%3Db0XJTAgNh0MrPrutz7%252FZIaLDSADX9Mz%252BjxqlALn2JyKMN5XK%252B6P4fKVeEcvwI6RFIkCoqwSjGfwIG7nSMB%252F%252FaCvIJ%252BMUuXQnJsPfvKkIoCABwc4FbApS9iQ%253D%253D%22%5D; _pk_id.100001.4cf6=1b200cbfe27faf34.1653376674.6.1655039749.1655024170.; _pk_ses.100001.4cf6=*; __utma=30149280.97809329.1653376674.1655024145.1655039749.6; __utmb=30149280.0.10.1655039749; __utmc=30149280; __utmz=30149280.1655039749.6.6.utmcsr=so.com|utmccn=(referral)|utmcmd=referral|utmcct=/link; __utma=223695111.52041398.1653376674.1655024145.1655039749.6; __utmb=223695111.0.10.1655039749; __utmc=223695111; __utmz=223695111.1655039749.6.6.utmcsr=so.com|utmccn=(referral)|utmcmd=referral|utmcct=/link',
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"
   }

添加之后应该就可以正常运行了。