Splash抓取动态页面

一、安装

        环境：Ubuntu18.04

        1）安装docker

        2）下拉splash镜像

sudo docker pull scrapinghub/splash

        3) 启动

sudo docker run -p 8050:8050 scrapinghub/splash

        4)访问http://服务器ip:8050/验证

 二、基本使用

import requests


def splash_render(url):
    splash_url = "http://服务器ip:8050/render.html"

    args = {
        "url": url,
        "timeout": 10,
        "image": 0,
    }

    response = requests.get(splash_url, params=args)
    return response.text


if __name__ == '__main__':
    url = "http://quotes.toscrape.com/js/"
    html = splash_render(url)
    with open("test.html", "w", encoding="utf-8") as w:
        w.write(html)