10.第十部分 Scrapy框架

文章目录

第十部分 Scrapy 框架
- 知识点一 : Scrapy 框架释义
- 知识点二 : Scrapy 框架学习目的
- 知识点三 : Scrapy框架安装
- 知识点四 : Scrapy框架入门使用
- 知识点五 : Scrapy框架提取数据
- - 1.parse函数 response响应对象的常用属性
  - 2.parse函数提取某蜓FM网站排行榜数据
  - - 2.1 第一步 : 新建并进入文件夹
    - 2.2 第二步 : 新建并进入项目
    - 2.3 第三步 : 新建爬虫并修改爬取地址
    - 2.4 第四步 : 代码实现提取数据并运行爬虫
- 知识点六 : Scrapy框架存储数据
- - 第一步 : 取消 settings.py 文件中被注释掉的代码
  - 第二步 : 在工程文件中添加上 yield 对象
  - 第三步 : 在管道文件中添加具体存储数据的细节
- 知识点七 : Scrapy框架整体流程
- 知识点八 : Scrapy框架数据多次下载
- - 第一步 : 配置 Scrapy 框架所需前置准备
  - 第二步 : 配置 qingting.py 文件
  - 第三步 : 配置 settings.py 文件
  - 第四步 : 配置 pipelines.py 文件
  - 第五步 : 运行 Scrapy 框架下的爬虫项目
- 知识点九 : 案例-爬取某瓣电影Top250

第十部分 Scrapy 框架知识点一 : Scrapy 框架释义

1.Scrapy 框架是一个为了爬取网站数据, 提取结构性数据而编写的应用框架, 我们只需要实现少量的代码, 就能快速抓取网站数据.

2.Scrapy 框架使用了 Twisted 异步网络框架, 可以加快我们的下载速度.

知识点二 : Scrapy 框架学习目的

1.普通爬虫 (requests, Xpath, selenium等) 爬取单一固定页面游刃有余, 但是爬取嵌套页面较为乏力. 而 Scrapy 框架能够满足我们对嵌套页面爬取的需求.

2.Scrapy 框架能够提高我们的爬虫效率, 让开发过程更加方便快速.

知识点三 : Scrapy框架安装

# Scrapy 框架安装命令
pip install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple

知识点四 : Scrapy框架入门使用

1.需求 : 爬取百度首页
2.流程 :
2.1 在终端使用命令进入到一个空文件夹: cd ......
2.2 在空文件夹路径下使用命令新建一个 Scrapy 项目: scrapy startproject <项目名>
2.3 进入新创建的项目: cd <项目名>
2.4 在项目中生成一个爬虫: scrapy genspider <爬虫名> <允许爬取的域名>

2.5 运行刚刚生成的爬虫: scrapy crawl <爬虫名>

知识点五 : Scrapy框架提取数据 1.parse函数 response响应对象的常用属性

属性	作用
response.url	当前响应的 url 地址
response.request.url	当前响应对应的请求的 url 地址
response.headers	响应头
response.request.headers	当前响应的请求头
response.body	响应体 ( 即 html 代码, 类型为 byte 类型 )
response.status	响应状态码

import scrapy


class BaiduSpider(scrapy.Spider):

    # 1.name:自定义的爬虫项目名称
    name = 'baidu'

    # 2.allowed_domains:允许爬取的域名
    allowed_domains = ['www.baidu.com']

    # 3.start_urls:启动时最开始爬取的url
    start_urls = ['http://www.baidu.com/']

    # 4.parse(解析) 函数: 用于数据的处理
    def parse(self, response):

        # 5.response 响应对象的常见属性
        print("----------->", response)
        print("-----------1>", response.url)
        print("-----------2>", response.headers)    # 在响应头中可以提取cookie
        print("-----------3>", response.body)       # 在响应体中国可以提取数据内容
        print("-----------4>", response.status)

2.parse函数提取某蜓FM网站排行榜数据 2.1 第一步 : 新建并进入文件夹

在 [2022_07_27]第十部分_Scrapy框架文件夹下创建一个新文件夹, 命名为02_爬取蜻蜓FM排行榜, 完成之后在终端使用命令进入到02_爬取蜻蜓FM排行榜文件夹中.

2.2 第二步 : 新建并进入项目

2.3 第三步 : 新建爬虫并修改爬取地址

2.4 第四步 : 代码实现提取数据并运行爬虫

import scrapy


class QingtingSpider(scrapy.Spider):
    name = 'qingting'
    allowed_domains = ['qingting.fm']
    start_urls = ['https://m.qingting.fm/rank/']

    def parse(self, response):

        # 1.存储提取到的30个a标签组成的列表对象
        a_list = response.xpath("//div[@class='rank-list']/a")

        # 2.使用for循环依次处理每一个a标签对象
        for a_temp in a_list:

            # 2.1 排名
            rank_number = a_temp.xpath("./div[@class='badge']/text()")
            # 2.2 图片地址
            img_src = a_temp.xpath("./img/@src")
            # 2.3 标题
            title = a_temp.xpath(".//div[@class='title']/text()")
            # 2.4 简介
            desc = a_temp.xpath(".//div[@class='desc']/text()")
            # 2.5 播放量
            play_number = a_temp.xpath(".//div[@class='info-item'][1]/span/text()")
            
            # 2.6 方法一: 直接返回一个类似 list 的类型
            # print("--------------->", rank_number, img_src, title, desc, play_number)
            
            # 2.6 方法二: 使用extract(),返回一个含有字符串的列表
            # print("--------------->", rank_number.extract(), img_src.extract(), title.extract(), desc.extract(), play_number.extract())
            
            # 2.6 方法三(推荐): 使用extract_first(),返回列表这个的第一个字符串,列表为空则返回None
            print("--------------->", rank_number.extract_first(), img_src.extract_first(), title.extract_first(), desc.extract_first(), play_number.extract_first())

知识点六 : Scrapy框架存储数据第一步 : 取消 settings.py 文件中被注释掉的代码

第二步 : 在工程文件中添加上 yield 对象

Q: 为什么使用 yield 而非 return ?
A: 第一个原因是使用 return 则需要用一个列表去接收想要传递的数据,会占用较高的内存;第二个原因是使用return接收,for循环执行时其他任务无法执行,而通过 yield 能够将运行权限交给其他的协程任务去执行, 整个程序效率会更高.

Q: yield能传递的对象有哪些?
A: 解析函数中的yield能够传递的对象只能是 : Baseltem, Request, dict, None

第三步 : 在管道文件中添加具体存储数据的细节

知识点七 : Scrapy框架整体流程

上图1~12序号对应的释义如下:

① Scrapy 从 Spider(蜘蛛) 子类中提取 start_urls, 然后构造为 request 请求对象;

② 将 request 请求对象传递给爬虫中间件

③ 将 request 请求对象传递给 Scrapy 引擎 (即核心代码)

④ 将 request 请求对象传递给调度器 (调度器负责多个request的安排, 好比交警负责交通的指挥, 避免道路拥堵)

⑤ 将 request 请求对象传递给 Scrapy 引擎

⑥ Scrapy 引擎将 request 请求对象传递给下载中间件 (可以更换代理IP, 更换Cookies, 更换User-Agent, 自动重试等)

⑦ request请求对象传给到下载器 (它通过异步的发送HTTP(s)请求), 得到响应封装为 response 对象

⑧ 将 response 对象传递给下载中间件

⑨ 下载中间件将 response 对象传递给 Scrapy 引擎

⑩ Scrapy 引擎将 response 对象传递给爬虫中间件 (这里可以处理异常等情况)

⑪ 爬虫对象中的 parse 函数被调用 (在这里可以对得到的 response 对象进行处理, 例如用 status 得到响应状态码,Xpath表达式提取数据)

⑫ 将提取到的数据传递给 Scrapy 引擎, 它将数据再传递给管道 (在管道中我们可以将数据存储到 csv、MongoDB、MySQL 等)

知识点八 : Scrapy框架数据多次下载

需求 : 使用 Scrapy 框架爬取蜻蜓FM听书排行榜前30名书籍相关数据并下载对应的书籍对应封面.

第一步 : 配置 Scrapy 框架所需前置准备

第二步 : 配置 qingting.py 文件

import scrapy


class QingtingSpider(scrapy.Spider):
    name = 'qingting'

    # 设置起始域名为空,目的是防止yield scrapy.Requeste()不能把数据传到下一个解析函数日志报错:DEBUG: Filtered offsite request to xxx
    allowed_domains = []
    start_urls = ['https://m.qingting.fm/rank/']

    def parse(self, response):

        # 1.存储提取到的30个a标签组成的列表对象
        a_list = response.xpath("//div[@class='rank-list']/a")

        # 2.使用for循环依次处理每一个a标签对象
        for a_temp in a_list:

            # 2.1 排名
            rank_number = a_temp.xpath("./div[@class='badge']/text()").extract_first()
            # 2.2 图片地址
            img_src = a_temp.xpath("./img/@src").extract_first()
            # 2.3 标题
            title = a_temp.xpath(".//div[@class='title']/text()").extract_first()
            # 2.4 简介
            desc = a_temp.xpath(".//div[@class='desc']/text()").extract_first()
            # 2.5 播放量
            play_number = a_temp.xpath(".//div[@class='info-item'][1]/span/text()").extract_first()

            # 2.6 使用extract_first(),返回列表这个的第一个字符串,列表为空则返回None
            # print("--------------->", rank_number.extract_first(), img_src.extract_first(), title.extract_first(), desc.extract_first(), play_number.extract_first())

            # 生成的是信息 (需要在管道中进行处理)
            yield {
                "type": "info",
                "rank_number": rank_number,
                "img_src": img_src,
                "title": title,
                "desc": desc,
                "play_number": play_number
            }

            # 生成了一个请求对象
            yield scrapy.Request(url=img_src, callback=self.parse_img, cb_kwargs={"img_name": title})


    def parse_img(self, response, img_name):
        print("---2-->", response.url)

        # 图片的二进制数据:response.body
        yield {
            "type": "img",
            "img_name": img_name + ".png",
            "img_bytes": response.body
        }

Q: 是否可以有多个 parse 函数?
A: 可以, 只要处理的回调函数当中有新的请求发生, 我们将来就可以进行下载.

第三步 : 配置 settings.py 文件

第四步 : 配置 pipelines.py 文件

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter


class Fm2RankPipeline:
    def process_item(self, item, spider):

        # 提取类型
        msg_type = item.get("type")
        if msg_type == "img":
            # 如果 item 中存储的是图片二进制等数据, 则应创建一个文件,然后将二进制输入写入到文件, 即可完成图片的保存
            # "w":文本写入; "wb":二进制写入
            print("正在保存到图片文件...")
            path = item.get("img_name").replace('?', '').replace('*', '').replace('<', '').replace('> ', '').replace('|', ' ')
            print(path)
            with open(path, "wb") as fp:
                fp.write(item.get("img_bytes"))

        elif msg_type == "info":
            # 可以将30条信息存储到CSV文件
            print("正在存储到文件...")
            with open("./qingting.csv", "a") as fp:
                fp.write(item.get("rank_number"))
                fp.write(item.get("title"))
                fp.write(item.get("desc"))
                fp.write(item.get("play_number"))
                fp.write("n")

第五步 : 运行 Scrapy 框架下的爬虫项目

知识点九 : 案例-爬取某瓣电影Top250

① 创建一个文件夹命名为 03_爬取豆瓣电影Top250

② 在终端通过命令进入文件夹03_爬取豆瓣电影Top250

PS D:...[2022_07_27]第十部分_Scrapy框架> cd .3_爬取豆瓣电影Top250
PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250>

③ 在终端通过命令终端新建一个 Scrapy 项目

PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250> scrapy startproject douban
New Scrapy project 'douban', using template directory 'C:Users32495Envspython3.9.2libsite-packagesscrapytemplatesproject', created in:
    D:softwareCode0_CodePython代码爬虫路线学习[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250douban
                                                                                                                
You can start your first spider with:                                                                           
    cd douban                                                                                                   
    scrapy genspider example example.com                                                                        
PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250>

④ 进入刚刚新建的 Scrapy 项目

PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250> cd .douban               
PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250douban>

⑤ 在新建的 Scrapy 项目下生成一个爬虫

PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250douban> scrapy genspider top250 douban.com
Created spider 'top250' using template 'basic' in module:
  douban.spiders.top250                                  
PS D:...[2022_07_27]第十部分_Scrapy框架3_爬取豆瓣电影Top250douban>

⑥ 配置 spiderstop250.py文件 : 提取数据

# -*-coding:utf-8 -*-

import scrapy


class Top250Spider(scrapy.Spider):
    name = 'top250'
    # allowed_domains = []
    allowed_domains = ['douban.com', 'doubanio.com']
    start_urls = ['https://movie.douban.com/top250?start=0&filter=']

    # 记录当前要爬取的页数
    page = 1

    def parse(self, response):
        print("----回调函数被执行----")

        # 对start_urls中的url对应的响应进行处理
        # response.url/headers/body/request.url/status
        li_list = response.xpath("//ol[@class='grid_view']/li")
        for li_temp in li_list:
            rank = li_temp.xpath(".//div[@class='pic']/em/text()").extract_first()
            title = li_temp.xpath(".//span[@class='title'][1]/text()").extract_first()
            rating_num = li_temp.xpath(".//span[@class='rating_num']/text()").extract_first()
            people_num = li_temp.xpath(".//div[@class='star']/span[4]/text()").extract_first()
            img_src = li_temp.xpath(".//img/@src").extract_first()
            print("-------->", rank, title, rating_num, people_num, img_src)


            # 传递给 pipelines 文件里面的存储代码去执行
            # 生成信息,用来传递给管道让其保存到CSV文件
            yield {
                "type": "info",
                "rank": rank,
                "title": title,
                "rating_num": rating_num,
                "people_num": people_num,
                "img_src": img_src
            }

            # 生成新的Request对象 (实现下载图片)
            yield scrapy.Request(url=img_src, callback=self.parse_img, cb_kwargs={"img_name": title})


            # 翻页操作 (提取下一页URL)
            # next_page_url = response.xpath("//[@class='next']/a/@href").extract_first()
            # next_page_url = "https://movie.douban.com/top250" + next_page_url

            # 生成下一页的地址
            self.page += 1
            if self.page <= 10:
                next_page_url = "https://movie.douban.com/top250?start=%d&filter=" % ((self.page-1) * 25)
                yield scrapy.Request(url=next_page_url, callback=self.parse)

    def parse_img(self, response, img_name):
        print("------图片处理的回调函数------")
        # 生成图片信息
        yield {
            "type": "img",
            "img_name": img_name + ".jpg",
            "img_bytes": response.body
        }

⑦ 配置 doubanpipelines.py文件 : 存储数据

# -*-coding:utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
import csv
import os

from itemadapter import ItemAdapter


class DoubanPipeline:
    def process_item(self, item, spider):

        # 判断是否存在download文件夹,如果没有则创建
        download_path = os.getcwd() + "/download/"   # 当前文件夹下的download文件夹
        if not os.path.exists(download_path):       # 判断文件夹或文件
            os.makedirs(download_path)

        msg_type = item.get("type")
        if msg_type == "info":
            # 信息
            # 如果类型是信息,则保存到csv文件
            with open(download_path + "豆瓣电影Top250.csv", "a") as f:

                # 创建一个csv的DictWriter对象,这样才能够将希尔csv格式数据到这个文件
                f_csv = csv.DictWriter(f, ["rank", "title", "rating_num", "people_num", "img_src"])
                # 写入多行行 (当做数据)
                item.pop("type")    # 删除type 这个key-value
                f_csv.writerows([item])
                print("保存信息到CSV...ok")

        elif msg_type == "img":
            # 存储图片
            with open(download_path + item.get("img_name"), "wb") as f:
                f.write(item.get("img_bytes"))
                print("保存图片到本地...ok")
        return item

⑦ 配置 doubanmiddlewares.py文件 : 中间件伪装爬虫

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
import random

from scrapy import signals

# useful for handling different item types with a single interface
from itemadapter import is_item, ItemAdapter


class DoubanSpiderMiddleware:
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the spider middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_spider_input(self, response, spider):
        # Called for each response that goes through the spider
        # middleware and into the spider.

        # Should return None or raise an exception.
        return None

    def process_spider_output(self, response, result, spider):
        # Called with the results returned from the Spider, after
        # it has processed the response.

        # Must return an iterable of Request, or item objects.
        for i in result:
            yield i

    def process_spider_exception(self, response, exception, spider):
        # Called when a spider or process_spider_input() method
        # (from other spider middleware) raises an exception.

        # Should return either None or an iterable of Request or item objects.
        pass

    def process_start_requests(self, start_requests, spider):
        # Called with the start requests of the spider, and works
        # similarly to the process_spider_output() method, except
        # that it doesn’t have a response associated.

        # Must return only requests (not items).
        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)


class DoubanDownloaderMiddleware:
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None

    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)


# 操作:自定义User-Agent中间件(中间件:简单来理解即用来伪装Scrapy的一个步骤,用代码来实现)
class UserAgentMiddleware(object):

    # 定义一个列表,其中包含大量浏览器的版本信息(User-Agent池)
    USER_AGENTS_LIST = [
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_0) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.79 Safari/537.4',
        'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_0; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/4.0.206.1 Safari/532.0',
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20110517 Firefox/5.0 Fennec/5.0',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 Camino/2.2.1',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.1) Gecko/20110318 Firefox/4.0b13pre Fennec/4.0',
        'Mozilla/5.0 (Windows NT 6.0; rv:2.1.1) Gecko/20110415 Firefox/4.0.2pre Fennec/4.0.1',
        'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b2pre) Gecko/20081015 Fennec/1.0a1',
        'Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.66 Safari/535.11 ',
        'Chrome/15.0.860.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/15.0.860.0',
        'Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1',
        "Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)",
        "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.0 Safari/532.5",
        "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.310.0 Safari/532.9",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7",
        "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14",
        "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14",
        "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.27 (KHTML, like Gecko) Chrome/12.0.712.0 Safari/534.27",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1",
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
        "Mozilla/5.0 (Windows; U; Windows NT 6.0 x64; en-US; rv:1.9pre) Gecko/2008072421 Minefield/3.0.2pre",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10",
        "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)",
        "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E)",
        "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
        "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
        "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110622 Firefox/6.0a2",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b4pre) Gecko/20100815 Minefield/4.0b4pre",
        "HTC_Dream Mozilla/5.0 (Linux; U; Android 1.5; en-ca; Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.2; U; de-DE) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/234.40.1 Safari/534.6 TouchPad/1.0",
        "Mozilla/5.0 (Linux; U; Android 1.5; en-us; sdk Build/CUPCAKE) AppleWebkit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 1.5; en-us; htc_bahamas Build/CRB17) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (Linux; U; Android 2.1-update1; de-de; HTC Desire 1.19.161.5 Build/ERE27) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 1.5; de-ch; HTC Hero Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 2.1; en-us; HTC Legend Build/cupcake) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 1.5; de-de; HTC Magic Build/PLAT-RC33) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 FirePHP/0.3",
        "Mozilla/5.0 (Linux; U; Android 1.6; en-us; HTC_TATTOO_A3288 Build/DRC79) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (Linux; U; Android 1.0; en-us; dream) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
        "Mozilla/5.0 (Linux; U; Android 1.5; en-us; T-Mobile G1 Build/CRB43) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari 525.20.1",
        "Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile_G2_Touch Build/CUPCAKE) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Milestone Build/ SHOLS_U2_01.03.1) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 2.0.1; de-de; Milestone Build/SHOLS_U2_01.14.0) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
        "Mozilla/5.0 (Linux; U; Android 0.5; en-us) AppleWebKit/522 (KHTML, like Gecko) Safari/419.3",
        "Mozilla/5.0 (Linux; U; Android 1.1; en-gb; dream) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
        "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 2.2; en-ca; GT-P1000M Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
        "Mozilla/5.0 (Linux; U; Android 3.0.1; fr-fr; A500 Build/HRI66) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13",
        "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10 (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
        "Mozilla/5.0 (Linux; U; Android 1.6; es-es; SonyEricssonX10i Build/R1FA016) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        "Mozilla/5.0 (Linux; U; Android 1.6; en-us; SonyEricssonX10i Build/R1AA056) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1"
    ]

    def process_request(self, request, spider):

        print("-----下载中间件-----")

        # 随机挑选一个UA
        user_agent = random.choice(self.USER_AGENTS_LIST)
        request.headers['User-Agent'] = user_agent
        return None # 如果返回为None,那么意味着将request继续传递给下载器去下载


    def process_response(self, request, response, spider):
        print("-----下载中间件---response--")
        return response


# 操作:设置IP代理中间件(搭建IP代理池绕过网页反爬检测)
# class ProxyMiddleware(object):
#
#     proxy_pool = [
#         'http://112.29.178.9:29947',
#         'http://112.28.231.247:36192',
#         'http://112.29.178.14:30045',
#         'http://112.29.178.91:23512',
#         'http://112.29.178.3:35156',
#         'http://112.29.178.11:39683',
#         'http://112.29.178.93:34701',
#         'http://112.28.231.248:27003'
#     ]
#
#     def process_request(self, request, spider):
#
#         # proxies可以在settings.py文件中,也可以来源于代理ip的webapi
#         print("-----下载中间件-request-----")
#         request.meta["proxy"] = random.choice(self.proxy_pool)
#         return None
#
#     def process_response(self, request, response, spider):
#         print("-----下载中间件-response-----", response.status)
#         if response.status != 200:
#             request.dont_filter = True  # 重新发送的请求对象能够再次进入队列
#             return request
#         return response

⑧ 配置 doubansettings.py文件 : 整合scrapy框架

# Scrapy settings for douban project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'douban'

SPIDER_MODULES = ['douban.spiders']
NEWSPIDER_MODULE = 'douban.spiders'

# 操作1:将Scrapylog日志调整为最低级别,即 waring
LOG_LEVEL = "WARNING"

# 操作2:设置两页下载间的时间间隔 (防止爬取频率过快被网站检测为爬虫)
DOWNLOAD_DELAY = 10   # Scrapy不会等待固定的3秒,而是等待它乘以 0.5~1.5 的时间 (5~15秒钟之间随机)

# 操作3:设置允许403状态码 (HTTP 403:表示服务器获取到了请求, 但是拒绝提供服务)
HTTPERROR_ALLOWED_CODES = [403]

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'douban (+http://www.yourdomain.com)'

# Obey robots.txt rules

# 操作4:设置不遵循robot协议
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html

# 操作1:开启爬虫中间件并配置
# SPIDER_MIDDLEWARES = {
#    'douban.middlewares.DoubanSpiderMiddleware': 543,
# }

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html

# 操作:开启下载器中间件并配置
DOWNLOADER_MIDDLEWARES = {
   # 'douban.middlewares.ProxyMiddleware': 350,
   'douban.middlewares.UserAgentMiddleware': 400,       # 这里的数值是权重,可自定义,权重的数值越小权重越大(即数值越小约先执行)
   'douban.middlewares.DoubanDownloaderMiddleware': 543
}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
# EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
# }

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html

# 操作5:开启管道 (即top250.py中提取到的数据可以通过yield传递给pipelines.py文件去执行存储到数据库或者是本地csv)
ITEM_PIPELINES = {
   'douban.pipelines.DoubanPipeline': 300,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

10.第十部分 Scrapy框架

Python相关栏目本月热门文章