scrapy

单节点的redis太容易挂了,总是出现意外情况,借助scrapy-redis-sentinel库实现scrapy_redis对redis集群的使用

按照官方库的说明去修改后启动项目会报错,报错的原因其实是作者的源码包中一个叫queue.py的文件中PriorityQueue类有一些逻辑上的问题

1、首先安装库

pip3 install scrapy-redis-sentinel

2、安装后修改queue.py源码

pip3 show scrapy-redis-sentinel   #查看源码包路径

进入包路径内

cd /usr/local/lib/python3.6/dist-packages/scrapy_redis_sentinel

修改queue.py中PriorityQueue类

class PriorityQueue(base):
    """Per-spider priority queue abstraction using redis' sorted set"""

    def __len__(self):
        """Return the length of the queue"""
        return self.server.zcard(self.key)

    def push(self, request):
        """Push a request"""
        data = self._encode_request(request)
        score = -request.priority
        # We don't use zadd method as the order of arguments change depending on
        # whether the class is Redis or StrictRedis, and the option of using
        # kwargs only accepts strings, not bytes.
        self.server.execute_command("ZADD", self.key, score, data)

    def pop(self, timeout=0):
        """
        Pop a request
        timeout not support in this queue class
        """
        if not isinstance(self.server, RedisCluster):
            # use atomic range/remove using multi/exec
            pipe = self.server.pipeline()
            pipe.multi()
            pipe.zrange(self.key, 0, 0).zremrangebyrank(self.key, 0, 0)
            results, count = pipe.execute()
            if results:
                return self._decode_request(results[0])

            # 使用集群的时候不能使用 multi/exec 来完成一个事务操作；使用lua脚本来实现类似功能
            pop_lua_script = """
            local result = redis.call('zrange', KEYS[1], 0, 0)
            local element = result[1]
            if element then
                redis.call('zremrangebyrank', KEYS[1], 0, 0)
                return element
            else
                return nil
            end
            """
            script = self.server.register_script(pop_lua_script)
            results = [script(keys=[self.key])]
            if results:
                return self._decode_request(results[0])

修改完成后保存退出,其实就是把原来的else去掉了把if往后缩进了
3、修改settings.py配置文件,我就截图下需要修改的地方

4、修改spider文件

5、启动项目

scrapy_redis配置redis集群成功