Scrapy的文档在给出两者的实际应用示例方面做得非常糟糕。
CrawlerProcess假设scrapy是唯一使用twisted反应堆的东西。如果你在python中使用线程来运行其他代码，则并非总是如此。让我们以此为例。
from scrapy.crawler import CrawlerProcessimport scrapydef notThreadSafe(x): """do something that isn't thread-safe""" # ...class MySpider1(scrapy.Spider): # Your first spider definition ...class MySpider2(scrapy.Spider): # Your second spider definition ...process = CrawlerProcess()process.crawl(MySpider1)process.crawl(MySpider2)process.start() # the script will block here until all crawling jobs are finishednotThreadSafe(3) # it will get executed when the crawlers stop
现在，你可以看到，该函数仅在搜寻器停止时才执行，如果我希望在搜寻器在同一反应堆中爬行时执行该函数，该怎么办？
from twisted.internet import reactorfrom scrapy.crawler import CrawlerRunnerimport scrapydef notThreadSafe(x): """do something that isn't thread-safe""" # ...class MySpider1(scrapy.Spider): # Your first spider definition ...class MySpider2(scrapy.Spider): # Your second spider definition ...runner = CrawlerRunner()runner.crawl(MySpider1)runner.crawl(MySpider2)d = runner.join()d.addBoth(lambda _: reactor.stop())reactor.callFromThread(notThreadSafe, 3)reactor.run() #it will run both crawlers and pre inside the function
Runner类不限于此功能
