安裝
- 安裝庫
pip install scrapy-redis
- 安裝redis
配置
在setting.py文件中配置
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderQueue'
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
編寫spider
class MySpider(RedisSpider):
name = "myspider"
start_urls = ["http://weather.sina.com.cn/", ]
redis_key = "myspider:start_urls"
# 注意一定要重寫start_requests,否則出不來數(shù)據(jù)
def start_requests(self):
for url in self.start_urls:
yield Request(url=url, callback=self.parse)
def parse(self, response):
.....
上述中一定要注意重寫start_requests，不然一直沒數(shù)據(jù)出來，這是本人走過的坑，希望能幫到您，如果你覺得這篇文章對你有幫助，麻煩點個贊哦！
