創(chuàng)建一個Scrapy項目

scrapy startproject myPorject

Scrapy的項目結(jié)構(gòu)

image.png

spiders:編寫爬蟲的目錄

爬蟲的編寫規(guī)則

import scrapy
from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):
  name = 'dmoz' #爬蟲唯一名稱
  allowed_domains = ['dmoz-odp.org'] #爬取主路徑
  start_urls = [
    'https://dmoz-odp.org/Computers/Programming/Languages/Python/Books/',
    'https://dmoz-odp.org/Computers/Programming/Languages/Python/Resources/'
  ] #需要爬取的路徑

  def parse(self, response):#爬取成功后的回調(diào)函數(shù)
    sel = scrapy.selector.Selector(response)
    sites = sel.xpath('//*[@id="site-list-content"]/div/div[3]')
    items = []
    for site in sites:
      item = DmozItem()
      item['title'] = site.xpath('a/div/text()').extract()
      item['link'] = site.xpath('a/@href').extract()
      item['desc'] = site.xpath('div/text()').extract()
      items.append(item)
    print(items)
    return items

運行你的爬蟲

 scrapy crawl name