scrapy作為一個(gè)強(qiáng)大的爬蟲框架，就不多作介紹。
今天剛?cè)腴Tscrapy，所以做個(gè)簡(jiǎn)單的使用

Scrapy中文文檔

http://scrapy-chs.readthedocs.io/zh_CN/latest/index.html

Ubuntu安裝scrapy

在Ubuntu下安裝scrapy，需要的命令為：

pip install zope.interface
pip install twisted
pip install pyopenssl
pip install scrapy

開始

1. 使用scrapy創(chuàng)建文件夾命令：

scrapy startproject doubanTest

命令執(zhí)行完系統(tǒng)將在當(dāng)前目錄下生成doubanTest文件

2. 看看最后的代碼文件目錄（開始時(shí)創(chuàng)建不存在douban_spider.py，main.py文件）

3. 網(wǎng)頁(yè)提取的元素需要在items.py定義

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class DoubantestItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field() # 抓取電影標(biāo)題
    content = scrapy.Field() # 抓取電影簡(jiǎn)介

比如我在代碼中定義了抓取電影的標(biāo)題和電影的簡(jiǎn)介

關(guān)于item的使用，官方文檔這么描述:

Item用法類似字典dict

4. 在settings.py定義一些防反爬元素

# -*- coding: utf-8 -*-

# Scrapy settings for doubanTest project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'doubanTest'

SPIDER_MODULES = ['doubanTest.spiders']
NEWSPIDER_MODULE = 'doubanTest.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36'

FEED_URI = 'doubanmovie.csv' # 將信息存儲(chǔ)到doubanmovie.csv文件中
FEED_FORMAT = 'CSV' # 存儲(chǔ)的文件類型

5. 在spiders下創(chuàng)建douban_spider.py，編寫主要代碼

#-*_coding:utf8-*-

import scrapy
from scrapy.spider import Spider
from scrapy.http import Request
from scrapy.selector import Selector
from doubanTest.items import DoubantestItem

class doubanSpider(Spider):
    name="doubanSpider" # 爬蟲程序名稱
    start_urls = ['https://movie.douban.com/top250'] # URL訪問列表

    def parse(self, response):
        # print response.body
        item = DoubantestItem() # 定義一個(gè)Item對(duì)象
        selector = Selector(response)
        Movies = selector.xpath('//div[@class="info"]')
        for each in Movies:
            title = each.xpath('div[@class="hd"]/a/span/text()').extract() # 獲取電影標(biāo)題
            full_title = "".join(title)
            content = each.xpath('div[@class="bd"]/p/text()').extract() # 獲取電影簡(jiǎn)介

            # item類似dict, 它的key名稱必須與items.py里的名稱一致
            item['title'] = full_title
            item['content'] = ';'.join(content).strip()
            yield item

6. 在doubanTest目錄下創(chuàng)建一個(gè)main.py

編寫這兩句代碼，就可以使用pycharm運(yùn)行了

#-*_coding:utf8-*-
from scrapy import cmdline
cmdline.execute("scrapy crawl doubanSpider".split())

運(yùn)行main.py得到的結(jié)果

ok,就當(dāng)作scrapy入門例子吧

好好學(xué)習(xí)天天向上~