Scrapy反爬設(shè)置（User-Agent、IPProxy、COOKIE）

一、Scrapy設(shè)置隨機(jī)User-Agent

1. settings.py文件中，設(shè)置隨機(jī)User-Agent

MY_USER_AGENT = [

? ? "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

? ? "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",

? ? "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

? ? "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",

? ? "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",

? ? "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",

? ? "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",

? ? "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",

? ? "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",

? ? "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",

? ? "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",

? ? "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",

? ? "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",

? ? "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",

? ? "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",

? ? "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",

? ? "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",

? ? "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",

? ? "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",

? ? "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",

? ? "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",

? ? "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",

? ? "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",

? ? "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",

? ? "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",

? ? "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",

? ? "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",

? ? "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",

? ? "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",

? ? "Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",

? ? "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",

? ? "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",

? ? "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",

? ? "Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10",

? ? "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",

? ? ]

2、在middlewares.py文件中，添加下面的代碼。

import scrapy

from scrapy import signals

from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware

import random

class MyUserAgentMiddleware(UserAgentMiddleware):

? ? def __init__(self, user_agent):

? ? ? ? self.user_agent = user_agent

? ? @classmethod

? ? def from_crawler(cls, crawler):

? ? ? ? return cls(

? ? ? ? ? ? user_agent=crawler.settings.get('MY_USER_AGENT')

? ? ? ? )

? ? def process_request(self, request, spider):

? ? ? ? agent = random.choice(self.user_agent)

? ? ? ? request.headers['User-Agent'] = agent

3、將我們自定義的這個(gè)MyUserAgentMiddleware類添加到DOWNLOADER_MIDDLEWARES，像下面這樣

DOWNLOADER_MIDDLEWARES = {

? ? 'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,

? ? 'myproject.middlewares.MyUserAgentMiddleware': 400,

}

二、Scrapy設(shè)置隨機(jī)IP代理（IPProxy）

1、settings.py文件中，設(shè)置代理列表

PROXIES = ['http://183.207.95.27:80', 'http://111.6.100.99:80', 'http://122.72.99.103:80',

? ? ? ? ? 'http://106.46.132.2:80', 'http://112.16.4.99:81', 'http://123.58.166.113:9000']

2. 在middlewares.py文件中，添加下面的代碼。

import scrapy

from scrapy import signals

import random

class ProxyMiddleware(object):

? ? def __init__(self, ip):

? ? ? ? self.ip = ip

? ? @classmethod

? ? def from_crawler(cls, crawler):

? ? ? ? return cls(ip=crawler.settings.get('PROXIES'))

? ? def process_request(self, request, spider):

? ? ? ? ip = random.choice(self.ip)

? ? ? ? request.meta['proxy'] = ip

3、將我們自定義的ProxyMiddleware類添加到DOWNLOADER_MIDDLEWARES，像下面這樣

DOWNLOADER_MIDDLEWARES = {

? ? 'myproject.middlewares.ProxyMiddleware': 543,

}

三、cookie設(shè)置

request_with_cookies = Request(url="http://www.example.com",

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? cookies={'currency': 'USD', 'country': 'UY'})

詳情：https://doc.scrapy.org/en/latest/topics/request-response.html


四、http status code 設(shè)置（http status code is not handled or allowed 解決方法）

1、在項(xiàng)目settings.py中添加

HTTPERROR_ALLOWED_CODES = [301,302,404]

2、專對(duì)爬蟲(chóng)設(shè)置

class TothegoSpider(Spider):

? ? handle_httpstatus_list = [301,302,404]

Set Handle_httpstatus_list = [302];

On request, set dont_redirect to True on meta.

For example: Request(URL, meta={'dont_redirect': True});

3、request加上meta={'handle_httpstatus_list': range(400,600)} （這個(gè)牛叉）

五、在你的爬蟲(chóng)里刪除 allowed_domains屬性（我在寫爬蟲(chóng)的過(guò)程也發(fā)現(xiàn)這個(gè)屬性是個(gè)雞肋）

Remove Domain LimitationPermalink

Our spider has an attribute called allowed_domains to prevent downloading unwanted URLs. Without this attribute, the spider may attempt to traverse the entire web and never complete its task.

參考：https://www.linode.com/docs/development/python/use-scrapy-to-extract-data-from-html-tags/

六、重寫爬蟲(chóng)的__init__ 還實(shí)現(xiàn) 通過(guò)命令行傳遞start_url

import re

from urllib.parse import urlparse

import scrapy

from scrapy import signals

class LinkCheckerSpider(scrapy.Spider):

? ? name = 'link_checker'

? ? handle_httpstatus_list = [404]

? ? domain = ''

? ? def __init__(self, url='http://www.example.com', *args, **kwargs):

? ? ? ? super(LinkCheckerSpider, self).__init__(*args, **kwargs)

? ? ? ? self.start_urls = [url]

scrapy crawl link_checker -a url="http://www.linode.com"

參考：https://www.linode.com/docs/development/python/use-scrapy-to-extract-data-from-html-tags/


七、error的處理

yield scrapy.Request(url, callback=self.parse_httpbin,

? ? ? ? ? ? ? ? ? ? ? ? ? errback=self.errback_httpbin,

? ? ? ? ? ? ? ? ? ? ? ? ? dont_filter=True)


彩蛋：

scrapy默認(rèn)是遵守爬蟲(chóng)準(zhǔn)則的，即settings里面，ROBOTSTXT_OBEY = True。

比如抓取百度，在https://www.baidu.com/robots.txt里，有這樣的一個(gè)規(guī)范。

如果遵守，比如今日頭條，是不能用scrapy爬取的。

這個(gè)時(shí)候需要把ROBOTSTXT_OBEY=False.也就不遵守它的規(guī)則。

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容