內(nèi)容來源于網(wǎng)絡(luò)，本人只是在此稍作整理，如有涉及版權(quán)問題，歸小甲魚官方所有。

0.請寫下這一節(jié)課你學習到的內(nèi)容：格式不限，回憶并復(fù)述是加強記憶的好方式！

• 修改User-Agent
  下圖是urllib.request.Request部分有關(guān)于設(shè)置User-Agent的敘述。

設(shè)置User-Agent.png

設(shè)置這個headers參數(shù)有兩種辦法：
1??實例化Request參數(shù)時將headers參數(shù)傳進去。

import urllib.request
import urllib.parse
import json

content = input("請輸入需要翻譯的內(nèi)容：")
url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule&smartresult=ugc&sessionFrom=http://www.youdao.com"
head = {}
head['Referer'] = 'http://fanyi.youdao.com/'
head['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'

data = {}
data['type'] = 'AUTO'
data['i'] = content
data['doctype'] = 'json'
data['version'] = '2.1'
data['keyfrom'] = 'fanyi.web'
data['ue'] = 'UTF-8'
data['typoResult'] = 'true'

data = urllib.parse.urlencode(data).encode('utf-8')
req = urllib.request.Request(url, data, headers=head)
response = urllib.request.urlopen(req)
html = response.read().decode('utf-8')
target = json.loads(html)
print("翻譯結(jié)果：%s" % (target['translateResult'][0][0]['tgt']))
print(req.headers)

輸出：

請輸入需要翻譯的內(nèi)容：我愛你
翻譯結(jié)果：I love you
{'Referer': 'http://fanyi.youdao.com/', 'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}

2??通過add_header()方法往Request對象添加headers。

import urllib.request
import urllib.parse
import json

content = input("請輸入需要翻譯的內(nèi)容：")
url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule&smartresult=ugc&sessionFrom=http://www.youdao.com"
data = {}
data['type'] = 'AUTO'
data['i'] = content
data['doctype'] = 'json'
data['version'] = '2.1'
data['keyfrom'] = 'fanyi.web'
data['ue'] = 'UTF-8'
data['typoResult'] = 'true'

data = urllib.parse.urlencode(data).encode('utf-8')
req = urllib.request.Request(url, data)
req.add_header('Referer', 'http://fanyi.youdao.com')
req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36')
response = urllib.request.urlopen(req)
html = response.read().decode('utf-8')
target = json.loads(html)
print("翻譯結(jié)果：%s" % (target['translateResult'][0][0]['tgt']))
print(req.headers)

輸出：

請輸入需要翻譯的內(nèi)容：I want to go back.
翻譯結(jié)果：我想回去。
{'Referer': 'http://fanyi.youdao.com', 'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}

• 如果同一個IP地址在短時間內(nèi)對服務(wù)器訪問很頻繁，顯然這不是正常人操作的，所以服務(wù)器會設(shè)置每個IP的訪問頻率，一旦超過這個闕值，便認為就是爬蟲，于是返回一個驗證碼頁面，要求用戶填寫驗證碼，爬蟲自然就不能正常爬取信息了，所以被拒絕。所以想到有2種方法可以解決這種情況：

1??延遲提交的時間；

import urllib.request
import urllib.parse
import json
import time

url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule&smartresult=ugc&sessionFrom=http://www.youdao.com"
while True:
    content = input('請輸入待翻譯的內(nèi)容（輸入"q!"退出程序")：')
    if content == 'q!':
        break
    data = {}
    data['type'] = 'AUTO'
    data['i'] = content
    data['doctype'] = 'json'
    data['version'] = '2.1'
    data['keyfrom'] = 'fanyi.web'
    data['ue'] = 'UTF-8'
    data['typoResult'] = 'true'
    data = urllib.parse.urlencode(data).encode('utf-8')
    req = urllib.request.Request(url, data)
    req.add_header('Referer', 'http://fanyi.youdao.com')
    req.add_header('User-Agent',
                   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36')
    response = urllib.request.urlopen(req)
    html = response.read().decode('utf-8')
    target = json.loads(html)
    print("翻譯結(jié)果：%s" % (target['translateResult'][0][0]['tgt']))
    time.sleep(5)  # 5秒

2??使用代理；

import urllib.request

url = 'http://www.whatismyip.com.tw/'
proxy_support = urllib.request.ProxyHandler({'http':'211.138.121.38:80'})
# 接著創(chuàng)建一個包含代理IP的opener
opener = urllib.request.build_opener(proxy_support)
# 安裝進默認環(huán)境
urllib.request.install_opener(opener)
# 試試看IP地址改了沒
response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')
print(html)

import urllib.request
import random

url = 'http://www.whatismyip.com.tw/'
print("添加代理IP地址（IP：端口號），多個IP地址間用英文的分號隔開！")
iplist = input("請開始輸入：").split(sep=";")
while True:
    ip = random.choice(iplist)
    proxy_support = urllib.request.ProxyHandler({'http':ip})
    opener = urllib.request.build_opener(proxy_support)
    opener.addheaders = [('User-Agent',
                   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36')]
    urllib.request.install_opener(opener)
    try:
        print("正在嘗試使用%s訪問..." % ip)
        response = urllib.request.urlopen(url)
    except urllib.error.URLError:
        print("訪問出錯！")
    else:
        print("訪問成功！")
    if input("請問是否繼續(xù)？（Y/N）") == 'N':
        break