廢話不多說，首先上圖：

抓取結(jié)果.png

抓取圖片.png

這是抓取了一個(gè)多小時(shí)的結(jié)果，代碼沒有做過優(yōu)化，也沒用多線程、分布式，就用最簡單的結(jié)構(gòu)爬取，速度大概在3500條/小時(shí)。第一張圖片展示的是數(shù)據(jù)庫中抓取的信息（姓名、城市、身高、體重、個(gè)人主頁url、圖片url），第二張展示的是保存下來的信息（*.txt + *.jpg）。

下面講一下爬取過程。按步驟來

1、目標(biāo)網(wǎng)頁分析。淘寶美人庫

網(wǎng)站頁面圖

淘女郎-美人庫.png

用chrome的頁面檢查工具(F12)查看頁面加載過程

F12.png

發(fā)現(xiàn)首頁鏈接的Response中并沒有返回頁面內(nèi)容，因?yàn)榫W(wǎng)站采用的是動態(tài)加載，那我們就開始找內(nèi)容請求的Request

Request.png

在眾多Request中我們找到了如上圖紅圈中的Request，這就是我們要找的頁面內(nèi)容請求Request，返回的Response是json，看一下請求信息

Request-General.png

Request-Params.png

這是一個(gè)POST請求，觀察post參數(shù)不難發(fā)現(xiàn)關(guān)鍵參數(shù)'currentPage'，猜測修改該參數(shù)就可獲取不同頁面的內(nèi)容，將url驗(yàn)證一下

驗(yàn)證url.png

沒毛病！
這樣一來，目標(biāo)網(wǎng)頁分析完畢！
通過對url(https://mm.taobao.com/tstar/search/tstar_model.do?_input_charset=utf-8) post相應(yīng)參數(shù)就可獲取頁面內(nèi)容，控制參數(shù)’current Page‘就可以遍歷頁面內(nèi)容。

2、代碼實(shí)現(xiàn)。

(1)下載器實(shí)現(xiàn)

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}
param = {'_input_charset': 'utf-8',
          'q': '',
          'viewFlag': 'A',
          'sortType': 'default',
          'searchStyle': '',
          'searchRegion': 'city',
          'searchFansNum': '',
          'currentPage': '',
          'pageSize': '20'
          }
url = 'https://mm.taobao.com/tstar/search/tstar_model.do'

def getJson(page):
    param['currentPage'] = str(page)
    params = parse.urlencode(param).encode('utf-8')
    req = request.Request(url, data=params, headers=headers)
    content = request.urlopen(req)
    content = json.loads(content.read().decode('gbk'))
    # 如果出錯(cuò)會返回 'status' = -1
    if content['status'] == -1:
        return -1
    #print(content)
    return content

(2)解析器實(shí)現(xiàn)

def parserJson(content, page):
    mmList = []

    data = content['data']['searchDOList']
    for l in data:
        temp = {}
        temp['id'] = str(l['userId'])
        temp['name'] = l['realName']
        temp['city'] = l['city']
        temp['height'] = str(l['height'])
        temp['weight'] = str(l['weight'])
        temp['favornum'] = str(l['totalFavorNum'])
        temp['profile'] = 'http:'+l['avatarUrl']
        temp['pic'] = 'http:'+l['cardUrl']
        #print(temp)
        mmList.append(temp)
        mkdir(temp['name'])
        print('第%s頁-->正在抓取%s'%(page, temp['name']))
        getImg(temp['profile'], temp['name'], 'profile')
        getImg(temp['pic'], temp['name'], 'pic')
        if not os.path.exists('./'+temp['name']+'/info.txt'):
            with open('./'+temp['name']+'/info.txt', 'w') as f:
                f.write(temp['name']+'\n')
                f.write(temp['city']+'\n')
                f.write(temp['height']+'\n')
                f.write(temp['weight']+'\n')

    return mmList

(3)圖片保存

def mkdir(path):
    if not os.path.exists(path):
        os.makedirs(path)
    else:
        print('目錄已存在！')

def getImg(url, path, name):
    if os.path.exists('./' + path + '/' + name + '.jpg'):
        print('文件已存在！')
        return 0
    try:
        req = request.Request(url, headers=headers)
        reponse = request.urlopen(req)
        get_img = reponse.read()
        with open('./' + path + '/' + name + '.jpg', 'wb') as fp:
            fp.write(get_img)
    except error.URLError as e:
        print(e.reason)

(4)主程序

# python3.6

from urllib import request, parse, error
import json
import os

if __name__ == '__main__':
    page = 1
    while True:
        content = getJson(page)
        if content == -1:
            print('抓取完畢！')
            exit()
        parserJson(content, page)
        page += 1

以上就把各位MM的高清照片(1張或2張)以及簡介信息下載到本地了，如果嫌占內(nèi)存，可以把數(shù)據(jù)保存在數(shù)據(jù)庫中，隨時(shí)可以拿來用（比如后續(xù)想通過個(gè)人主頁url訪問個(gè)人信息）。以下添加數(shù)據(jù)庫保存代碼。

數(shù)據(jù)庫保存

# 使用mysql數(shù)據(jù)庫

import pymysql

tablename = 'taobaomm'  # 定義一張表名
conn = pymysql.connect(host='127.0.0.1', user='root', passwd='root', db='mysql', charset='utf8')
cur = conn.cursor()
cur.execute('USE ***')  # 選擇一個(gè)數(shù)據(jù)庫
try:
    cur.execute('CREATE TABLE '+tablename+' (id BIGINT(7) NOT NULL AUTO_INCREMENT, name VARCHAR(100), city VARCHAR(20), height VARCHAR(10), weight VARCHAR(10), homepage VARCHAR(100), profile VARCHAR(100), pic VARCHAR(100), created TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY(id))')
except:
    pass
# 以下使數(shù)據(jù)庫支持中文不亂碼
cur.execute('ALTER DATABASE wyq CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE name name VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE city city VARCHAR(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE height height VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE weight weight VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE homepage homepage VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE profile profile VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')
cur.execute('ALTER TABLE '+tablename+' CHANGE pic pic VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci')

def store(name, city, height, weight, hompage, profile, pic):
    cur.execute('INSERT INTO '+tablename+' (name, city, height, weight, homepage, profile, pic) VALUES (\"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\", \"%s\")', (name, city, height, weight, hompage, profile, pic))
    cur.connection.commit()

只需在解析器里調(diào)用數(shù)據(jù)存儲函數(shù)store()即可。
嗯，就是這么簡單??

*完整代碼在這里：淘寶美人庫

下一節(jié)：Pyspider批量抓取網(wǎng)站圖片