目標(biāo)網(wǎng)站：https://movie.douban.com/top250
爬取目的：豆瓣電影排行榜前250的電影的電影名，保存到 movies.txt 文件

源代碼：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2017/4/19 16:30
# @Author  : zxp
# @Site    : 
# @File    : Douban_top250.py
# @Software: PyCharm
import codecs
import requests
from bs4 import BeautifulSoup

DOWNLOAD_URL = 'http://movie.douban.com/top250'

def download_page(url):
    headers = {
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'
               }
    data = requests.get(url, headers=headers).content
    return data

def parse_html(html):
    soup = BeautifulSoup(html, 'html.parser')
    movie_list_soup = soup.find('ol', attrs={'class':'grid_view'})

    movie_name_list = []

    for movie_li in movie_list_soup.find_all('li'):
        detail = movie_li.find('div', attrs={'class':'hd'})
        movie_name = detail.find('span', attrs={'class':'title'}).getText()
        movie_name_list.append(movie_name)

    next_page = soup.find('span', attrs={'class':'next'}).find('a')
    if next_page:
        return movie_name_list, DOWNLOAD_URL + next_page['href']
    return movie_name_list, None


def main():
    url = DOWNLOAD_URL

    with codecs.open('movies.txt', 'wb', encoding='utf-8') as fp:
        while url:
            html = download_page(url)
            movies, url = parse_html(html)
            fp.write(u'{movies}\n'.format(movies='\n'.join(movies)))


if __name__ == '__main__':
    main()

①使用了 User-Agent 模擬瀏覽器訪問(wèn)，防止被服務(wù)器認(rèn)為是爬蟲(chóng)而拒絕訪問(wèn)。
②一般在我們確定內(nèi)容的前提下，可以直接在代碼中寫(xiě)死如何跳轉(zhuǎn)頁(yè)面，但是為了讓我們的爬蟲(chóng)更像爬蟲(chóng)，我們讓它找到頁(yè)碼導(dǎo)航中的下一頁(yè)的鏈接。

參考博客