一個小爬蟲，爬一個本地的網(wǎng)頁。

Screen Shot 2016-06-26 at 10.07.06 PM.png

需要提取的信息包括：
頁面中顯示的所有商品的標題，價格，點擊量，評分以及商品的圖片地址。

#!usr/bin/env python
#_*_ coding: utf-8 _*_
#
# filter out specifical info from a local webpage
#
from bs4 import BeautifulSoup

path = '1_2answer_of_homework/index.html'

with open(path, 'r') as webpage:
    content = webpage.read()
    #print content
    soup = BeautifulSoup(content, "lxml")

    images = soup.select('body > div > div > div.col-md-9 > div > div > div > img')
    titles = soup.select('body > div > div > div.col-md-9 > div > div > div > div.caption > h4 > a')
    prices = soup.select('body > div > div > div.col-md-9 > div > div > div > div.caption > h4.pull-right')
    reviews = soup.select('body > div > div > div.col-md-9 > div > div > div > div.ratings > p.pull-right')
    stars = soup.select('body > div > div > div.col-md-9 > div > div > div > div.ratings')
    #print stars
    for (img, title, price, review, star) in zip(images, titles, prices, reviews, stars):
        img_content = img.get('src')
        title_content = title.get_text()
        price_content = price.get_text()
        review_content = review.get_text()
        star_content = len(star.select('p > span.glyphicon-star'))
    #print stars

        data = {
            "title": title_content,
            "review": review_content,
            "image": img_content,
            "price": price_content,
            "rating": star_content
        }
        print(data)

運行結(jié)果

{'image': 'img/pic_0000_073a9256d9624c92a05dc680fc28865f.jpg', 'price': '$24.99', 'review': '65 reviews', 'rating': 5, 'title': 'EarPod'}
{'image': 'img/pic_0005_828148335519990171_c234285520ff.jpg', 'price': '$64.99', 'review': '12 reviews', 'rating': 4, 'title': 'New Pocket'}
{'image': 'img/pic_0006_949802399717918904_339a16e02268.jpg', 'price': '$74.99', 'review': '31 reviews', 'rating': 4, 'title': 'New sunglasses'}
{'image': 'img/pic_0008_975641865984412951_ade7a767cfc8.jpg', 'price': '$84.99', 'review': '6 reviews', 'rating': 3, 'title': 'Art Cup'}
{'image': 'img/pic_0001_160243060888837960_1c3bcd26f5fe.jpg', 'price': '$94.99', 'review': '18 reviews', 'rating': 4, 'title': 'iphone gamepad'}
{'image': 'img/pic_0002_556261037783915561_bf22b24b9e4e.jpg', 'price': '$214.5', 'review': '18 reviews', 'rating': 4, 'title': 'Best Bed'}
{'image': 'img/pic_0011_1032030741401174813_4e43d182fce7.jpg', 'price': '$500', 'review': '35 reviews', 'rating': 4, 'title': 'iWatch'}
{'image': 'img/pic_0010_1027323963916688311_09cc2d7648d9.jpg', 'price': '$15.5', 'review': '8 reviews', 'rating': 4, 'title': 'Park tickets'}

總結(jié)

• 爬之前要先了解網(wǎng)頁的結(jié)構(gòu)
• 熟悉soup.select()如何篩選目標元素
• 學(xué)習(xí)如何從篩選出的元素中提取目標信息，get_text(), get('class')方法的使用
• 將目標信息保存為一個dict, 方便整理
• 計算星星個數(shù)的方法: star_content = len(star.select('p > span.glyphicon-star'))，
  先篩選出目標標簽再用len()函數(shù)得到個數(shù)