1. 練習(xí)準(zhǔn)備：

MongoDB的準(zhǔn)備：

• MongoDB學(xué)習(xí)—（1）安裝時出現(xiàn)The default storage engine 'wiredTiger' is not available問題解決 http://blog.csdn.net/u013457382/article/details/50775268

• windows下mongodb安裝與使用整理:
  http://www.cnblogs.com/lecaf/archive/2013/08/23/mongodb.html

• MongoDB數(shù)據(jù)導(dǎo)入導(dǎo)出：

  
  
  image.png
  
  
  image.png
  
  
  image.png
  
  
  image.png

• 新建 collection：db.creatCollection('filename')

• 簡單的MongoDB使用方法：

  
  
  image.png
  
  
  image.png
  
  
  image.png

2. CODE:

import requests
from bs4 import BeautifulSoup
import pymongo

client = pymongo.MongoClient('localhost',27017)
xiaozhu = client['xiaozhu']
sheet_tab = xiaozhu['sheet_tab']

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}

def main():
    urls = ["http://bj.xiaozhu.com/search-duanzufang-p{}-0/".format(number) for number in range(1, 2)]
    for url in urls:
        wb_data = requests.get(url)

    # 開始解析網(wǎng)頁數(shù)據(jù)
        soup = BeautifulSoup(wb_data.text, 'lxml')

        # 鼠標(biāo)放到圖片上，右鍵，審查元素，找到鏈接的css selector
        links = soup.select("#page_list > ul > li > a")

        #  由于鏈接有好多個，soup.select返回的是列表，需要用for一個個取出來
        for link in links:
            # 由于鏈接地址在標(biāo)簽的href屬性里面，所以要用get獲取
            href = link.get("href")

            # 把得到的詳情頁鏈接，傳給函數(shù)，這個函數(shù)可以得到詳細(xì)數(shù)據(jù)
            get_detail_info(href)

def get_lorder_sex(class_name):
    if class_name == ['member_boy_ico']:
        return '男'
    elif class_name == ['member_girl_ico']:
        return '女'

def get_detail_info(url):
    wb_data = requests.get(url)

# 開始解析詳情頁數(shù)據(jù)
    soup = BeautifulSoup(wb_data.text, 'lxml')

# 獲取名稱
    titles = soup.select("body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > h4 > em")

# 獲取地址
    addresss = soup.select("body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > p > span.pr5")

# 獲取價格
    prices = soup.select("#pricePart > div.day_l > span")

# 獲取圖片
    images = soup.select("#curBigImage")

# 獲取房東頭像
    avartars = soup.select("#floatRightBox > div.js_box.clearfix > div.member_pic > a > img")

# 獲取房東姓名
    names = soup.select("#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a")

# 獲取房東性別
    sexs = soup.select("#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > span")

    for title, address, price, image, avartar, name, sex in zip(titles, addresss, prices, images, avartars, names, sexs):
    # 從標(biāo)簽里面提取內(nèi)容
        data = {
            "title": title.get_text(),
            "address": address.get_text(),
            "price": int(price.get_text()),
            "image": image.get("src"),
            "avartar": avartar.get("src"),
            "name": name.get_text(),
            "sex": get_lorder_sex(sex.get("class"))
        }
  
        sheet_tab.insert_one(data)
        for item in sheet_tab.find({'price':{'$gte':500}}):
            print('title: '+ item['title'],'價格： '+ str(item['price']) + '元')



if __name__ == '__main__':
    main()

3. 運(yùn)行結(jié)果：

……
title: 勁松雙井10號線地鐵華騰園大三室整租家庭聚會 價格： 698元
title: 北京居住核心地段，國貿(mào)CBD雙井，地鐵站旁 價格： 548元
title: 簡約大氣，浪漫舒適，緊鄰9號線科怡路 價格： 769元
Process finished with exit code 0