爬蟲(chóng)實(shí)戰(zhàn)第二天

任務(wù)

爬取小豬短租杭州地區(qū)的房源信息：

成果

共爬取284條信息，并保存到.xls文件中，寫(xiě)入時(shí)還有些編碼的問(wèn)題。

源碼

from bs4 import BeautifulSoup
import requests

pages = ['http://hz.xiaozhu.com/search-duanzufang-p{}-0/'.format(str(i)) for i in range(1, 14)]
info = []


def get_info(url):
    wb_data = requests.get(url)
    soup = BeautifulSoup(wb_data.text, 'lxml')
    data = {
        'title': soup.select('div.pho_info > h4 > em')[0].get_text(),
        'address': soup.select('div.pho_info > p > span')[0].get_text().strip(' ').strip('\n'),
        'price': soup.select('#pricePart > div.day_l > span')[0].get_text() + '/' + soup.select('#pricePart > div.day_r')[0].get_text(),
        'house_image': soup.select('#curBigImage')[0]['src'],   #圖片鏈接在chrome中不是直接打開(kāi)而是下載，在IE中可以直接打開(kāi)
        'master_name': soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a')[0]['title'],
        'master_sex': soup.select('#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > span')[0]['class'][0].split('_')[1],
        'master_image': soup.select('#floatRightBox > div.js_box.clearfix > div.member_pic > a > img')[0]['src']
    }
    print(data)
    info.append(data)


def get_url(start_url):
    wb_data = requests.get(start_url)
    soup = BeautifulSoup(wb_data.text, 'lxml')
    urls = soup.select('#page_list > ul > li > a')
    return urls

for page in pages:
    urls = get_url(page)
    for url in urls:
        try:
            get_info(url['href'])
        except Exception as e:
            pass

with open('xiaozhu.xls', 'w') as f:
    for i in info:
        for key in i:
            try:
                f.write(key)
                f.write('\t')
                f.write(i[key])
                f.write('\t')
            except Exception as e:
                break
        f.write('\n')

小結(jié)

• 數(shù)據(jù)寫(xiě)入的編碼問(wèn)題待研究。