這次利用scrapy抓取了深圳所有在鏈家網(wǎng)的租住房信息，一直對房租價格比較感興趣，這次終于能利用自己的技能分析一下了，至于為什么現(xiàn)在鏈家網(wǎng)，時候覺得這里數(shù)據(jù)比較齊全。
這是網(wǎng)址

Paste_Image.png

下面是scrapy框架圖

Paste_Image.png

Paste_Image.png

提取這些數(shù)據(jù)都是為了分析與價格的關系
這是setting里面鏈接MySql的一些設定，包括密碼，用戶，以及端口，在pipelines里面要用

Paste_Image.png

下面看pipelines代碼，主要是對于抓取到的數(shù)據(jù)進行操作，存進Mysql數(shù)據(jù)庫，這里不得不吐槽一下，mogodb數(shù)據(jù)庫在爬蟲里面感覺比MySql方便多了，因為在插進數(shù)據(jù)之前，自己必須現(xiàn)在數(shù)據(jù)庫里面創(chuàng)建好表。

import mysql.connector
from lianjia import settings
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
MYSQL_HOSTS = settings.MYSQL_HOSTS
MYSQL_USER = settings.MYSQL_USER
MYSQL_PASSWORD = settings.MYSQL_PASSWORD
MYSQL_PORT = settings.MYSQL_PORT
MYSQL_DB = settings.MYSQL_DB#這幾條是鏈接MySql數(shù)據(jù)庫的，端口密碼，都在setting文件里面設定好了

cnx = mysql.connector.connect(user = MYSQL_USER,password= MYSQL_PASSWORD,host =MYSQL_HOSTS,database=MYSQL_DB)#創(chuàng)建鏈接
cur = cnx.cursor(buffered=True)
class Sql:#這里的sql語句為下面數(shù)據(jù)插入以及去重做準備
    @classmethod
    def insert_tenement_message(cls,title,rental,distance,area,room_number,floor,direction,year_build):
        sql = 'INSERT INTO tenement_message(`title`,`rental`,`distance`,`area`,`room_number`,`floor`,`direction`,
`year_build`) 
VALUES
 (%(title)s,%(rental)s,%(distance)s,%(area)s,%(room_number)s,%(floor)s,%(direction)s,%(year_build)s)'
        value = {
            'title':title,
            'rental':rental,
            'distance':distance,
            'area':area,
            'room_number':room_number,
            'floor':floor,
            'direction':direction,
            'year_build':year_build,
        }
        cur.execute(sql,value)#執(zhí)行語句，插進數(shù)數(shù)據(jù)庫
        cnx.commit()
    @classmethod
    def select_title(cls,title):#這個是利用標題去重的，雖然按照區(qū)域劃分應該不會重復，只是預防萬一
        sql= 'SELECT EXISTS (SELECT 1 FROM tenement_message WHERE title = %(title)s)'
        value = {
            'title':title
        }
        cur.execute(sql,value)
        return  cur.fetchall()[0]



class LianjiaPipeline(object):
    def process_item(self, item, spider):
        title = item['title']
        ret = Sql.select_title(title)
        if ret[0] ==1:

            print('房子已經(jīng)存在')
        else:
            rental = item['rental']
            distance = item['distance']
            area = item['area']
            room_number = item['room_number']
            floor = item['floor']
            direction = item ['direction']
            year_build = item['year_build']
            Sql.insert_tenement_message(title,rental,distance,area,room_number,floor,direction,year_build)
            print('開始存租房信息')

下面是鏈家lianjia主爬蟲程序腳本

import scrapy
from bs4 import BeautifulSoup
from scrapy.http import Request
from lianjia.items import LianjiaItem
import requests
import re

class myspider(scrapy.Spider):
    name = 'lianjia'
    allowed_domains =['sz.lianjia.com']



    def start_requests(self):
        theme_url = 'http://sz.lianjia.com/zufang/luohuqu/pg1/'#爬蟲開始的頁面
        html = requests.get(theme_url)
        content = BeautifulSoup(html.text, 'lxml')

        urls = []
        links = content.find('div', class_='option-list').find_all('a')#找出所有區(qū)域的鏈接
        for link in links:
            i = re.findall(r'g/(.*)/', link['href'])
            if i:
                urls.extend(i)#提取每個區(qū)域的鏈接
        all_url = ['http://sz.lianjia.com/zufang/{}/pg1/'.format(i) for i in urls]#構造出每一個區(qū)域的鏈接
        for url in all_url:
            print(url)
            yield Request(url,self.parse)#對每個鏈接調用parse函數(shù)
    def parse(self, response):
        page = BeautifulSoup(response.text, 'lxml').find('div', class_='page-box house-lst-page-box')#找
        出每個區(qū)域最大的頁數(shù)，
        然后遍歷
        max_page = re.findall('Page":(\d+)."cur', str(page))[0]
        bashurl = str(response.url)[:-2]
        for num in range(1,int(max_page)+1):
            url = bashurl+str(num)+'/'
            #print(url)
            yield Request(url,callback=self.get_message)
    def get_message(self,response):
        item = LianjiaItem()
        content = BeautifulSoup(response.text, 'lxml')

        house_list = content.find_all('div', {'class': 'info-panel'})#找到所以租房信息所在的標簽里

        for li in house_list:
            try:
                data = li.find('span', class_='fang-subway-ex').find('span').get_text()
                item['distance'] = re.findall(r'(\d+)', data)[1]  # 將離地鐵站的距離多少米提取出來，切片選
                取第二個數(shù)字是因為第一個是地鐵線號，要提出
            except:
                item['distance'] = "沒有附近地鐵數(shù)據(jù)"
              # 取出樓層，因為里面用/分割了多段文字，所以用split提取
            try:
                item['year_build'] = re.findall(r'(\d+)', li.find('div', class_='con').get_text().split('/')[-1])[0]  # 把房

                屋的建造年份提取出來

            except:
                item['year_build'] = '沒有建造年份'
            item['title'] = li.find('h2').find('a').attrs['title']
            item['rental'] = li.find('div', class_='price').find('span').get_text()
            item['area'] = re.findall(r'(\d+)', li.find('span', class_='meters').get_text().replace('&nbsp', ''))[
                0]  # 將面積數(shù)據(jù)提取出來
            item['room_number'] = li.find('span', class_='zone').find('span').get_text().replace('\xa0','')
            item['floor'] = li.find('div', class_='con').get_text().split('/')[1]  # 取出樓層，因為里面用/分割了多段文字，
            所以用split提取
            item['direction'] = li.find('div', class_='where').find_all('span')[-1].get_text()  # 提取房屋朝向，
            先找到這個標簽在提取文字

            yield item

來看看運行結果吧

Paste_Image.png

一共差不多7000間房出租，第一次看到驚呆了，這也太少了吧比我想得，哈哈
繪制了一下價格跟地鐵距離關系的圖，以后再補上，順便一提，如果遍歷深圳鏈家租房主頁，是只有100個頁面的，爬的數(shù)據(jù)是不全的，只有5000套左右