本文使用的語言為Python, 用到的幾個模塊有：BeautifulSoup（爬數(shù)據(jù)），pandas（數(shù)據(jù)處理），seaborn（可視化），部分圖表由Tableau生成。

1. 數(shù)據(jù)獲取

計劃要抓取的字段包括：片名，導演，年份，國別，評分，評價數(shù)量，看過數(shù)量，想看數(shù)量，短評數(shù)量，長評數(shù)量。

需要抓取的影片信息有250條，每頁25部影片，一共有10頁。簡單瀏覽網(wǎng)頁不難發(fā)現(xiàn)，翻頁的鏈接不需要從頁面底端抓取，直接修改url參數(shù)即可。

在這里插入圖片描述

import requests
from bs4 import BeautifulSoup
from time import sleep
from csv import DictWriter
base_url = r'https://movie.douban.com/top250'
records = []
for start in [x*25 for x in range(10)]:
    #Every single page
    url = base_url+f'?start={start}&filter='
    response = requests.get(url).text
    soup = BeautifulSoup(response,'html.parser')
    movies = soup.find(class_='grid_view').find_all('li')
    rank=1+start
    for movie in movies:
        #Every single movie on the page
        movie_link = movie.find(class_='info').find(class_='hd').find('a')['href']
        movie_dict = {'rank':rank, 'link':movie_link}
        records.append(movie_dict)
        rank += 1

這樣我們就得到了一個list, list中有250個dictionary，每個dictionary中有影片在榜單中的排名和影片詳細信息頁面鏈接。

下一步就是進一步從已經(jīng)得到的鏈接中抓取影片的詳細信息。具體代碼如下，仔細分析html標簽做簡單的測試即可。注意，爬取數(shù)據(jù)的過程中要加上sleep(5)，禮貌爬取，防止IP被封。

#Use the scapped link to further scrape movie details
for record in records:
    rank = record.get('rank')
    print(f'Scarpping rank {rank} of 250')
    link = record.get('link')
    response = requests.get(link).text
    soup = BeautifulSoup(response,'html.parser')
    record['title'] = soup.find('h1').find('span').get_text()
    record['year'] = soup.find('h1').find(class_='year').get_text()[1:5]
    record['director'] = soup.find(id='info').find(class_='attrs').find('a').get_text()
    record['length'] = soup.find(id='info').find(property='v:runtime').get_text()[:-2]
    
    attrs = soup.find(id='info').find_all(class_='pl')
    for attr in attrs:
        if attr.get_text().startswith('制片國家'):
            record['country_region']=attr.nextSibling.strip()
        elif attr.get_text().startswith('語言'):
            record['language']=attr.nextSibling.strip()
    
    record['avg_rating'] = soup.find(class_='ll rating_num').get_text()
    record['num_of_ratings'] = soup.find(class_='rating_people').find('span').get_text()
    interests = soup.find(class_='subject-others-interests-ft').find_all('a')
    record['people_watched'] = interests[0].get_text()[:-3]
    record['people_wants_to_watch'] = interests[1].get_text()[:-3]
    record['num_comment'] = soup.find(id='comments-section').find('h2').find('a').get_text().split()[1]
    record['num_reviews'] = soup.find(class_='reviews mod movie-content').find('h2').find('a').get_text().split()[1]
    #Set scrapping interval in case of ip blocking
    sleep(5)

下一步是將爬取的數(shù)據(jù)存入csv文件，代碼如下：

注意，寫入文件時中文可能會亂碼，需要加上encoding='utf-8-sig'，加上newline=''解決每行記錄之間存在空行的問題。

#Write the 250 movies into a csv file
with open('douban_top_250.csv','w', newline='', encoding='utf-8-sig') as file:
    headers = [key for key in records[0].keys()]
    csv_writer = DictWriter(file, fieldnames=headers)
    csv_writer.writeheader()
    for record in records:
        csv_writer.writerow(record)

2. 數(shù)據(jù)清洗

由于抓取的數(shù)據(jù)只有250條，可以直接用excel打開，簡單看一下數(shù)據(jù)有沒有問題。

在這里插入圖片描述

import numpy as np
import pandas as pd
import re
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats

path = r'C:\Users\yingk\Desktop\Douban_Top250\douban_top_250.csv'
data = pd.read_csv(path,encoding='utf-8-sig')

#Preprocessing
#extract Chinese title
data['Chinese_title'] = data['title'].apply(lambda title:title.split()[0])
data.drop(['title'],axis=1,inplace=True)

#re.sub() - Use "" to repleace characters which are not digit
data['movie_length'] = data['length'].apply(lambda length:re.sub("\D","",length))
data.drop(['length'], axis=1, inplace=True)

#extract main country_region when there are multiple
data['main_country_region'] = data['country_region'].apply(lambda cr:cr.split()[0])
data.drop(['country_region'],axis=1,inplace=True)

#extract main language
data['main_language'] = data['language'].apply(lambda lan:lan.split()[0])
data.drop(['language'],axis=1,inplace=True)

data.to_csv(r'C:\Users\yingk\Desktop\Douban_Top250\cleaned_data.csv',encoding='utf-8-sig')

3. 探索分析

將清洗好的csv文件導入Tableau，下面是豆瓣電影TOP250上的制片國家/地區(qū)分布和各個語言所占的比重。比重越大，字體越大。類似的圖表也可以用Python wordcloud來做。

在這里插入圖片描述

在這里插入圖片描述

在這里插入圖片描述

metrics = data.drop(['link','director','Chinese_title','main_country_region','main_language'],axis=1)

增加/轉(zhuǎn)化一些字段:

相較于平均評分，評價人數(shù)，可能增加一個總評分=avg_rating*num_of_ratings會更直接體現(xiàn)影片的質(zhì)量。

上映年份可以轉(zhuǎn)換成已經(jīng)上映了多少年，即2019-year，在時間顯得更直觀。

metrics.eval('total_rating_scores = avg_rating*num_of_ratings', inplace=True)
metrics['years to 2019'] = metrics['year'].apply(lambda y:2019-y)
metrics.drop(['year'],axis=1,inplace=True)

生成相關(guān)系數(shù)和heatmap。

corr = metrics.corr()
plt.figure(figsize=(6,5), dpi=100)
sns.heatmap(corr,cmap='coolwarm',linewidths=0.5)

在這里插入圖片描述

corr.head(1)

在這里插入圖片描述

#total_rating_scores t-test
x = list(metrics['rank'])
y = list(metrics['total_rating_scores'])
r,p = stats.pearsonr(x,y)
print(r)
print(p)

在這里插入圖片描述

comment_reviews = metrics.loc[:,['num_comment','num_reviews']]
def bin_years(value):
    for i in range(10,91,10):
        if value<i:
            return f'{i-9}-{i}'
comment_reviews['years_bin'] = metrics['years to 2019'].apply(bin_years)
bins = [f'{i-9}-{i}' for i in range(10,91,10)]
plt.figure(figsize=(12,5), dpi=100)
sns.boxplot(x='years_bin', y='num_comment', data=comment_reviews, order=bins)
plt.figure(figsize=(12,5), dpi=100)
sns.boxplot(x='years_bin', y='num_reviews', data=comment_reviews, order=bins)

在這里插入圖片描述

在這里插入圖片描述