在上一篇Machine Learning文章,我寫了如何在網(wǎng)絡(luò)上簡單爬取資源。這次介紹關(guān)于爬取資源的處理，即Data Processing;

我思

二、Data Processing

a、中文詞頻統(tǒng)計(jì)及詞云可視化

工具為：中文分詞jieba模塊,jieba是一款優(yōu)秀的中文分詞處理器，簡單、方便且開源；python WordCloud 模塊,功能齊全，可玩性及展示性較強(qiáng)；

以下為相關(guān)代碼：

from scipy.misc import imread
import jieba
import jieba.analyse #關(guān)鍵字提取
from os import path
from wordcloud import WordCloud
import matplotlib.pyplot as plt


file=open(r'./art/鹿鼎記.txt',encoding='utf-8',errors='ignore')
url=r'./art/stop_word.txt'
content=file.read()
file_one=[]
try:
    jieba.analyse.set_stop_words(url) # 除去中文停止詞庫
    tags=jieba.analyse.extract_tags(content,topK=160,withWeight=True) # 獲得關(guān)鍵詞及其次數(shù)，數(shù)量為前160
    for tag in tags:
        file_one.append([tag[0],tag[1]*1000]) # 寫入file_one 列表內(nèi)
        print(tag[0]+'\t'+str(tag[1]*1000)) # 顯示

finally:
    print('OK')
patch=r'C:\Users\22109\Downloads\字體-方正蘭亭黑體.ttf'      #設(shè)定詞云顯示中文類型
dictionary=dict(file_one) # list——dict類型
bg_pic = imread(r'./art/img.png')
wc = WordCloud(
            # 設(shè)置字體
            font_path = patch,
            # 設(shè)置背景色
            background_color='white',
            # 允許最大詞匯
            max_words=200,
            # 詞云形狀
            mask=bg_pic,
            # 最大號字體
            max_font_size=200,
            )
wc.generate_from_frequencies(dictionary) #引入字典類型
plt.figure() 
plt.imshow(wc) # plt顯示
plt.axis('off')

詞云顯示圖：

Figure_1.png

b、統(tǒng)計(jì)可視化處理

b-1、直方圖比較

import os
import jieba
import jieba.analyse
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sys
import matplotlib

def file_read(url,n):
    matplotlib.rcParams['font.family'] = 'sans-serif'
    matplotlib.rcParams['font.sans-serif'] = [u'SimHei']
    matplotlib.rcParams['font.size'] = '10'
    content=open(links,'r').read()
    file_one=[]
    try:
        jieba.analyse.set_stop_words('D:\My Documents\Downloads\jinyong\停用詞.txt')
        jieba.load_userdict('D:\My Documents\Downloads\jinyong\dict.txt')
        tags = jieba.analyse.extract_tags(content,topK=120,withWeight=True)
        for i in range(len(tags)):
            file_one.append(tags[i])
    finally:
        print 'OK'
    dictionary=pd.DataFrame(file_one).iloc[0:n,:]
    return dictionary
    
width,n=0.4,21
links='D:\My Documents\Downloads\jinyong\誅仙.txt'
dictionary=file_read(links,n)
plt.bar(range(len(dictionary[0])),dictionary[1]*100,width=width,color='rgy')
for i in range(len(dictionary)):
    plt.text(i-width/4*3,dictionary[1][i]*100,dictionary[0][i])
plt.show()

結(jié)果如下（以誅仙為例）：

誅仙詞頻圖

可以看出，主角光環(huán)十分強(qiáng)大，而且與故事情節(jié)較為呼應(yīng)的是，張小凡及鬼厲的頻率幾乎相等，即符合誅仙小說的情節(jié)變化（小凡黑化）；有意思的是，從數(shù)據(jù)來看，陸雪琪的出場率大于碧瑤的出場率，若是不懂情節(jié)或未看過原著的話，大部分會認(rèn)為是：陸雪琪為女主角 ；但是，如果爬取碧瑤青云之戰(zhàn)之前的數(shù)據(jù)，估計(jì)碧瑤是遙遙領(lǐng)先的。。。

將武俠小說（鹿鼎記）與誅仙對比：
代碼如下：

import os
import jieba
import jieba.analyse
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sys
import matplotlib


def file_read(url,n,width):
    matplotlib.rcParams['font.family'] = 'sans-serif'
    matplotlib.rcParams['font.sans-serif'] = [u'SimHei']
    matplotlib.rcParams['font.size'] = '8'
    long=len(url)
    for j in range(long):
        content=open(url[j],'r').read()
        file_one=[]
        try:
            jieba.analyse.set_stop_words('D:\My Documents\Downloads\jinyong\停用詞.txt')
            jieba.load_userdict('D:\My Documents\Downloads\jinyong\dict.txt')
            tags = jieba.analyse.extract_tags(content,topK=120,withWeight=True)
            for i in range(len(tags)):
                file_one.append(tags[i])
        finally:
            print 'ok'
        dictionary=pd.DataFrame(file_one).iloc[0:n,:]
        plt.bar(np.arange(len(dictionary[0]))-width/long*pow(-1,j),dictionary[1]*100,width=width/long)
        for i in range(len(dictionary)):
            plt.text(np.array(i)-width/long*pow(-1,j)-width/4*3,dictionary[1][i]*100,dictionary[0][i])
            
    plt.show()
    
width,n=0.4,21
links='D:\My Documents\Downloads\jinyong\誅仙.txt'
url='D:\My Documents\Downloads\jinyong\鹿鼎記.txt'
urls=[links,url]
file_read(urls,n,width)

可視化處理：

對比圖