一、文本預處理階段###

1.1 設定訓練集和測試集

訓練集每一類的數量為500個文檔，測試集每一類的數量也為500個文檔。

image.png

1.2 計算每個文本的DF

為每一個文本計算TF，return格式為：'word', 'file_name', term-frequency
先算出每個文檔中的'word', term-frequency, 在結束改文本的循環(huán)后將該文本中出現的詞以 'word', 'file_name', term-frequency的形式加入 word_docid_tf

def compute_tf_by_file(self):
    word_docid_tf = []
    for name in self.filenames:
        with open(join(name), 'r') as f:
            tf_dict = dict()
            for line in f:
                line = self.process_line(line)
                words = jieba.cut(line.strip(), cut_all=False)
                for word in words:
                    tf_dict[word] = tf_dict.get(word, 0) + 1
        tf_list = tf_dict.items()
        word_docid_tf += [[item[0], name, item[1]] for item in tf_list]
    return word_docid_tf  

1.3 計算每個詞項的TF、DF
為每一個詞項計算TF，return的term_freq格式為：'word', dict ( 'file_name ', tf )
為每一個詞項計算DF，return的doc_freq格式為：'word', df

def compute_tfidf(self):
    word_docid_tf = self.compute_tf_by_file()
    word_docid_tf.sort()
    doc_freq = dict()
    term_freq = dict()
    for current_word, group in groupby(word_docid_tf, itemgetter(0)):
        doclist = []
        df = 0
        for current_word, file_name, tf in group:
            doclist.append((file_name, tf))
            df += 1
        term_freq[current_word] = dict(doclist)
        doc_freq[current_word] = df
    return term_freq, doc_freq

1.4 精簡term_freq, doc_freq
除去只出現在一個或0個文檔中的詞項
除去數字詞項

def reduce_tfidf(self, term_freq, doc_freq):
    remove_list = []        
    for key in term_freq.keys():
        if len(key) < 2:#該詞只出現在一個或0個文檔中
            remove_list.append(key)
        else:
            try:
                float(key)#該詞是數字
                remove_list.append(key)
            except ValueError:
                continue
    for key in remove_list:
        term_freq.pop(key)
        doc_freq.pop(key)
    return term_freq, doc_freq

1.5 為每個文本構建特征向量train_feature, train_target
為term_freq, doc_freq中的key，也就是詞項標明index
用jieba分詞，將分好的詞放入一個臨時的數組中。
遍歷數組，由doc_freq[word]取得DF并計算iDF，由term_freq[word][name]
取得該詞項在該文檔中的TF，并計算每個詞項的tf-idf值，并作為向量中詞項對應index那一維的值。
train_feature, train_target = train_tfidf.tfidf_feature(os.path.join(input_path, 'train'),train_tf, train_df, N)

def tfidf_feature(self, dir, term_freq, doc_freq, N):
    filenames = []
    for (dirname, dirs, files) in os.walk(dir):
        for file in files:
            filenames.append(os.path.join(dirname, file))
    word_list = dict()
    for idx, word in enumerate(doc_freq.keys()):
        word_list[word] = idx
    features = []    
    target = []
    for name in filenames:
        feature = np.zeros(len(doc_freq.keys()))
        words_in_this_file = set()
        tags = re.split('[/\\\\]', name)
        tag = tags[-2]            
        with open(name, 'rb') as f:
            for line in f:
                line = self.process_line(line)
                words = jieba.cut(line.strip(), cut_all=False)
                for word in words:
                    words_in_this_file.add(word)
        for word in words_in_this_file:       
            try:
                idf = np.log(float(N) / doc_freq[word])
                tf = term_freq[word][name]
                feature[word_list[word]] = tf*idf
            except KeyError:
                continue
        features.append(feature)
        target.append(tag)
    return sparse.csr_matrix(np.asarray(features)), np.asarray(target)

1.6 存儲&加載
為了節(jié)約之后運行的時間，可以通過如下方式把測試集tf和df的值直接存儲：

Pickle.dump(train_tf, open(os.path.join(input_path, 'train_tf.pkl'), 'wb'))
print "saved train_tf.pkl"
Pickle.dump(train_df, open(os.path.join(input_path, 'train_df.pkl'), 'wb'))
print "saved train_df.pkl"

之后運行時，可以通過如下方式把測試集tf和df的值直接加載到內存，省去了重新計算的時間：

train_tf = Pickle.load(open(os.path.join(input_path, 'train_tf.pkl'), 'rb'))
print "loaded train_tf.pkl"
train_df = Pickle.load(open(os.path.join(input_path, 'train_df.pkl'), 'rb'))
train_tfidf.doc_freq=train_df
print "loaded train_df.pkl"