1. One-Hot
2. 詞袋
Bag of Words（詞袋表示），也稱為Count Vectors，每個(gè)文檔的字/詞可以使用其出現(xiàn)次數(shù)來(lái)進(jìn)行表示。
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
'This is the first document.',
'This document is the second document.',
'And this is the third one.',
'Is this the first document?',
]
vectorizer = CountVectorizer()
vectorizer.fit_transform(corpus).toarray()
Output：
array([[0, 1, 1, 1, 0, 0, 1, 0, 1],
[0, 2, 0, 1, 0, 1, 1, 0, 1],
[1, 0, 0, 1, 1, 0, 1, 1, 1],
[0, 1, 1, 1, 0, 0, 1, 0, 1]], dtype=int64)
3. N-gram
4. TF-IDF
由兩部分組成：
? 第一部分是詞語(yǔ)頻率（Term Frequency），
? 第二部分是逆文檔頻率（Inverse Document Frequency）。
其中計(jì)算語(yǔ)料庫(kù)中文檔總數(shù)除以含有該詞語(yǔ)的文檔數(shù)量，然后再取對(duì)數(shù)就是逆文檔頻率。TF(t)= 該詞語(yǔ)在當(dāng)前文檔出現(xiàn)的次數(shù) / 當(dāng)前文檔中詞語(yǔ)的總數(shù)
IDF(t)= log_e（文檔總數(shù) / 出現(xiàn)該詞語(yǔ)的文檔總數(shù)）
對(duì)比不同文本表示算法的精度，通過(guò)本地構(gòu)建驗(yàn)證集計(jì)算F1得分
PlanA：Count Vectors + RidgeClassifier
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_score
train_df = pd.read_csv('/Users/summer/Desktop/xul_data/learning/DataWhale/20200719NLP/task01_preparing_20200719/input/train_set.csv', *sep*='t', *nrows*=15000)
vectorizer = CountVectorizer(*max_features*=3000)
train_test = vectorizer.fit_transform(train_df['text'])
# https://blog.csdn.net/LOLUN9/article/details/106012418/
# https://blog.csdn.net/fantacy10000/article/details/90647686
'''RidgeClassifier()通過(guò)Ridge()以下方式使用回歸模型來(lái)創(chuàng)建分類器：
為了簡(jiǎn)單起見(jiàn)，讓我們考慮二進(jìn)制分類，目標(biāo)變量等于+1或-1。
建立一個(gè)Ridge()回歸模型（這是一個(gè)回歸模型）來(lái)預(yù)測(cè)我們的目標(biāo)變量。損失函數(shù)是RMSE + l2 penality
如果Ridge()回歸的預(yù)測(cè)值（基于decision_function()函數(shù)計(jì)算）大于0，則將其預(yù)測(cè)為正類，否則為負(fù)類。
'''
# L2嶺回歸，壓縮最優(yōu)解的系數(shù)，計(jì)算效率高，模型穩(wěn)定性好；L1減少項(xiàng)的個(gè)數(shù)
clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])
val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, *average*='macro'))
Output：
>>>> 0.65441877581244
PlanB：TF-IDF + RidgeClassifier
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_score
# train_df = pd.read_csv('../data/train_set.csv', sep='t', nrows=15000)
tfidf = TfidfVectorizer(*ngram_range*=(1,3), *max_features*=3000)
train_test = tfidf.fit_transform(train_df['text'])
clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])
val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, *average*='macro'))
Output：
>>> 0.8719372173702
本章作業(yè)
Q1：嘗試改變TF-IDF的參數(shù)，并驗(yàn)證精度
A1：Tfidf Vectorizer
train_df_hw = pd.read_csv('/Users/summer/Desktop/xul_data/learning/DataWhale/20200719NLP/task01_preparing_20200719/input/train_set.csv', sep='\t', nrows=10000)
tfidf_hw = TfidfVectorizer(ngram_range=(1,5), max_features=3000)
train_hw_test = tfidf_hw.fit_transform(train_df_hw['text'])
clf = RidgeClassifier()
clf.fit(train_hw_test[:7000], train_df_hw['label'].values[:7000])
val_pred_hw = clf.predict(train_hw_test[7000:10000]) # [:N]表示從第一個(gè)開(kāi)始取到第N個(gè)
print(f1_score(train_df_hw['label'].values[7000:10000], val_pred_hw, average='macro'))
Output：
# Test1-ngram_range=(1,3)
>>> 0.9317302315325816
# Test2-ngram_range=(1,5)
>>> 0.9326016109802603
# Test3-增加停用詞stop_words='world'
>>> 0.9322304435086377
# Test4-norm='l2'
>>> 0.9322304435086377
# Test5-norm='l1'
>>> 0.5073894598279685
Q2：嘗試使用其他機(jī)器學(xué)習(xí)模型，完成訓(xùn)練和驗(yàn)證
常用分類器 線性：LR、SVM 非線性：DF、RF、GBDT、XGBOOST
原理：https://www.cnblogs.com/andy-0212/p/10630608.html
對(duì)比：https://www.cnblogs.com/wkang/p/9657032.html
https://blog.csdn.net/twt520ly/article/details/79769705
http://www.itdecent.cn/p/96173f2c2fb4
GBDT：https://blog.csdn.net/weixin_40924580/article/details/85043801?utm_medium=distribute.pc_relevant.none-task-blog-baidujs-2&spm=1001.2101.3001.4242
Test1：GBDT
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder as OHE
from xgboost.sklearn import XGBClassifier
train_df_hw = pd.read_csv('/Users/summer/Desktop/xul_data/learning/DataWhale/20200719NLP/task01_preparing_20200719/input/train_set.csv', sep='\t', nrows=10000)
tfidf_hw = TfidfVectorizer(ngram_range=(1,5), max_features=3000)
train_hw_test = tfidf_hw.fit_transform(train_df_hw['text'])
x_train = train_hw_test[:7000]
y_train = train_df_hw['label'].values[:7000]
x_test = train_hw_test[7000:10000]
y_test = train_df_hw['label'].values[7000:10000]
gbm1 = GradientBoostingClassifier(n_estimators=50, random_state=10, subsample=0.6, max_depth=4,
min_samples_split=400)
gbm1.fit(x_train, y_train)
gbm1_pred_hw = gbm1.predict(x_test)
print(f1_score(y_test, gbm1_pred_hw, average='macro'))
Output：
>>> 0.8165503231061779
Test2：TF-IDF+GBDT+LR，基于Test1
import numpy as np
## 特征轉(zhuǎn)換
## model.apply(x_train)返回訓(xùn)練數(shù)據(jù)x_train在訓(xùn)練好的模型里每棵樹(shù)中所處的葉子節(jié)點(diǎn)的位置（索引）
y_pred = gbm1.apply(x_train)
y_pred = y_pred.reshape(7000, -1) # 一個(gè)ID對(duì)應(yīng)一個(gè)特征，訓(xùn)練集中有7000個(gè)ID，因此reshape(7000, -1)
## 打印上面結(jié)果的輸出，可以看到shape是(7000, 50)，即訓(xùn)練數(shù)據(jù)量*樹(shù)的棵樹(shù)
print(np.array(y_pred).shape)
print(y_pred[0])
enc = OneHotEncoder()
enc.fit(y_pred)
y_pred2 = np.array(enc.transform(y_pred).toarray())
### 對(duì)測(cè)試集相同操作
y_pred_test = gbm1.apply(x_test)
y_pred_test = y_pred_test.reshape(3000, -1)
print(np.array(y_pred_test).shape) #(3000, 700)
print(y_pred_test[0])
y_pred_test2 = np.array(enc.transform(y_pred_test).toarray())
## 預(yù)測(cè)
LR = LogisticRegression(penalty='l2')
LR.fit(y_pred2, y_train)
lr_pred_hw = LR.predict(y_pred_test2)
print(f1_score(y_test, lr_pred_hw, average='macro'))
Output：
>>> 0.8515161019132009
通過(guò)LR提升3.5%
