神經(jīng)網(wǎng)絡(luò)，英文neural network，跟random forest一樣，是眾多機(jī)器學(xué)習(xí)方法的一種。
tensorflow，是實(shí)現(xiàn)神經(jīng)網(wǎng)絡(luò)的其中一種框架。如官網(wǎng)所說:an open-sourse mechine learning framework for everyone，當(dāng)然還有其他框架可以選擇，比如caffe，PyTorch，keras等等。
tensorflow基礎(chǔ)知識(shí)點(diǎn)：
關(guān)于tensorflow實(shí)現(xiàn)neural network的一些基本概念和實(shí)現(xiàn)，請(qǐng)參考系列文章。
- 深度學(xué)習(xí)模型之激活函數(shù)（Activation Function）
- 深度學(xué)習(xí)策略之損失函數(shù)（Loss Function OR Cost Function）
- 深度學(xué)習(xí)之算法（Algorithm）
- 深度學(xué)習(xí)之評(píng)估標(biāo)準(zhǔn)(F1)
- 深度學(xué)習(xí)訓(xùn)練之Batch
- 深度學(xué)習(xí)可視化之Tensorboard
正式開啟重啟版的第三部。
一、讀取數(shù)據(jù)并轉(zhuǎn)換格式
讀取數(shù)據(jù)，并將label標(biāo)簽one-hot，再將Dataframe結(jié)果轉(zhuǎn)成ndarray。
# 讀取數(shù)據(jù)
data = pd.read_csv("haoma11yue_after_onehot_and_RobustScaler.csv", index_col=0, parse_dates=True)
print(data.shape) #(43777, 70)
# 將X和Y拆分開
from sklearn.model_selection import train_test_split
X = data.loc[:, data.columns != 'yonghuzhuangtai']
y = data.loc[:, data.columns == 'yonghuzhuangtai']
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.22, random_state = 0)
# label進(jìn)行onthot
y_train_one_hot = pd.get_dummies(y_train['yonghuzhuangtai'], prefix= 'yonghuzhuangtai')
y_test_one_hot = pd.get_dummies(y_test['yonghuzhuangtai'], prefix= 'yonghuzhuangtai')
# DataFrame轉(zhuǎn)ndarray
y_train_one_hot_n=y_train_one_hot.values
X_train_n =X_train.values
y_test_one_hot_n=y_test_one_hot.values
X_test_n =X_test.values
接下來就是逐步構(gòu)建這個(gè)flow graph
image.png
二、為inputs占坑placeholder
placeholder只是明確了列數(shù)，行是使用None，動(dòng)態(tài)的。
#-------輸入輸出的維度參數(shù)----------
features = X_train.shape[1] # 輸入層，xx個(gè)特征
numClasses = 2 # 輸出層，稀疏表示
# 指定X（輸入層）Y（輸出層）的大小，占坑
with tf.name_scope("inputs"):
X_input = tf.placeholder(tf.float32, shape = [None, features],name="X_input")
y_true = tf.placeholder(tf.float32, shape = [None, numClasses],name="y_true")
image.png
三、模型neural network
- 先定義神經(jīng)網(wǎng)絡(luò)中一層的函數(shù)，后續(xù)直接調(diào)用構(gòu)建多層neural network。后續(xù)深入看看weight和baises的初始化值對(duì)結(jié)果的影響。
# 函數(shù)add_layer，添加一個(gè)神經(jīng)網(wǎng)絡(luò)隱藏層
# layoutname，隱藏層的名字
# inputs，隱藏層的輸入，也就是前一層
# in_size，輸入的緯度，也就是前一層的neurons數(shù)目
# out_size，輸出的緯度，也就是該隱藏層的neurons數(shù)目
# activatuib_funaction，激活函數(shù)
def add_layer(layoutname, inputs, in_size, out_size, activatuib_funaction=None):
with tf.name_scope(layoutname):
with tf.name_scope('weights'):
Weights=tf.Variable(tf.random_normal([in_size,out_size], stddev=0.1),name='W') #stddev=0.1，無意中試過，加這個(gè)，效率速度快很多。其實(shí)涉及到W的初始化問題了。
tf.summary.histogram('Weights',Weights) #histogram_summary用于生成分布圖，也可以用scalar_summary記錄存數(shù)值
with tf.name_scope('biases'):
biases = tf.Variable(tf.constant(0.1, shape=[out_size]), name='b')
tf.summary.histogram('Biases',biases)
with tf.name_scope('Wx_plus_b'):
Wx_plus_b=tf.add(tf.matmul(inputs,Weights),biases)
if activatuib_funaction is None:
outputs=Wx_plus_b
else :
outputs=activatuib_funaction(Wx_plus_b)
return outputs
- 構(gòu)建neural network，這里構(gòu)建三個(gè)隱藏層，加上輸入層和輸出層，總共有5層，使用到的激活函數(shù)暫時(shí)是tf.nn.relu，后續(xù)試試tf.tanh，看看差別。
num_HiddenNeurons1 = 50 # 隱藏層第一層
num_HiddenNeurons2 = 40 # 隱藏層第二層
num_HiddenNeurons3 = 20 # 隱藏層第三層
with tf.name_scope('first_hindden_layer'):
first_hindden_layer=add_layer("first_hindden_layer",X_input,features,num_HiddenNeurons1,activatuib_funaction=tf.nn.relu)
with tf.name_scope('second_hindden_layer'):
second_hindden_layer=add_layer("second_hindden_layer",first_hindden_layer,num_HiddenNeurons1,num_HiddenNeurons2,activatuib_funaction=tf.nn.relu)
with tf.name_scope('third_hindden_layer'):
third_hindden_layer=add_layer("third_hindden_layer",second_hindden_layer,num_HiddenNeurons2,num_HiddenNeurons3,activatuib_funaction=tf.nn.relu)
with tf.name_scope('prediction'):
y_prediction =add_layer('prediction',third_hindden_layer,num_HiddenNeurons3,numClasses,activatuib_funaction=None)
# y_prediction的輸出并不是概率分布，沒有經(jīng)過softmax
# [[ 84.97052765 47.09545517]
# [ 84.97052765 47.09545517]]
with tf.name_scope('prediction_softmax'):
y_prediction_softmax = tf.nn.softmax(y_prediction)
with tf.name_scope('Save'):
saver = tf.train.Saver(max_to_keep=4) #保留最近四次的模型
image.png
image.png
四、策略loss function
暫時(shí)使用square再mean的loss function，后續(xù)試試softmax_cross_entropy_with_logits，看看區(qū)別。
其實(shí)這里有個(gè)疑問，策略是優(yōu)化的方向，mean_square，主要體現(xiàn)的應(yīng)該是precision，但是最后評(píng)判的結(jié)果是f1，有l(wèi)oss function直接體現(xiàn)f1的嗎？
with tf.name_scope("loss"):
# 結(jié)構(gòu)風(fēng)險(xiǎn)=經(jīng)驗(yàn)風(fēng)險(xiǎn)+正則化，經(jīng)驗(yàn)風(fēng)險(xiǎn)使用交叉熵，正則化使用L2。
# 暫時(shí)不使用正則化，效果好像好點(diǎn)，或者說正則化還用得不好啊。
# ------------------------------------------------square--------------------------------------------------------
loss = tf.reduce_mean(tf.square(y_true - y_prediction_softmax))
image.png
五、算法
with tf.name_scope("train"):
# ----------------------------------指數(shù)衰減學(xué)習(xí)率---------------------------------------
# exponential_decay(learning_rate, global_step, decay_steps, decay_rate, staircase=False, name=None)
# decayed_learning_rate = learning_rate * decay_rate ^ (global_step / decay_steps) ，（If the argument `staircase` is `True）
Iterations = 0
learning_rate = tf.train.exponential_decay(learning_rate=0.1, global_step=Iterations, decay_steps=10000, decay_rate=0.99, staircase=True) #staircase 樓梯。
# ----------------------------------算法---------------------------------------
opt = tf.train.GradientDescentOptimizer(learning_rate = learning_rate).minimize(loss)
image.png
六、評(píng)價(jià)
主要還是那幾個(gè)：f1，recall，precision，只是用tensor實(shí)現(xiàn)而已，最看重的還是f1。
def tf_confusion_metrics(model, actual_classes, session, feed_dict):
predictions = tf.argmax(model, 1)
actuals = tf.argmax(actual_classes, 1)
ones_like_actuals = tf.ones_like(actuals) # tf.ones_like: A `Tensor` with all elements set to 1.
zeros_like_actuals = tf.zeros_like(actuals)
ones_like_predictions = tf.ones_like(predictions)
zeros_like_predictions = tf.zeros_like(predictions)
# true positive 猜測(cè)和真實(shí)一致
tp_op = tf.reduce_sum( # tf.reduce_sum，統(tǒng)計(jì)1的個(gè)數(shù)
tf.cast( # tf.cast： Casts a tensor to a new type.把true變回1
tf.logical_and( # tf.logical_and: A `Tensor` of type `bool`. 把預(yù)測(cè)的true和實(shí)際的true取且操作
tf.equal(actuals, ones_like_actuals), # tf.equal：A `Tensor` of type `bool`.其實(shí)就是把1變成TRUE.
tf.equal(predictions, ones_like_predictions)
),
"float"
)
)
# true negative 猜測(cè)和真實(shí)一致
tn_op = tf.reduce_sum(
tf.cast(
tf.logical_and(
tf.equal(actuals, zeros_like_actuals),
tf.equal(predictions, zeros_like_predictions)
),
"float"
)
)
# false positive 實(shí)際是0，猜測(cè)是1
fp_op = tf.reduce_sum(
tf.cast(
tf.logical_and(
tf.equal(actuals, zeros_like_actuals),
tf.equal(predictions, ones_like_predictions)
),
"float"
)
)
# false negative 實(shí)際是1，猜測(cè)是0
fn_op = tf.reduce_sum(
tf.cast(
tf.logical_and(
tf.equal(actuals, ones_like_actuals),
tf.equal(predictions, zeros_like_predictions)
),
"float"
)
)
tp, tn, fp, fn = \
session.run(
[tp_op, tn_op, fp_op, fn_op],
feed_dict
)
with tf.name_scope("confusion_matrix"):
with tf.name_scope("precision"):
if((float(tp) + float(fp)) == 0):
precision = 0
else:
precision = float(tp)/(float(tp) + float(fp))
tf.summary.scalar("Precision",precision)
with tf.name_scope("recall"):
if((float(tp) + float(fn)) ==0):
recall = 0
else:
recall = float(tp) / (float(tp) + float(fn))
tf.summary.scalar("Recall",recall)
with tf.name_scope("f1_score"):
if((precision + recall) ==0):
f1_score = 0
else:
f1_score = (2 * (precision * recall)) / (precision + recall)
tf.summary.scalar("F1_score",f1_score)
with tf.name_scope("accuracy"):
accuracy = (float(tp) + float(tn)) / (float(tp) + float(fp) + float(fn) + float(tn))
tf.summary.scalar("Accuracy",accuracy)
print ('F1 Score = ', f1_score, ', Precision = ', precision,', Recall = ', recall, ', Accuracy = ', accuracy)
image.png
除了TensorFlow方式實(shí)現(xiàn)外，還可以用sklearn實(shí)現(xiàn)。
import sklearn as sk
import numpy as np
from sklearn.metrics import confusion_matrix
# 打印所有的scores參數(shù)，包括precision、recall、f1等等
# y_pred_score,神經(jīng)網(wǎng)絡(luò)的預(yù)測(cè)結(jié)果,經(jīng)過softmax,type: <class 'numpy.ndarray'>
# y_true_onehot_score,神經(jīng)網(wǎng)絡(luò)的true值輸入，是one-hot編碼后的type: <class 'numpy.ndarray'>
def scores_all(y_pred_onehot_score, y_true_onehot_score):
y_pred_score = np.argmax(y_pred_onehot_score, axis = 1) # 反one-hot編碼
y_true_score = np.argmax(y_true_onehot_score, axis = 1) # 反one-hot編碼
# print("precision:",sk.metrics.precision_score(y_true_score,y_pred_score), \
# "recall:",sk.metrics.recall_score(y_true_score,y_pred_score), \
# "f1:",sk.metrics.f1_score(y_true_score,y_pred_score))
print("f1:",sk.metrics.f1_score(y_true_score,y_pred_score))
七、batch
batch的實(shí)現(xiàn)。
# --------------函數(shù)說明-----------------
# sourceData_feature ：訓(xùn)練集的feature部分
# sourceData_label ：訓(xùn)練集的label部分
# batch_size ： 牛肉片的厚度
# num_epochs ： 牛肉翻煮多少次
# shuffle ： 是否打亂數(shù)據(jù)
def batch_iter(sourceData_feature,sourceData_label, batch_size, num_epochs, shuffle=True):
data_size = len(sourceData_feature)
num_batches_per_epoch = int(data_size / batch_size) # 樣本數(shù)/batch塊大小,多出來的“尾數(shù)”，不要了
for epoch in range(num_epochs):
# Shuffle the data at each epoch
if shuffle:
shuffle_indices = np.random.permutation(np.arange(data_size))
shuffled_data_feature = sourceData_feature[shuffle_indices]
shuffled_data_label = sourceData_label[shuffle_indices]
else:
shuffled_data_feature = sourceData_feature
shuffled_data_label = sourceData_label
for batch_num in range(num_batches_per_epoch): # batch_num取值0到num_batches_per_epoch-1
start_index = batch_num * batch_size
end_index = min((batch_num + 1) * batch_size, data_size)
yield (shuffled_data_feature[start_index:end_index] , shuffled_data_label[start_index:end_index])
八、訓(xùn)練train
進(jìn)行迭代訓(xùn)練train。
batchSize = 1000 # 定義具體的牛肉厚度
epoch_count = 200 # 訓(xùn)練的epoch次數(shù)
Iterations = 0 # 記錄迭代的次數(shù)
print("how many steps would train: ", (epoch_count * int((len(X_train_n)/batchSize))))
print('---------------------------start training------------------------------')
# sess
sess = tf.Session()
merged = tf.summary.merge_all() #Merges all summaries collected in the default graph.
# 定義訓(xùn)練過程中的參數(shù)（比如loss，weight，baises）保存到哪里
writer_val = tf.summary.FileWriter("logs/val", sess.graph)
# 保存訓(xùn)練過程中各個(gè)環(huán)節(jié)消耗的時(shí)間和內(nèi)存
writer_timeandplace = tf.summary.FileWriter("logs/timeandplace", sess.graph)
sess.run(tf.global_variables_initializer())
# 迭代 必須注意batch_iter是yield→generator，所以for語句有特別
for (batchInput, batchLabels) in batch_iter(X_train_n, y_train_one_hot_n, batchSize, epoch_count, shuffle=True):
if Iterations%1000 == 0:
# --------------------------------訓(xùn)練并記錄-----------------------------------------------
run_options = tf.RunOptions(trace_level = tf.RunOptions.FULL_TRACE) # 配置運(yùn)行時(shí)需要記錄的信息
run_metadata = tf.RunMetadata() # 運(yùn)行時(shí)記錄運(yùn)行信息的proto
# train
trainingopt,trainingLoss,merged_r,y_prediction_softmax_r,y_prediction_r= \
sess.run([opt,loss,merged,y_prediction_softmax,y_prediction], \
feed_dict={X_input:batchInput, y_true:batchLabels}, options =run_options, run_metadata = run_metadata)
# print(batchInput[0:5,:])
# print(y_prediction_r[0:5,:])
# print(y_prediction_softmax_r[0:5,:])
# print(batchLabels[0:5,:])
# 記錄參數(shù)
writer_val.add_summary(merged_r, Iterations)
# 將節(jié)點(diǎn)在運(yùn)行時(shí)的信息寫入日志文件
writer_timeandplace.add_run_metadata(run_metadata, 'Iterations%03d' % Iterations)
# 輸出效果
print("step %d, %d people leave in this batch, loss is %g" \
%(Iterations, sum(np.argmax(batchLabels,axis = 1)) ,trainingLoss ))
print('--------------------train scores------------------')
tf_confusion_metrics(y_prediction_softmax_r, batchLabels, sess, feed_dict={X_input:batchInput, y_true:batchLabels})
# scores_all(y_prediction_softmax_r ,batchLabels)
# test set 的效果
trainingLoss, y_prediction_softmax_r = sess.run([loss,y_prediction_softmax], feed_dict = {X_input: X_test_n, y_true:y_test_one_hot_n})
print('**********************test score**********************')
tf_confusion_metrics(y_prediction_softmax_r, y_test_one_hot_n, sess, feed_dict = {X_input: X_test_n, y_true:y_test_one_hot_n})
# scores_all(y_prediction_softmax_r ,y_test_one_hot_n)
else:
# train
trainingopt, trainingLoss, merged_r = sess.run([opt,loss,merged], feed_dict={X_input: batchInput, y_true:batchLabels})
# 記錄參數(shù)
writer_val.add_summary(merged_r, Iterations)
if Iterations%3000 == 0: # 每迭代三千次，save模型
saver.save(sess, 'tf_model/my_test_model',global_step=Iterations)
Iterations=Iterations+1
writer_val.close()
writer_timeandplace.close()
# sess.close()
訓(xùn)練結(jié)果：64000次訓(xùn)練后，f1大概是56.77%，倒數(shù)第二個(gè)是59.31%。
image.png
九、tensorboard看參數(shù)
loss：很快在0.0140附近震蕩了。
image.png
image.png
十、優(yōu)化
- 對(duì)weight和baises的初始值進(jìn)行微調(diào)，效果不明顯。
Weights=tf.Variable(tf.random_normal([in_size,out_size], mean =0, stddev=0.2),name='W')
biases = tf.Variable(tf.random_normal(shape=[out_size], mean =0, stddev=0.2),name='b')
image.png
- 試試改變網(wǎng)絡(luò)結(jié)果，增加層或改激活函數(shù)
- 改loss function，最想改這個(gè)，掛鉤f1，還有加入正則化
- 過采樣到10:1
- 看方差，看偏差
后續(xù)，繼續(xù)嘗試特征優(yōu)化，獲取更多的特征，優(yōu)化各種機(jī)器學(xué)習(xí)方法，
其實(shí)還有如何實(shí)現(xiàn)，呈現(xiàn)等
還有更重要的，回歸到基礎(chǔ)數(shù)學(xué)，統(tǒng)計(jì)學(xué)，線性代數(shù)，最優(yōu)化等
跟算法玩?zhèn)€游戲，偷偷把label放到feature里面，看看算法能找到嗎？
結(jié)果的二維邊界怎么畫
