吳恩達(dá)Deep Learning第二課作業(yè)（第二周）

目錄鏈接：吳恩達(dá)Deep Learning學(xué)習(xí)筆記目錄

??1.Gradient Descent
??2.mini-batch Gradient Descent
??3.Momentum
??4.Adam
??5.Model with different optimization algorithms

1. Gradient Descent

import numpy as np
import matplotlib.pyplot as plt
from opt_utils import load_params_and_grads, initialize_parameters, forward_propagation, backward_propagation
from opt_utils import compute_cost, predict, predict_dec, plot_decision_boundary, load_dataset
from testCase import *

plt.rcParams['figure.figsize'] = (7.0, 4.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

def update_params_with_gd(params,grads,learning_rate):
    L = len(params) // 2
    for layer in range(L):
        params["W"+str(layer + 1)] = params["W"+str(layer + 1)] - learning_rate * grads["dW"+str(layer + 1)]
        params["b"+str(layer + 1)] = params["b"+str(layer + 1)] - learning_rate * grads["db"+str(layer + 1)]
        
    return params

(1) (Batch) Gradient Descent
??Batch Gradient Descent 相當(dāng)于mini-batch 梯度下降法 batch_size為整個(gè)訓(xùn)練集大小m：

def model_with_batch_gd(X,Y,layer_dims,epochs,learning_rate):
    params = initialize_parameters(layer_dims)  
    for epoch in range(epochs):
        AL,caches = forward_propagation(X,params)
        
        cost = compute_cost(AL,Y)
        
        grads = backward_propagation(AL,caches,params)
        
        params = update_params_with_gd(params,grads,learning_rate)
                        
    return params

(2)Stochastic Gradient Descent
??需要在每一epoch中，每次輸入一個(gè)樣本，計(jì)算這個(gè)樣本對(duì)應(yīng)的loss，然后進(jìn)行梯度下降，更新所有參數(shù)，循環(huán)m（樣本數(shù)）次后完成一個(gè)epoch。SGD雖然計(jì)算速度快，但由于其下降時(shí)是振蕩的，收斂速度并不快。

def model_with_batch_gd(X,Y,layer_dims,epochs,learning_rate):
    m = X.shape[1]
    params = initialize_parameters(layer_dims)  
    for epoch in range(epochs):
        for sample in range(m):
            AL,caches = forward_propagation(X[:,sample],params)
            cost = compute_cost(AL,Y[:,sample])
            grads = backward_propagation(AL,caches)
            params = update_params_with_gd(params,grads,learning_rate)
    
    return params

2. mini-batch Gradient Descent

??實(shí)際上，我們可以采用比較折中的方法，batch size取1到m之間，兼顧Batch gradient descent 和 Stochastic gradient descent的優(yōu)點(diǎn)，振蕩較小，計(jì)算速度快。

??①Shuffle：將樣本的順序打亂，確保樣本被隨機(jī)的分割僅不同的mini-batch；

??②Partition：將數(shù)據(jù)集分割為mini-batch，大小為mini_batch_size，當(dāng)最后一組樣本數(shù)量小于mini_batch_size時(shí)，將所有剩下樣本作為一個(gè)mini-batch。
數(shù)據(jù)集分割： 
def random_mini_batch(X,Y,mini_batch_size = 64,seed = 0):
    np.random.seed(seed)
    m = X.shape[1]
    mini_batches = []
    
    permutation = list(np.random.permutation(m))#將0-m的整數(shù)隨機(jī)排序
    shuffled_X = X[:,permutation]
    shuffled_Y = Y[:,permutation].reshape((1,m))
    
    num_complete_mini_batches = math.floor(m / mini_batch_size)#向下取整
    
    for i in range(num_complete_mini_batches):
        mini_batch_X = shuffled_X[:,i * mini_batch_size : (i + 1) * mini_batch_size]
        mini_batch_Y = shuffled_Y[:,i * mini_batch_size : (i + 1) * mini_batch_size]
        mini_batch = (mini_batch_X,mini_batch_Y)
        mini_batches.append(mini_batch)
    
    #求m被mini_batch_size除后的余數(shù)即最后一個(gè)minibatch的樣本數(shù)
    if m % mini_batch_size != 0:
        num_samples_of_end = m - mini_batch_size * num_complete_mini_batches
        mini_batch_X = shuffled_X[:,mini_batch_size * num_complete_mini_batches :]
        mini_batch_Y = shuffled_Y[:,mini_batch_size * num_complete_mini_batches :]
        mini_batch = (mini_batch_X,mini_batch_Y)
        mini_batches.append(mini_batch)
        
    return mini_batches

3. Momentum

??由于mini-batch梯度下降法僅僅采用樣本集的一個(gè)子集，所以梯度的更新方向是振蕩的，采用Momentum可以減小這種振蕩。Momentum通過計(jì)算前面mini-batch的梯度的EWMA值來代替當(dāng)前原梯度，將梯度數(shù)據(jù)進(jìn)行平滑。

??Momentum計(jì)算規(guī)則：
??①由于velocity初始化為0，所以初始幾步迭代計(jì)算出的EWMA值有偏差，多次迭代后該偏差消失；如果β=0，則該算法退變?yōu)闃?biāo)準(zhǔn)的梯度下降更新算法。
??②如何選擇β？

????β越大，則梯度下降越平滑，因?yàn)閜ast gradient在參與計(jì)算EWMA值時(shí)，權(quán)值（越往前權(quán)值指數(shù)式下降）大足夠起到作用的past gradient數(shù)量越多，但太大也會(huì)導(dǎo)致太平滑；
????β的取值范圍一般為0.8-0.999，0.9一般為默認(rèn)值；
????需要對(duì)β和α進(jìn)行超參數(shù)搜索。

#初始化EWMA
def initialize_velocity(params):
    """
    function: initialize the veclocity as a dict containing dW1,dW2,...
    
    return: a dict of dW1,dW2,...
    """
    L = len(params) // 2
    velocity = {}
    
    for layer in range(L):
        velocity["dW" + str(layer + 1)] = np.zeros_like(params["W" + str(layer + 1)])
        velocity["db" + str(layer + 1)] = np.zeros_like(params["b" + str(layer + 1)])
    
    return velocity
#更新梯度
def update_params_with_momentum(params,grads,velocity,beta,learning_rate):
    """
    arguments:
        beta:the momentum hyperparam,scalar
    return:
        params: params by updated
        velocity:velocity by updated
    """
    L = len(params) // 2
    
    for layer in range(L):
        velocity["dW" + str(layer + 1)] = beta * velocity["dW" + str(layer + 1)] + (1 - beta) * grads["dW" + str(layer + 1)]
        velocity["db" + str(layer + 1)] = beta * velocity["db" + str(layer + 1)] + (1 - beta) * grads["db" + str(layer + 1)]
        
        params["W" + str(layer + 1)] = params["W" + str(layer + 1)] - learning_rate * velocity["dW" + str(layer + 1)]
        params["b" + str(layer + 1)] = params["b" + str(layer + 1)] - learning_rate * velocity["db" + str(layer + 1)]
    
    return params,velocity

4. Adam

??Adam結(jié)合了Monmentum和RMSProp，計(jì)算過程：

??①計(jì)算EWMA，和修正后的EWMA并存儲(chǔ)；
??②計(jì)算EWMA的平方及其修正值；
??③更新參數(shù)

def initialize_adam(params):
    """
    return:
        v: v["dW" + str(layer)] = ...,v["db" + str(layer)] = ...
        s: s["dW" + str(layer)] = ...,s["db" + str(layer)] = ...
    """
    L = len(params) // 2
    v = {}
    s = {}
    for layer in range(L):
        v["dW" + str(layer + 1)] = np.zeros_like(params["W" + str(layer + 1)])
        v["db" + str(layer + 1)] = np.zeros_like(params["b" + str(layer + 1)])
        
        s["dW" + str(layer + 1)] = np.zeros_like(params["W" + str(layer + 1)])
        s["db" + str(layer + 1)] = np.zeros_like(params["b" + str(layer + 1)]) 
        
    return v,s

def update_params_with_adam(params,grads,v,s,t,learning_rate = 0.01,beta1 = 0.9,beta2 = 0.999,epsilon = 1e-8):
    """
    t:num of iteration current
    """
    L = len(params) // 2
    v_corrected = {}
    s_corrected = {}
    
    for layer in range(L):
        v["dW" + str(layer + 1)] = beta1 * v["dW" + str(layer + 1)] + (1 - beta1) * grads["dW" + str(layer + 1)]
        v["db" + str(layer + 1)] = beta1 * v["db" + str(layer + 1)] + (1 - beta1) * grads["db" + str(layer + 1)]
        
        v_corrected["dW" + str(layer + 1)] = v["dW" + str(layer + 1)] / (1 - np.power(beta1,t))
        v_corrected["db" + str(layer + 1)] = v["db" + str(layer + 1)] / (1 - np.power(beta1,t))
        
        s["dW" + str(layer + 1)] = beta2 * s["dW" + str(layer + 1)] + (1 - beta2) * np.power(grads["dW" + str(layer + 1)],2)
        s["db" + str(layer + 1)] = beta2 * s["db" + str(layer + 1)] + (1 - beta2) * np.power(grads["db" + str(layer + 1)],2)
        
        s_corrected["dW" + str(layer + 1)] = s["dW" + str(layer + 1)] / (1 - np.power(beta2,t))
        s_corrected["db" + str(layer + 1)] = s["db" + str(layer + 1)] / (1 - np.power(beta2,t))
        
        params["W" + str(layer + 1)] = params["W" + str(layer + 1)] - learning_rate * v_corrected["dW" + str(layer + 1)] / np.sqrt(s["dW" + str(layer + 1)] + epsilon)
        params["b" + str(layer + 1)] = params["b" + str(layer + 1)] - learning_rate * v_corrected["db" + str(layer + 1)] / np.sqrt(s["db" + str(layer + 1)] + epsilon)
        
    return params,v,s

5.Model with different optimization algorithms

??訓(xùn)練集數(shù)據(jù)分布：


??模型（通過optimizer參數(shù)選擇優(yōu)化方法）：

def model(X,Y,layer_dims,optimizer,learning_rate = 7e-4,mini_batch_size = 64,
          beta = 0.9,beta1 = 0.9,beta2 = 0.999,epsilon = 1e-8,epochs = 10000):
    L = len(layer_dims)
    costs = []
    t = 0
    seed = 10
    
    params = initialize_parameters(layer_dims)
    
    if optimizer == "gd":
        pass
    elif optimizer == "momentum":
        v = initialize_velocity(params)
    elif optimizer == "adam":
        v,s = initialize_adam(params)
    
    for epoch in range(epochs):
        seed = seed + 1
        mini_batches = random_mini_batch(X,Y,mini_batch_size,seed)
        
        for mini_batch in mini_batches:
            (mini_batch_X,mini_batch_Y) = mini_batch
            
            AL,caches = forward_propagation(mini_batch_X,params)
            
            cost = compute_cost(AL,mini_batch_Y)
            
            grads = backward_propagation(mini_batch_X,mini_batch_Y,caches)
            
            if optimizer == "gd":
                params = update_params_with_gd(params,grads,learning_rate)
            elif optimizer == "momentum":
                params,v = update_params_with_momentum(params,grads,v,beta,learning_rate)
            elif optimizer == "adam":
                t = t + 1
                params,v,s = update_params_with_adam(params,grads,v,s,t,learning_rate,beta1,beta2,epsilon) 
            
        if epoch % 1000 == 0:
            print("epoch: %d ,loss: %3.3f" % (epoch,cost))
        if epoch % 100 == 0:   
            costs.append(cost)
        
    plt.rcParams['figure.figsize'] = (15.0, 4.0)
    plt.subplot(1,2,1)
    plt.plot(costs)
    plt.ylabel('cost')
    plt.xlabel('epochs')
    plt.title("learning rate: "+str(learning_rate))
    
    plt.subplot(1,2,2)
    plt.title("Model with "+str(optimizer))
    axes = plt.gca()
    axes.set_xlim([-1.5, 2.5])
    axes.set_ylim([-1, 1.5])
    plot_decision_boundary(lambda x: predict_dec(params, x.T), X, np.squeeze(Y))
    
    return params

??結(jié)果對(duì)比：


??結(jié)論：
??由于learning_rate較小及數(shù)據(jù)集較為簡(jiǎn)單，standard gradient descent 和 momentum的結(jié)果相近，且兩個(gè)算法還未收斂，需要更多的迭代次數(shù)；
??但是Adam展現(xiàn)出顯著的效果，收斂速度較快，其優(yōu)點(diǎn)是：
????①對(duì)內(nèi)存需求相對(duì)較低（但高于standard gradient descent
和momentum）；
????②就算是在較小的learning_ratex下，表現(xiàn)也很好。

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容