[toc]

前面沒(méi)細(xì)看，這只是個(gè) mathjax 測(cè)試 $\delta^4$

Perceptron

感知器：step function

參數(shù)的數(shù)學(xué)表示：weights和bias

矩陣

能實(shí)現(xiàn)各種邏輯運(yùn)算

用神經(jīng)網(wǎng)絡(luò)可以實(shí)現(xiàn)各種邏輯計(jì)算

可以通過(guò)算法自學(xué)習(xí)

最重要的特性，通過(guò)輸入更新參數(shù)(hyper arg)

Sigmoid/Logistic neurons

重要特性： 因?yàn)樵摵瘮?shù)小的輸入變化產(chǎn)生小的輸出變化。

: Δoutput is a linear function of the changes Δwj and Δb in the weights and bias.

代數(shù)上方便：指數(shù)的微分性質(zhì)

The Architecture of neural network

MLP(Multiple Perceptron)

選取隱藏層沒(méi)有統(tǒng)一的經(jīng)驗(yàn)法則。

feedforward： 輸入不受輸出影響

Recurrent Neural Network：一種有限時(shí)間激發(fā)的，輸入受輸出影響的神經(jīng)網(wǎng)絡(luò)。

梯度下降

Cost Function: Loss function, Objective function.

quadratic cost function(mean squared error or just MSE.):

$$
\begin{eqnarray} C(w,b) \equiv
\frac{1}{2n} \sum_x | y(x) - a|^2.
\end{eqnarray}
$$

變量很多時(shí)，通過(guò)微分計(jì)算極值很麻煩的

Summing up, the way the gradient descent algorithm works is to repeatedly compute the gradient ?C, and then to move in the opposite direction,

$$
\begin{eqnarray}
\Delta C \approx \nabla C \cdot \Delta v.
\tag{9}\end{eqnarray}
$$

$$
\begin{eqnarray}
\Delta v = -\eta \nabla C,
\tag{10}\end{eqnarray}
$$

The rule doesn't always work - several things can go wrong and prevent gradient descent from finding the global minimum of C, a point we'll return to explore in later chapters. But, in practice gradient descent often works extremely well, and in neural networks we'll find that it's a powerful way of minimizing the cost function, and so helping the net learn.

rule

$$
\begin{eqnarray}
w_k & \rightarrow & w_k' = w_k-\eta \frac{\partial C}{\partial w_k} \tag{16}\end{eqnarray}
$$

$$
\begin{eqnarray}
b_l & \rightarrow & b_l' = b_l-\eta \frac{\partial C}{\partial b_l}.
\tag{17}\end{eqnarray}
$$

stochastic gradient descent,mini-batch

$$
\begin{eqnarray}
\nabla C \approx \frac{1}{m} \sum_{j=1}^m \nabla C_{X_{j}},
\tag{19}\end{eqnarray}
$$

$$
\begin{eqnarray}
w_k & \rightarrow & w_k' = w_k-\frac{\eta}{m}
\sum_j \frac{\partial C_{X_j}}{\partial w_k} \tag{20}
\end{eqnarray}
$$

$$
\begin{eqnarray}
b_l & \rightarrow & b_l' = b_l-\frac{\eta}{m}
\sum_j \frac{\partial C_{X_j}}{\partial b_l},
\tag{21}\end{eqnarray}
$$

online learning?

In general, debugging a neural network can be challenging. This is especially true when the initial choice of hyper-parameters produces results no better than random noise.

More generally, we need to develop heuristics for choosing good hyper-parameters and a good architecture

sophisticated algorithm ≤ simple learning algorithm + good training data. 

backpropagation

At the heart of backpropagation is an expression for the partial derivative ?C/?w of the cost function C with respect to any weight w (or bias b) in the network.

feedforward matrix-based computation

$$
\begin{eqnarray}
a^{l} = \sigma(w^l a^{l-1}+bl).
\tag{25}\end{eqnarray}
$$

The goal of backpropagation is to compute the partial derivatives ?C/?w and ?C/?b of the cost function C with respect to any weight w or bias b in the network.

two assumptions

1. This is the case for the quadratic cost function, where the cost for a single training example is $C_x=\frac{1}{2}∥y?a^L∥2. $

   backpropagation actually lets us do is compute the partial derivatives ?Cx/?w and ?Cx/?b for a single training example.

2. The second assumption we make about the cost is that it can be written as a function of the outputs from the neural network:

Hadamard product or Schur product: element-wise multiply

反向傳播的四個(gè)基礎(chǔ)公式

定義z的變化率 $\delta^l_j$

$$
\begin{eqnarray}
\delta^l_j \equiv \frac{\partial C}{\partial z^l_j}.
\tag{29}\end{eqnarray}
$$

之所以不用 $\frac{\partial{C}}{\partial{a^l_j}}$ 只是因?yàn)橛?jì)算方便。

An equation for the error in the output layer

$$
\begin{eqnarray}
\delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j).
\tag{BP1}\end{eqnarray}
$$

$$
\begin{eqnarray}
\delta^L = \nabla_a C \odot \sigma'(z^L).
\tag{BP1a}\end{eqnarray}
$$

if cost function is quardratic function:

$$
\begin{eqnarray}
\delta^L = (a^L-y) \odot \sigma'(z^L).
\tag{30}\end{eqnarray}
$$

An equation for the error $ \delta_l $ ** in terms of the error in the next layer** $ δ_{l+1} $

$$
\begin{eqnarray}
\delta^l = ((w^{l+1})T \delta^{l+1}) \odot \sigma'(z^l),
\tag{BP2}\end{eqnarray}
$$

By combining (BP2) with (BP1) we can compute the error δl for any layer in the network. We start by using (BP1) to compute δL, then apply Equation (BP2) to compute δL?1, then Equation (BP2) again to compute δL?2, and so on, all the way back through the network.

An equation for the rate of change of the cost with respect to any bias in the network

$$
\begin{eqnarray} \frac{\partial C}{\partial b^l_j} =
\delta^l_j.
\tag{BP3}\end{eqnarray}
$$

$$
\begin{eqnarray}
\frac{\partial C}{\partial b} = \delta,
\tag{31}\end{eqnarray}
$$

An equation for the rate of change of the cost with respect to any weight in the network

$$
\begin{eqnarray}
\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j.
\tag{BP4}\end{eqnarray}
$$

$$
\begin{eqnarray} \frac{\partial
C}{\partial w} = a_{\rm in} \delta_{\rm out},
\tag{32}\end{eqnarray}
$$

satureted: learn slowly , ?C/?w will also tend to be small

Summing up, we've learnt that a weight will learn slowly if either the input neuron is low-activation, or if the output neuron has saturated, i.e., is either high- or low-activation.

Summary: equations of backpropagation

Input a set of training examples

結(jié)合向量化的minibatch來(lái)更新權(quán)重和偏差。