Sigmoid neurons simulating perceptrons, part I

Suppose we take all the weights and biases in a network of perceptrons, and multiply them by a positive constant, c>0c>0. Show that the behaviour of the network doesn't change. 對于perceptron rule :

圖片.png

兩邊同時乘上c，等式不變

圖片.png

Sigmoid neurons simulating perceptrons, part II

Suppose we have the same setup as the last problem - a network of perceptrons. Suppose also that the overall input to the network of perceptrons has been chosen. We won't need the actual input value, we just need the input to have been fixed. Suppose the weights and biases are such that w?x+b≠0 for the input x to any particular perceptron in the network. Now replace all the perceptrons in the network by sigmoid neurons, and multiply the weights and biases by a positive constant c>0. Show that in the limit as c→∞the behaviour of this network of sigmoid neurons is exactly the same as the network of perceptrons. How can this fail when w?x+b=0 for one of the perceptrons?

對于sigmoid 函數(shù)來說，(wx + b)同時乘上c，不影響wx + b > 0 或 wx + b < 0的結(jié)果，因此對于σ(wx + b) > 0.5 或 σ(wx + b) < 0.5 的判定沒有影響。但是當(dāng)wx + b = 0時，σ(wx + b) = 0.5，判斷不了結(jié)果的類別，因此不能進(jìn)行二元分類。

Four output neural exercise

There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous la!yer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.

圖片.png

二進(jìn)制數(shù)字：

• 0 ：0000
• 1 ：0001
• 2 ：0010
• 3 ：0011
• 4 ：0100
• 5 ：0101
• 6 ：0110
• 7 ：0111
• 8 ：1000
• 9 ：1001
  output layer從上至下依次表示2^0, 2^1, 2^2, 2^{3，那么1,3,5,7,9會使2}0的output neural輸出1,而2,3,6,7會使2^1的output neural輸出1，依次類推。 那么對于第一個output neural，我們可以令向量w=-1,1,-1,1,-1,1,-1,1,-1,1，b = 0則對于這個neural來說，wx + b = -1x0 + x1 - x2 + x3 - x4 + x5 - x6 + x7 - x8 + x9 > 0 當(dāng)且僅當(dāng)輸入中x1,x3,x5,x7,x9的值的和大于其余的和（正常情況下只有一項為0.99，其余都是0.01）。比如x5=0.99,其余是0.01，那么wx + b = 0.98 > 0 ,所以σ（wx + b）> 0.5，2^0位輸出為1。 同理，得到4個ouput neural的w：
  1. -1, 1,-1, 1,-1, 1,-1, 1,-1, 1
  2. -1,-1, 1, 1,-1,-1, 1, 1,-1,-1
  3. -1,-1,-1,-1, 1, 1, 1, 1,-1,-1
  4. -1,-1,-1,-1,-1,-1,-1,-1, 1, 1
     注意前面給出的條件是：正確的結(jié)果一定大于0.99，錯誤的一定小于0.01，這樣就避免了比如數(shù)字0為0.8最大，其余的是0.7比0.8小，但是其他的和加起來大于0.8，輸出錯誤的情況。