1. softmax_cross_entropy_with_logits

方法定義：

def softmax_cross_entropy_with_logits(
    _sentinel=None,  
    labels=None, 
    logits=None,  
    dim=-1, 
    name=None ):

計(jì)算方式：

let x = logits, y = labels, The loss is :
    
    y * log(softmax(x))
    softmax(x) = exp(xi) / sum(exp(xj))

2. sparse_softmax_cross_entropy_with_logits

方法定義：

def sparse_softmax_cross_entropy_with_logits(
    _sentinel=None,  
    labels=None, 
    logits=None, 
    name=None ):

3. 1和2兩種方法的對(duì)比

softmax_cross_entropy_with_logits和sparse_softmax_cross_entropy_with_logits這兩種方法的交叉熵計(jì)算方式是相同的，都是以下方法：

• 1. 對(duì)神經(jīng)網(wǎng)絡(luò)的輸出做softmax計(jì)算

     
     
     softmax函數(shù)

• 2. 計(jì)算交叉熵

     
     
     交叉熵計(jì)算

不同之處在于調(diào)用時(shí)labels參數(shù)不相同，softmax_cross_entropy_with_logits采用的是one hot的形式，而sparse_softmax_cross_entropy_with_logits采用的是原始標(biāo)簽形式。

import tensorflow as tf  
  
# 神經(jīng)網(wǎng)絡(luò)的輸出
logits=tf.constant([[1.0,2.0,3.0],[1.0,2.0,3.0],[1.0,2.0,3.0]]) 
# 對(duì)輸出做softmax操作
y=tf.nn.softmax(logits)  
# 真實(shí)數(shù)據(jù)標(biāo)簽，one hot形式
y_=tf.constant([[0.0,0.0,1.0],[0.0,0.0,1.0],[0.0,0.0,1.0]]) 
# 將標(biāo)簽稠密化
dense_y=tf.argmax(y_,1)   # dense_y = [2 2 2]
# 采用普通方式計(jì)算交叉熵
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
# 使用softmax_cross_entropy_with_logits方法計(jì)算交叉熵
cross_entropy2=tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_))
# 使用sparse_softmax_cross_entropy_with_logits方法計(jì)算交叉熵
cross_entropy3=tf.reduce_sum(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=dense_y))
  
with tf.Session() as sess:  
    softmax=sess.run(y)
    c_e = sess.run(cross_entropy)
    c_e2 = sess.run(cross_entropy2)
    c_e3 = sess.run(cross_entropy3)
    print("step1:softmax result=")  
    print(softmax)  
    print("y_ = ")
    print(sess.run(y_))
    print("tf.log(y) = ")
    print(sess.run(tf.log(y)))
    print("dense_y =")
    print(sess.run(dense_y))
    print("step2:cross_entropy result=")  
    print(c_e)  
    print("Function(softmax_cross_entropy_with_logits) result=")  
    print(c_e2)
    print("Function(sparse_softmax_cross_entropy_with_logits) result=")
    print(c_e3)

輸出結(jié)果：

step1:softmax result=
[[ 0.09003057  0.24472848  0.66524094]
 [ 0.09003057  0.24472848  0.66524094]
 [ 0.09003057  0.24472848  0.66524094]]
y_ = 
[[ 0.  0.  1.]
 [ 0.  0.  1.]
 [ 0.  0.  1.]]
tf.log(y) = 
[[-2.40760589 -1.40760589 -0.40760601]
 [-2.40760589 -1.40760589 -0.40760601]
 [-2.40760589 -1.40760589 -0.40760598]]
dense_y =
[2 2 2]
step2:cross_entropy result=
1.22282
Function(softmax_cross_entropy_with_logits) result=
1.22282
Function(sparse_softmax_cross_entropy_with_logits) result=
1.22282

4. sigmoid_cross_entropy_with_logits

方法定義：

def sigmoid_cross_entropy_with_logits(
    _sentinel=None,  
    labels=None, 
    logits=None,  
    name=None ):

計(jì)算方式：

let x = logits, z = labels, The loss is :
    
    z * log(sigmoid(x)) + (1-z) * log(1-sigmoid(x))
    sigmoid(x) = 1 / (1 + exp(-x))

5. weighted_sigmoid_cross_entropy_with_logits

方法定義：

def weighted_cross_entropy_with_logits(
    targets=None, 
    logits=None, 
    pos_weight=None, 
    name=None ):

計(jì)算方法：

let x = logits, z = labels, q = pos_weight, The loss is :
    
    q * z * log(sigmoid(x)) + (1-z) * log(1-sigmoid(x))
    sigmoid(x) = 1 / (1 + exp(-x))

6. 4和5的實(shí)現(xiàn)對(duì)比

值得注意的是，兩個(gè)方法中，一個(gè)參數(shù)是labels，另一個(gè)對(duì)應(yīng)的參數(shù)是targets。

import tensorflow as tf

# 神經(jīng)網(wǎng)絡(luò)的輸出
logits=tf.constant([[1.0,2.0,3.0],[1.0,2.0,3.0],[1.0,2.0,3.0]]) 
# 對(duì)輸出做softmax操作
y=tf.nn.sigmoid(logits)  
# 真實(shí)數(shù)據(jù)標(biāo)簽，one hot形式
y_=tf.constant([[0.0,0.0,1.0],[0.0,0.0,1.0],[0.0,0.0,1.0]]) 
# 將標(biāo)簽稠密化
dense_y=tf.argmax(y_,1)  
# 采用普通方式計(jì)算交叉熵
cross_entropy = -tf.reduce_sum(y_*tf.log(y)+(1-y_)*tf.log(1-y))
# 使用sigmoid_cross_entropy_with_logits方法計(jì)算交叉熵
cross_entropy2 = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=y_))
# 使用weighted_cross_entropy_with_logits方法計(jì)算交叉熵
cross_entropy3 = tf.reduce_sum(tf.nn.weighted_cross_entropy_with_logits(logits=logits, targets=y_, pos_weight=2.0))

with tf.Session() as sess:  
    sigmoid=sess.run(y)
    c_e = sess.run(cross_entropy)
    c_e2 = sess.run(cross_entropy2)
    c_e3 = sess.run(cross_entropy3)
    print("step1:sigmoid result=")
    print(sigmoid)
    print("step2:cross_entropy result=")  
    print(c_e)
    print("Function(sigmoid_cross_entropy_with_logits) result=")  
    print(c_e2)
    print("Function(weighted_cross_entropy_with_logits) result=")
    print(c_e3)

輸出結(jié)果：

step1:sigmoid result=
[[ 0.7310586   0.88079703  0.95257413]
 [ 0.7310586   0.88079703  0.95257413]
 [ 0.7310586   0.88079703  0.95257413]]
step2:cross_entropy result=
10.4663
Function(sigmoid_cross_entropy_with_logits) result=
10.4663
Function(weighted_cross_entropy_with_logits) result=
10.6121