ExponentialMovingAverage

Some training algorithms, such as GradientDescent and Momentum often benefit from maintaining a moving average of variables during optimization. Using the moving averages for evaluations often improve results significantly.
tensorflow 官網(wǎng)上對于這個方法功能的介紹。GradientDescent 和 Momentum 方式的訓練 都能夠從 ExponentialMovingAverage 方法中獲益。

什么是MovingAverage? 我的理解就是一段時間窗口內(nèi)的這個變量的歷史平均值。

tensorflow 中的 ExponentialMovingAverage

官方文檔中的公式:
shadowVariable=decay?shadowVariable+(1?decay)?variable
官網(wǎng)的example：

# Create variables.
var0 = tf.Variable(...)
var1 = tf.Variable(...)
# ... use the variables to build a training model...
...
# Create an op that applies the optimizer.  This is what we usually
# would use as a training op.
opt_op = opt.minimize(my_loss, [var0, var1])

# Create an ExponentialMovingAverage object
ema = tf.train.ExponentialMovingAverage(decay=0.9999)

# Create the shadow variables, and add ops to maintain moving averages
# of var0 and var1.
maintain_averages_op = ema.apply([var0, var1])

# Create an op that will update the moving averages after each training
# step.  This is what we will use in place of the usual training op.
with tf.control_dependencies([opt_op]):
    training_op = tf.group(maintain_averages_op)
    # run這個op獲取當前時刻 ema_value
    get_var0_average_op = ema.average(var0)

使用 ExponentialMovingAveraged parameters

假設我們使用了ExponentialMovingAverage方法訓練了神經(jīng)網(wǎng)絡， 在test階段，如何使用 ExponentialMovingAveraged parameters呢？ 官網(wǎng)也給出了答案

# Create a Saver that loads variables from their saved shadow values.
shadow_var0_name = ema.average_name(var0)
shadow_var1_name = ema.average_name(var1)
saver = tf.train.Saver({shadow_var0_name: var0, shadow_var1_name: var1})
saver.restore(...checkpoint filename...)
# var0 and var1 now hold the moving average values

或者

#Returns a map of names to Variables to restore.
variables_to_restore = ema.variables_to_restore()
saver = tf.train.Saver(variables_to_restore)
...
saver.restore(...checkpoint filename...)

轉載： CSDN Blog