estimator

estimator.png

dataset

tensorflow 底層 API 中都是使用 placeholder 和 feed_dict 向模型輸入數(shù)據(jù)的，這樣的方式效率較低。我們可以利用 dataset 庫，這里提供了高效讀取數(shù)據(jù)并且輸入給模型訓練的方式。

可以直接用 numpy 數(shù)組創(chuàng)建 dataset。直接用數(shù)組創(chuàng)建 dataset 的一個問題是 tensorflow 會直接把 dataset 中的數(shù)據(jù)寫到 graph 中，當數(shù)據(jù)量較大時會報錯，因為 graph 在序列化到 pb 文件時現(xiàn)在最大2GB。

def input_fn():
  features, labels = (np.random.sample((100,2)), np.random.sample((100,1)))
  dataset = tf.data.Dataset.from_tensor_slices((features,labels))
  dataset = dataset.shuffle(100000).repeat().batch(batch_size)
  return dataset

...
estimator.train(input_fn)

為了在大數(shù)據(jù)量時使用 dataset，我們可以用 placeholder 創(chuàng)建 dataset。這時數(shù)據(jù)就不會直接寫到 graph 中，graph 中只有一個 placeholder 占位符。但是，用了 placeholder 就需要我們在一開始對它進行初始化填數(shù)據(jù)，需要調(diào)用 sess.run(iter.initializer, feed_dict={ x: data })。更多關于 dataset 的使用介紹可以參考文獻[2]。

def input_fn():
  x = tf.placeholder(tf.float32, shape=[None,2])
  dataset = tf.data.Dataset.from_tensor_slices(x)
  dataset = dataset.shuffle(100000).repeat().batch(batch_size)
  iter = dataset.make_initializable_iterator()
  return iter.get_next()

SessionRunHook

既然前面說到 estimator 是 tensorflow 對模型的一種封裝，我們不需要也無法拿到訓練和測試時創(chuàng)建的 session，那么我們?nèi)绾卧?estimator 中對上一節(jié)使用 placeholder 的 dataset 的 initializeble_iterator 調(diào)用 sess.run 進行初始化呢？這時候就要用到 SessionRunHook 了。
先從字面意思理解一下 SessionRunHook 這個類。Session 就是 tensorflow 運行模型計算時的會話，Run就是整個 session 運行過程，Hook 是掛鉤的意思即把某些事情掛在這個對象上可以理解為回調(diào)。

再看一下 SessionRunHook 源碼[3]中的定義：
A SessionRunHook extends session.run() calls for the MonitoredSession.
SessionRunHooks are useful to track training, report progress, request early
stopping and more. SessionRunHooks use the observer pattern and notify at the
following points:

• when a session starts being used
• before a call to the session.run()
• after a call to the session.run()
• when the session closed

class SessionRunHook(object):
  """Hook to extend calls to MonitoredSession.run()."""

  def begin(self):
    """Called once before using the session.
    When called, the default graph is the one that will be launched in the
    session.  The hook can modify the graph by adding new operations to it.
    After the `begin()` call the graph will be finalized and the other callbacks
    can not modify the graph anymore. Second call of `begin()` on the same
    graph, should not change the graph.
    """
    pass

  def after_create_session(self, session, coord):  # pylint: disable=unused-argument
    """Called when new TensorFlow session is created.
    This is called to signal the hooks that a new session has been created. This
    has two essential differences with the situation in which `begin` is called:
    * When this is called, the graph is finalized and ops can no longer be added
        to the graph.
    * This method will also be called as a result of recovering a wrapped
        session, not only at the beginning of the overall session.
    Args:
      session: A TensorFlow Session that has been created.
      coord: A Coordinator object which keeps track of all threads.
    """
    pass

  def before_run(self, run_context):  # pylint: disable=unused-argument
    """Called before each call to run().
    You can return from this call a `SessionRunArgs` object indicating ops or
    tensors to add to the upcoming `run()` call.  These ops/tensors will be run
    together with the ops/tensors originally passed to the original run() call.
    The run args you return can also contain feeds to be added to the run()
    call.
    The `run_context` argument is a `SessionRunContext` that provides
    information about the upcoming `run()` call: the originally requested
    op/tensors, the TensorFlow Session.
    At this point graph is finalized and you can not add ops.
    Args:
      run_context: A `SessionRunContext` object.
    Returns:
      None or a `SessionRunArgs` object.
    """
    return None

  def after_run(self,
                run_context,  # pylint: disable=unused-argument
                run_values):  # pylint: disable=unused-argument
    """Called after each call to run().
    The `run_values` argument contains results of requested ops/tensors by
    `before_run()`.
    The `run_context` argument is the same one send to `before_run` call.
    `run_context.request_stop()` can be called to stop the iteration.
    If `session.run()` raises any exceptions then `after_run()` is not called.
    Args:
      run_context: A `SessionRunContext` object.
      run_values: A SessionRunValues object.
    """
    pass

  def end(self, session):  # pylint: disable=unused-argument
    """Called at the end of session.
    The `session` argument can be used in case the hook wants to run final ops,
    such as saving a last checkpoint.
    If `session.run()` raises exception other than OutOfRangeError or
    StopIteration then `end()` is not called.
    Note the difference between `end()` and `after_run()` behavior when
    `session.run()` raises OutOfRangeError or StopIteration. In that case
    `end()` is called but `after_run()` is not called.
    Args:
      session: A TensorFlow Session that will be soon closed.
    """
    pass

我們看到 SessionRunHook 源碼中為 5 中不同的事件提供了回調(diào)函數(shù)，用戶只需要繼承 SessionRunHook 這個類并且具體實現(xiàn)想要的回調(diào)函數(shù)即可，具體用法看下一節(jié)。

estimator 結(jié)合 SessionRunHook 實現(xiàn) placeholder 初始化

仔細看一下 estimator 的 train 和 evaluate 函數(shù)定義可以發(fā)現(xiàn)它們都接收 hooks 參數(shù)，這個參數(shù)的定義是：List of tf.train.SessionRunHook subclass instances. Used for callbacks inside the training loop. 就是上一節(jié)提到的用戶繼承自 SessionRunHook 的類的實例對象列表。

train(
    input_fn,
    hooks=None,
    steps=None,
    max_steps=None,
    saving_listeners=None
)

我們現(xiàn)在想要在訓練之前初始化 dataset 的 placeholder，那么我們就應該具體實現(xiàn) SessionRunHook 的after_create_session 成員函數(shù)：

class IteratorInitializerHook(tf.train.SessionRunHook):
   def __init__(self):
       super(IteratorInitializerHook, self).__init__()
       self.iterator_initializer_fn = None

   def after_create_session(self, session, coord):
       del coord
       self.iterator_initializer_fn(session)

def make_input_fn():
   iterator_initializer_hook = IteratorInitializerHook()

   def input_fn():
       x = tf.placeholder(tf.float32, shape=[None,2])
       dataset = tf.data.Dataset.from_tensor_slices(x)
       dataset = dataset.shuffle(100000).repeat().batch(batch_size)
       iter = dataset.make_initializable_iterator()
       data = np.random.sample((100,2))
       iterator_initializer_hook.iterator_initializer_fn = (
           lambda sess: sess.run(iter.initializer, feed_dict={x: data})
       )
       return iter.get_next()
   return input_fn, iterator_initializer_hook

...
input_fn, iterator_initializer_hook = make_input_fn()
estimator.train(input_fn, hooks=[iterator_initializer_hook])

當然，SessionRunHook 不光能用在初始化上，還有許多應用場景，可以參考源碼[3]中提供的幾個內(nèi)置 Hook 和文獻[4]。

[1] https://github.com/tensorflow/models/tree/master/samples/core/get_started
[2] https://www.jiqizhixin.com/articles/03137
[3] https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/python/training/session_run_hook.py
[4] https://blog.csdn.net/mrr1ght/article/details/81011280