• 傳統(tǒng)Seq2Seq模型的缺陷
  1）encoder 最后一個(gè) hidden state，與句子末端詞匯的關(guān)聯(lián)較大，難以保留句子起始部分的信息。因此當(dāng)句子過長時(shí)，模型性能下降很快。

  2）句子中每個(gè)詞都賦予相同的權(quán)重的做法是不合理的，這樣沒有足夠的區(qū)分度。

• Attention Based Model 則打破傳統(tǒng) Seq2Seq 模型在編碼解碼的時(shí)候都依賴于內(nèi)部一個(gè)固定長度的向量的限制。

  
  
  Seq2Seq vs. Attention based Model

• Attention Based Model 優(yōu)勢(shì)在于不管序列多長，中間訓(xùn)練出來的 context 能夠保留序列上的每個(gè)輸入點(diǎn)的信息以及序列每個(gè)輸入點(diǎn)之間的關(guān)聯(lián)

• Attention Based Model 架構(gòu)

  • encoder：(multi-rnn) LSTM / GRU
  • attention：softmax + attention metric
  • decoder：(multi-rnn) LSTM / GRU

Attention Model

• Attention Metric
  相對(duì)于原本的Seq2Seq模型，
  
  加入Attention機(jī)制的模型，
  
  
  相當(dāng)于每個(gè)輸出位置有一個(gè)對(duì)應(yīng)的向量。其中， 表示輸入經(jīng)過encoder層嵌入后得到的結(jié)果

• Attention Model 實(shí)現(xiàn)

self.encoder_inputs = tf.placeholder(tf.int32, shape=[None, None], name='encoder_inputs')
self.decoder_inputs = tf.placeholder(tf.int32, shape=[None, None], name='decoder_inputs')
self.decoder_targets = tf.placeholder(tf.int32, shape=[None, None], name='decoder_targets')
self.encoder_length = tf.placeholder(tf.int32, shape=[None], name='encoder_length')
self.decoder_length = tf.placeholder(tf.int32, shape=[None], name='decoder_length')

with tf.variable_scope("embedding"):
    encoder_embedding = tf.Variable(tf.truncated_normal(shape=[self.options["vocab_size"], self.options["embedding_dim"]], name='encoder_embedding'))
    decoder_embedding = tf.Variable(tf.truncated_normal(shape=[self.options["vocab_size"], self.options["embedding_dim"]], name='decoder_embedding'))
    # embedding
    self.encoder_inputs_embedded = tf.nn.embedding_lookup(encoder_embedding, self.encoder_inputs)
    self.decoder_inputs_embedded = tf.nn.embedding_lookup(decoder_embedding, self.decoder_inputs)

with tf.name_scope("encoder"):
    # encoder 
    encoder_layers = [tf.nn.rnn_cell.BasicLSTMCell(self.options["cell_size"]) for _ in range(2)]
    encoder = tf.nn.rnn_cell.MultiRNNCell(encoder_layers)
    # encoder output
    encoder_all_outputs, encoder_final_state = tf.nn.dynamic_rnn(encoder, self.encoder_inputs_embedded, sequence_length=self.encoder_length, dtype=tf.float32, time_major=False)
    if self.is_summaries:
       tf.summary.histogram('encoder_all_outputs', encoder_all_outputs)

with tf.name_scope("decoder"):
    # decoder init
    decoder_layers = [tf.nn.rnn_cell.BasicLSTMCell(self.options["cell_size"]) for _ in range(2)]
    decoder_cell = tf.nn.rnn_cell.MultiRNNCell(decoder_layers)
    # attention
    attention_mechanism = LuongAttention(num_units=self.options["cell_size"], memory=encoder_all_outputs, memory_sequence_length=self.encoder_length)
    # decoder & attention concat
    attention_decoder = AttentionWrapper(cell=decoder_cell, attention_mechanism=attention_mechanism, alignment_history=True, output_attention=True)
    attention_initial_state = attention_decoder.zero_state(tf.shape(self.encoder_inputs)[0], tf.float32).clone(cell_state=encoder_final_state)
    # assign decoder input 
    train_helper = TrainingHelper(self.decoder_inputs_embedded, self.decoder_length)
    # define decoder output_layer
    fc_layer = tf.layers.Dense(self.options["vocab_size"])
    train_decoder = BasicDecoder(cell=attention_decoder, helper=train_helper, initial_state=attention_initial_state, output_layer=fc_layer)
    # decoder output
    logits, final_state, final_sequence_lengths = dynamic_decode(train_decoder)
    decoder_logits = logits.rnn_output
    self.train_attention_matrices = final_state.alignment_history.stack(name="train_attention_matrix")
    if self.is_summaries:
        tf.summary.histogram('decoder_logits', decoder_logits)
        tf.summary.histogram('train_attention_matrix', self.train_attention_matrices)


with tf.name_scope("train"):
    maxlen = tf.reduce_max(self.decoder_length, name="mask_max_len")
    mask = tf.sequence_mask(self.decoder_length, maxlen=maxlen, dtype=tf.float32, name="mask")
    decoder_labels = tf.one_hot(self.decoder_targets, depth=self.options["vocab_size"], dtype=tf.int32, name="decoder_labels")
    stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(labels=decoder_labels, logits=decoder_logits, name="cross_entropy")
    _loss = tf.multiply(stepwise_cross_entropy, mask)
    self.loss = tf.reduce_sum(_loss, name="loss")
    optimizer = tf.train.AdadeltaOptimizer()
    self.train_op = optimizer.minimize(self.loss)
    if self.is_summaries:
        tf.summary.histogram('loss', self.loss)