筆記，閱讀的時(shí)候?qū)懴拢浶圆惶谩！?br> Bidirectional Encoder Representations from Transformers

介紹

在預(yù)訓(xùn)練bert結(jié)尾加輸出層，就可以實(shí)現(xiàn)多種任務(wù)的最先進(jìn)的模型。As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the- art models for a wide range of tasks.
不需要為多種任務(wù)構(gòu)建不同的模型。e.g. 問(wèn)答和語(yǔ)言推理question answering and language inference

貢獻(xiàn)點(diǎn)：

1. BERT使用深度雙向的模型，超越了之前的單向和淺層雙向。
2. 第一個(gè)通過(guò)fine-tune就能實(shí)現(xiàn)state of art結(jié)果
3. BERT為11個(gè)NLP任務(wù)提供了最先進(jìn)的技術(shù)
   總之，模型簡(jiǎn)單， 但是效果好。

在bert之前有：
feature-based：需要定義結(jié)構(gòu)，e.g.ELMo
fine-tuning：優(yōu)化參數(shù)， Generative Pre-trained Transformer (OpenAI GPT)
both using unidirectional language models to learn general language represen- tations.

bert訓(xùn)練: masked language model (MLM) 隨機(jī)將句子中的某個(gè)詞給抹掉，然后根據(jù)左右上下文預(yù)測(cè)。
next sentence prediction

現(xiàn)有task結(jié)果：
GLUE：80.4%
MultiNLI：86.7%
SQuAD v1.1 ： F1 to 93.2
sentence level tasks：natural language inference， paraphrasing。
token-level tasks：named entity recognition， SQuAD question answering

相關(guān)工作：

1. Feature-based Approaches：
   1.1 Pre-trained word embeddings
   1.2 sentence embeddings
   1.3 paragraph embeddings
   1.4 ELMo 他們建議從語(yǔ)言模型中提取上下文敏感的特性。將上下文嵌入與特定于任務(wù)的架構(gòu)集成。
2. Fine-tuning Approaches： transfer learning
   基于很少有參數(shù)需要重頭學(xué)，OpenAI GPT搞了
3. Transfer Learning from Supervised Data

架構(gòu)：

multi-layer bidirectional Transformer encoder
與這個(gè)相同，可以參考：Vaswani et al. (2017) as well as excellent guides such as “The Annotated Transformer.”

BERT-BASE: L=12, H=768, A=12, Total Pa- rameters=110M
BERT-LARGE: L=24, H=1024, A=16, Total Parameters=340M
BERT-BASE was chosen to have an identical model size as OpenAI GPT for comparison purposes

輸入介紹

image.png

第一個(gè)都是[CLS]對(duì)于分類(lèi)任務(wù)最后是[SEP]，非分類(lèi)任務(wù)不需要

• We use WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary. We denote split word pieces with ##.
• We use learned positional embeddings with supported sequence lengths up to 512 tokens.

架構(gòu)對(duì)比

image.png

GPT is trained on the BooksCorpus (800M words); BERT is trained on the BooksCor- pus (800M words) and Wikipedia (2,500M words).
GPT used the same learning rate of 5e-5 for all fine-tuning experiments; BERT chooses a task-specific fine-tuning learning rate which performs the best on the development set.

具體訓(xùn)練

pre-train BERT using two novel unsupervised prediction tasks
Masked LM. mask 15% one sentence。
Next Sentence Prediction:
50% of the time B is the actual next sentence that follows A, and 50% of the time it is a random sentence from the corpus.
They are sampled such that the combined length is ≤ 512 tokens.

image.png

image.png

training loss is the sum of the mean masked LM likelihood and mean next sentence prediction like- lihood.

實(shí)驗(yàn)：

GLUE : The General Language Understanding Evaluation (GLUE) benchmark
is a collection of diverse natural language understand- ing tasks.

數(shù)據(jù)集介紹：

Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.

MNLI：

Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classifi- cation task
Given a pair of sentences, the goal is to predict whether the second sentence is an entailment, contradiction, or neutral with respect to the first one.
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.

QQP：

Quora Question Pairs is a binary classifi- cation task where the goal is to determine if two questions asked on Quora are semantically equivalent
Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs.

QNLI Question Natural Language Inference is a version of the Stanford Question Answering Dataset (Rajpurkar et al., 2016) which has been converted to a binary classification task (Wang et al., 2018). The positive examples are (ques- tion, sentence) pairs which do contain the correct answer, and the negative examples are (question, sentence) from the same paragraph which do not contain the answer.

SST-2 The Stanford Sentiment Treebank is a binary single-sentence classification task consist- ing of sentences extracted from movie reviews with human annotations of their sentiment (Socher et al., 2013).

CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where the goal is to predict whether an English sentence is linguistically “acceptable” or not (Warstadt et al., 2018).

STS-B The Semantic Textual Similarity Bench- mark is a collection of sentence pairs drawn from news headlines and other sources (Cer et al., 2017). They were annotated with a score from 1 to 5 denoting how similar the two sentences are in terms of semantic meaning.

MRPC Microsoft Research Paraphrase Corpus consists of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semanti- cally equivalent (Dolan and Brockett, 2005).

RTE Recognizing Textual Entailment is a bi- nary entailment task similar to MNLI, but with much less training data (Bentivogli et al., 2009).6
WNLI Winograd NLI is a small natural lan- guage inference dataset deriving from (Levesque et al., 2011). The GLUE webpage notes that there are issues with the construction of this dataset, 7 and every trained system that’s been submitted to GLUE has has performed worse than the 65.1 baseline accuracy of predicting the majority class. We therefore exclude this set out of fairness to OpenAI GPT. For our GLUE submission, we al- ways predicted the majority class

SQuAD v1.1：span prediction task
100k crowdsourced question/answer pairs

image.png

At inference time, since the end prediction is not conditioned on the start, we add the constraint that the end must come after the start, but no other heuristics are used.（end必須在start之后）

Named Entity Recognition： CONLL
final hidden representation Ti ∈ RH for to each token i into a classification layer over the NER label set.
A cased WordPiece model is used for NER, whereas an uncased model is used for all other tasks.

image.png

分析

增加模型參數(shù)提高結(jié)果

Question: Does BERT really need such a large amount of pre-training (128,000 words/batch * 1,000,000 steps) to achieve high fine-tuning accuracy?
Answer: Yes, BERTBASE achieves almost 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps.

不同訓(xùn)練次數(shù)的影響

如果將bert作為特征的話，拼接后四層效果最好

不好重現(xiàn)結(jié)果，因?yàn)殚_(kāi)發(fā)機(jī)的內(nèi)存不同，一次能夠處理的batch不同~