寫在前面

1. 關(guān)于parsing，目前有兩種主流的方法，一種是Constituency Parsing，另一種是Dependency Parsing，在本文中我們主要討論Dependency Parsing
2. 知識點(diǎn)參見: 阿衡學(xué)姐的筆記 Dependency Parsing and Treebank
3. Stanford cs224N講義: Dependency Parsing
4. Basic論文: A Fast and Accurate Dependency Parser using Neural Networks，基于greedy transition-based parsing，結(jié)合了word_embedding和neural networks
5. Advanced論文
   Globally Normalized Transition-Based Neural Networks
   Universal Dependencies: A cross-linguistic typology
   Incrementality in Deterministic Dependency Parsing
   主要改進(jìn)點(diǎn)有: 使用更深層次的神經(jīng)網(wǎng)絡(luò)，處理non-projectivity問題，使用graph-based parsing
6. 基礎(chǔ)項(xiàng)目: Neural Dependency Parsing

Neural transition-based Dependency Parsing

Conventional transition-based Dependency Parsing

1. Structure: Stack s, Buffer b and A(record dependencies)
2. At each move, use a discriminative classifier(such as SVM) to decide what kind of operation next time

transition_based parsing.png

Feature Extract

1. In conventional ways, we represent word|tag|label as one-hot vectors, which is sparse and costs much time
2. In Neural transition-based dependency parsing model, we represent word|tag|label as pre-trained word_embeddings

feature_extracted.png

Choices of featurs

The choice of Sw, St, Sl
Following (Zhang and Nivre, 2011), we pick a rich set of elements for our final parser. In de- tail, Sw contains nw = 18 elements: (1) The top 3 words on the stack and buffer: s1, s2, s3, b1, b2, b3; (2) The first and second leftmost / rightmost children of the top two words on the stack: lc1(si), rc1(si), lc2(si), rc2(si), i = 1, 2. (3) The leftmost of leftmost / rightmost of right- most children of the top two words on the stack: lc1(lc1(si)), rc1(rc1(si)), i = 1, 2.
We use the corresponding POS tags for St (nt = 18), and the corresponding arc labels of words excluding those 6 words on the stack/buffer for Sl (nl = 12). A good advantage of our parser is that we can add a rich set of elements cheaply, instead of hand-crafting many more indicator fea- tures.

Neural Networks

after word_embedding,we feed the input to hidden-layer, which has a novel activation function(f(x) = x^3)，then use a softmax-layer to predict next transition

neural_model.png