【Sentence-bert】
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
論文: https://arxiv.org/abs/1908.10084
代碼: https://github.com/UKPLab/sentence-transformers
【Bert-flow】
On the Sentence Embeddings from Pre-trained Language Models
論文: https://arxiv.org/abs/2011.05864
代碼: https://github.com/bohanli/BERT-flow
動(dòng)機(jī): 論文作者認(rèn)為BERT生成的句向量具有的語(yǔ)義相似度信息無(wú)法通過(guò)簡(jiǎn)單的余弦相似度體現(xiàn)出來(lái)，論文通過(guò)一個(gè)基于流(flow)的生成模型訓(xùn)練一個(gè)標(biāo)準(zhǔn)的高斯分布，對(duì) BERT句向量空間中的點(diǎn)進(jìn)行一一映射，再在標(biāo)準(zhǔn)高斯空間中進(jìn)行相似度計(jì)算
【Bert-whitening】
Whitening Sentence Representations for Better Semantics and Faster Retrieval
論文: https://arxiv.org/abs/2103.15316
代碼: https://github.com/bojone/BERT-whitening
作者認(rèn)為Bert-flow模型的關(guān)鍵不在于flow模型，基于Bert模型輸出的[CLS]向量在文本語(yǔ)義計(jì)算任務(wù)中無(wú)法取得好的效果的原因可能是句向量所屬的坐標(biāo)系并非標(biāo)準(zhǔn)正交基的推論，通過(guò)一個(gè)簡(jiǎn)單的線性變換實(shí)現(xiàn)了和Bert-flow相近的效果。
參考:
細(xì)說(shuō)Bert-whitening的原理
https://zhuanlan.zhihu.com/p/380874824
你可能不需要BERT-flow：一個(gè)線性變換媲美BERT-flow
https://kexue.fm/archives/8069
【ConSERT】
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
論文: https://arxiv.org/pdf/2105.11741.pdf
代碼: https://github.com/yym6472/ConSERT
參考:
論文閱讀“ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer”
http://www.itdecent.cn/p/de79f4e166da
【SimCSE】
SimCSE: Simple Contrastive Learning of Sentence Embeddings
論文: https://arxiv.org/pdf/2104.08821.pdf
代碼: https://github.com/princeton-nlp/SimCSE
參考:
【論文復(fù)現(xiàn)】SimCSE對(duì)比學(xué)習(xí): 文本增廣是什么牛馬，我只需要簡(jiǎn)單Dropout兩下
https://blog.csdn.net/weixin_45839693/article/details/116302914
【Trans-Encoder】
TRANS-ENCODER: UNSUPERVISED SENTENCE-PAIR MODELLING THROUGH SELF- AND MUTUAL-DISTILLATIONS
論文://arxiv.org/pdf/2109.13059v3.pdf
代碼 ：https://github.com/amzn/trans-encoder
參考：
文本匹配——【ICLR 2022】Trans-Encoder
https://cloud.tencent.com/developer/article/2041362
TRANS-ENCODER：自蒸餾和互蒸餾的無(wú)監(jiān)督句對(duì)模型
https://zhuanlan.zhihu.com/p/437729611
【Poly-encoders】
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
論文: https://arxiv.org/pdf/1905.01969.pdf
參考:
論文筆記 | Poly-encoders
https://zhuanlan.zhihu.com/p/380867813
【PairSupCon】
Pairwise Supervised Contrastive Learning of Sentence Representations
論文: https://arxiv.org/abs/2109.05424
代碼: https://github.com/amazon-research/sentence-representations/PairSupCon
參考:
Pairwise Supervised Contrastive Learning of Sentence Representations
https://zhuanlan.zhihu.com/p/421196130
