使用檢查點(diǎn)支持容錯(cuò)訓(xùn)練 在整個(gè)RLHF訓(xùn)練過程中，可能會(huì)出現(xiàn)訓(xùn)練錯(cuò)誤或機(jī)器故障，因此建議啟用檢查點(diǎn)功能以最小化損失。 API接口已在 :ref:config-explain-...
IP屬地：湖南
-
2025-06-09
-
縮放律與模型架構(gòu)：歸納偏置如何影響縮放？
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay,...
-
UL2：統(tǒng)一語言學(xué)習(xí)范式
UL2: Unifying Language Learning Paradigms https://arxiv.org/abs/2205.05131v3 Yi Tay, Mo...
-
用0.1%的額外計(jì)算超越縮放定律
Transcending Scaling Laws with 0.1% Extra Compute https://arxiv.org/abs/2210.11399 Yi T...
-
大語言模型的涌現(xiàn)能力
Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 Jason Wei,...
-
預(yù)訓(xùn)練指南：測(cè)量數(shù)據(jù)年齡、領(lǐng)域覆蓋率、質(zhì)量和毒性的影響
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Covera...
-
自回歸生成建模的縮放律
Scaling Laws for Autoregressive Generative Modeling Oct 2020 https://arxiv.org/abs/2010...
-
神經(jīng)語言模型的縮放律
Scaling Laws for Neural Language Models Jan 2020 https://arxiv.org/abs/2001.08361 Jared...
-
DoReMi：優(yōu)化數(shù)據(jù)混合加速語言模型預(yù)訓(xùn)練
預(yù)訓(xùn)練數(shù)據(jù)域（如維基百科、書籍、網(wǎng)絡(luò)文本）的混合比例極大地影響了語言模型（LM）的性能。在本文中，我們提出了具有Minimax優(yōu)化的域重新加權(quán)（DoReMi），它首先在域上使...
-
LoRA微調(diào)LoRA: Low-Rank Adaptation of Large Language Models Jun 2021 Edward J. Hu*, Yelong Shen*...
-
LIMA: Less Is More for Alignment
May 2023 https://arxiv.org/abs/2305.11206 [Meta AI, Carnegie Mellon University, Univers...