LIMA: Pretraining Method on 1000 Examples Achieved GPT4-Level Accuracy
31 May 2023
LIMA: Pretraining Method on 1000 Examples Achieved GPT4-Level Accuracy
Language models typically undergo two stages of training: unsupervised pretraining and fine-tuning to specific tasks and user preferences. The novel LIMA method (Less Is More for Alignment) challenges the traditional…