EleutherAI

We build open-source AI infrastructure, pioneer interpretability and open-weight safety, and help researchers without frontier-lab resources do frontier work. Open research on large models requires open training data, open training and evaluation code, and models whose full development is public. We have built all three since 2020. Our datasets include the Pile and the Common Pile. Our training framework is GPT-NeoX and our evaluation framework is the LM Evaluation Harness. Model suites such as Pythia are released with their training data, code, and intermediate checkpoints. All of it is openly licensed and in use across the research community. Most interpretability research studies a finished model. We study how model internals develop over the course of training, and we have argued that a science of AI must study training dynamics rather than only the final artifact. The Pythia suite was built for this purpose. It provides checkpoints throughout training together with the exact data each checkpoint had seen, and it is now widely used in research on learning dynamics, memorization, and how capabilities form. Once model weights are public, safeguards cannot rely on API monitoring or on post-training methods that fine-tuning can remove. We work on safety methods that remain effective when anyone can inspect and modify the model, which in practice means intervening in the training data and the training process.