Hugging Face and EleutherAI launched FineBooks, a project to solve a critical data quality problem that degrades language model training. Historical texts digitized through optical character recognition introduce corruption that propagates into AI systems trained on this data.
The team tested 14 open-source OCR models across 2,000+ historical book pages to find the best performer. The leader, dots.mocr, achieved 97.6% character accuracy while costing under two dollars per thousand pages. This performance level works for training data but falls short of what scholars need for accurate transcription work.
OCR errors compound when language models learn from flawed text. A misspelled word or garbled phrase becomes embedded in the model's weights, degrading output quality downstream. This problem scales across decades of digitization projects that fed historical books into training datasets without quality control.
The FineBooks research reveals the economics and technical trade-offs of large-scale document cleaning. At sub-two-dollar pricing, dots.mocr makes it feasible to re-process millions of pages from public archives like Google Books and Project Gutenberg. The 2.4% error rate remains acceptable for models that learn statistical patterns despite noise, but researchers using the same texts for close reading cannot rely on the transcriptions.
This work highlights infrastructure gaps in AI development. Language models inherit the quality of their source materials. Fixing OCR at scale requires choosing between cost and accuracy. The FineBooks team's benchmarking gives practitioners concrete numbers to make that choice.
The project matters for open-source model development. Researchers building competing systems with limited budgets now have evidence that spending two dollars per thousand pages on OCR correction improves training data quality without massive expense. The specific performance numbers from dots.mocr set a baseline for future optimization efforts.