Data curation at the scale of pretraining

Abstract

While unsupervised learning has unlocked an enormous amount of data for models to train on, it abandoned previous notions of data quality like cleanliness, and it became unclear what data should be included in pretraining. This talk will go over the sensitivity results in pretraining, and the challenges in building new tools for assessing data quality from a statistical and systems perspective.