What should I do when model performance plateaus with more training data?

[Evelyn.L]

I’ve been trying to improve my model’s performance on a specific task, but I’m hitting a wall where adding more training data isn’t helping and might even be making things slightly worse. I’m starting to think the issue is with the quality and relevance of the data I’m collecting, not the quantity. Has anyone else run into this kind of plateau during their own projects?

[Jack88]

I hit that wall last year. I kept adding samples and the model barely moved, then I found a bunch of mislabeled examples in the new batch. Cleaning and re-labeling helped, but the gains were small. It wasn’t about more data; it was how clean and relevant that data was for the task. After we curated a smaller set with better data quality, the validation curve moved a touch.

[SavannahOL]

Me too. I kept collecting until I realized the distribution shifted between train and production, so more data just added drift. We ended up cutting data from noisy sources and tightening our labeling guidelines. Validation scores stabilized but didn’t surge.

[Noah95]

A quick thing I tried was a stricter holdout test and then I dropped a whole batch because it felt off but I wasn’t sure why. The metrics looked worse on that batch, so we paused and regrounded; still not sure if the issue was data or something else.

[Addison_W]

Do you think the evaluation setup actually reflects real use, or are we chasing a metric that isn’t the real goal? I kept wondering if maybe the problem is something else and the data isn’t the bottleneck after all.