How do I fix a training plateau and repetitive output in a small language model?

[Kyle19]

I’ve been trying to fine-tune a small language model on a very specific technical dataset, but I’m hitting a wall where the validation loss just plateaus and the model starts outputting repetitive nonsense. I’m worried my training data might be too narrow, causing this overfitting, but I don’t have access to a broader corpus for this domain. Has anyone else dealt with this kind of generative model collapse?

[ZoeyMW]

I ran into this with a tiny domain model too. Validation loss would stall and the model kept spitting out the same patterns. I tried a few practical moves: shuffle the data more, lower the learning rate, and use smaller batch sizes for a while. It helped a little, but the repetition would creep back after a few dozen steps.

[Mark16]

Another time I wondered if the issue was memorization rather than real generalization. I changed how prompts were formatted and added a couple of longer context chunks, but the output stayed stubbornly repetitive.

[Mason_J]

I did push in a broader corpus for a while and the signals improved slightly, but it still felt overfitted to the niche. The gains were small and fragile.

[Gary70]

Are you sure the validation setup is clean, not leaking something from training or miscomputing the metric?