How can I fix gradient issues with focal loss on imbalanced multi-label data?

[Samuel_A]

I’m trying to implement a custom loss function for a multi-label classification problem, but my model’s performance just plateaus. I think the issue might be with my gradient computation for the focal loss component—the training simply doesn’t converge properly. Has anyone else run into this when tweaking the loss function for imbalanced datasets?

[EvelynDW]

Yeah I ran into that when I tweaked focal loss for a multi label task. the gradient signal on the hard classes stayed decent but the easy ones drowned it out and the loss just kept hovering. I tried bumping gamma down and adding per class alpha, but the plateau didn’t disappear.

[Noah.H]

I did a few concrete things: sanity checked grads, lowered gamma, added class weights, and then I tried a very simple BCE with logits baseline. It moved the needle a bit on minority labels but the overall curve still plateaued and training got stuck around the same accuracy.

[VictoriaQR]

Could the real issue be data quality or label noise rather than the loss itself? Sometimes the model is already memorizing the easy patterns and the imbalance is masking problems in the labels or distribution shifts between train and test.

[Brandon33]

I once implemented the gradient by hand and kept hitting NaNs, so I swapped to a stable autograd implementation from a library. Training felt cleaner after that, but the plateau remained. I started logging per class losses and noticed some classes never started to decrease.