What causes unstable gradients with hinge loss in multi-label classification?

[Jacob.R]

I’m trying to implement a custom loss function for a multi-label image classification model, but my validation loss is behaving very erratically while accuracy seems fine. I’m worried my formulation might be causing unstable gradients or conflicting with the model's final sigmoid layer. Has anyone else run into this with a hinge-based loss?

[NoraBD]

Yep, I ran into something similar. I tried a hinge style loss on top of a sigmoid for a multi label setup and the validation loss would swing like crazy even when accuracy looked okay. It felt like margins kept flipping on rare classes and the gradients got skittish.

[IsabellaT]

Could it be the margin value or how the labels are mapped to -1 and +1? I had spikes when batches had a lot of negatives and the margin was too aggressive. Do you balance or sample differently per class to help?

[Gregory.T]

I drifted into tweaking the data pipeline a bit, like adding more augmentation and playing with label calibration, but the core issue stuck around. I did notice a calibration drift between val and train that made the hinge depend on where the distribution sat.

[MadisonMS]

I eventually dropped the hinge approach and used BCEWithLogitsLoss instead. Training got steadier and the val loss behaved, even though it wasn't a magic fix. It felt like the sigmoid was fighting the hinge too much, so the change helped.