Why is my gradient flow unstable with focal loss in multi-label classification?

[AvaJ]

I’m trying to implement a custom loss function for a multi-label image classification model, but my validation loss is behaving erratically while accuracy seems fine. I’m worried the gradient flow might be unstable due to my implementation of the focal loss component.

[Aurora_J]

That sounds like a real pain. I fixed similar issues by stacking a focal style factor on top of the standard logits loss. Compute p = sigmoid(logits); p_t = p if y=1 else 1-p; loss = BCEWithLogitsLoss(reduction='none')(logits, y) * (1 - p_t) ** gamma * alpha_t. Then take the mean. Also clamp p to [1e-6, 1 - 1e-6] to avoid numerical issues.

[EleanorC]

I had NaNs once; I added a tiny epsilon to p_t before raising to gamma and used reduction='none' so I could inspect grads. I also avoided in place ops to keep autograd clean.

[DanielB]

If targets were shaped differently for multi label, the loss would misbehave; double-check you’re not applying softmax across classes or mixing up dims.

[AndrewT]

Could the issue be data distribution in val vs train? Accuracy looks fine but loss jumps when a rare class shows up in a batch.

[EleanorGW]

I learned not to apply the same gamma to all samples; make sure the focal term uses p_t based on the true label; otherwise gradients skew.

[RonaldND]

Per class losses showed one class dominating; adding per class weighting or a custom alpha for that class helped calm the loss.

[Nora_M]

Sometimes I drifted into wondering if the val loss is telling me something about the data pipeline; maybe augmentation changed the distribution; still not sure if the loss is the root cause.