Why is my multi-label image classification loss dropping while precision stalls?

[Stella69]

I’ve been trying to implement a custom loss function for a multi-label image classification model, but my validation metrics are behaving counter-intuitively—the loss decreases yet precision and recall stagnate or drop. I’m not sure if the issue is in my gradient calculation or if the loss is simply not aligning with the evaluation metrics I care about.

[Paul86]

I had something similar. after I switched to sigmoid outputs with BCEWithLogitsLoss the loss dropped but precision stayed flat. I sanity checked a single weight perturbation and the loss responded okay, so I started looking at evaluation thresholds. Fixed 0.5 was masking improvements in some classes, and when I tried per class thresholds the macro metrics drifted up a bit but nothing dramatic.

[Madison_T]

In my case the gradient looked healthy but the metrics didn't improve. I realized the labels were imbalanced, so I added class weights and even tried focal loss. It helped a little on the rare classes, but overall precision/recall remained stubborn.

[DonaldM]

I also started to wonder if the real issue is the data split. The validation set had different label distribution, so the loss could go down while the model got better for common classes but not the rare ones. I checked per class confusion and saw the pattern.

[Kenneth70]

Do you compute precision and recall at a fixed threshold, or do you optimize the threshold per class? That mismatch is easy to miss and can make everything look off.