How can i fix jagged edges when masking moving subjects in vfx?

[KevinR]

I've been trying to get better at masking objects for VFX shots, but I keep running into this jagged, pixelated edge on moving subjects no matter how carefully I draw my bezier curves. It's especially bad when I try to pull a clean key from footage with motion blur. Is this just a limitation of the planar tracker in my software, or is there a fundamental step I'm missing in my workflow?

[Emily.B]

I've run into that too. Even when the curves feel tight, the edge shows jaggies once motion blur and subpixel sampling kick in. Planar trackers are fast, not magic; the mask is only as good as the edge it can separate, and with moving, blurred subjects you basically cut through a smeared halo. It feels more like a sampling problem than a user error.

[TimothyQR]

In a few shots I thought I had it, but the edge stayed pixelated in the walk cycles. I started blaming the footage but then realized the issue shows up more when the plate is compressed or your render uses a different color space. When I export a quick lossless pass, the jaggies soften a bit, but as soon as the final render comes through with the normal pipeline they reappear.

[Robert_J]

One concrete thing I tried was adding a tiny manual roto for the critical frames and feathering the edge a lot. It helped, but it also created a mismatch between the tracked area and the rest of the shot when the subject turned or changed speed. I kept a few frames as-is and did the rest by hand.

[EmmaR]

Could the real problem be that the blur and aliasing are baked into the plate, not something you can fix with a mask alone?