What tips fix jagged edges when rotoscoping with rotobrush in after effects?

[George87]

I’ve been trying to get better at masking out moving subjects, like a person walking in front of a busy background, but my edges always look jagged and unnatural. I’m using the rotobrush in After Effects and it just never seems to track cleanly on finer details like hair or fast motion, even after tweaking the feather and contrast settings.

[EricLG]

I've been in the same boat. I spent hours chasing clean edges on a walk sequence. Hair is the worst with rotobrush; even when I thought I had it, a stray strand would curl and the mask would look jagged. Feather and contrast helped a bit, but the transitions still felt like strobe frames.

[AdamDM]

One thing I did was cut the shot into shorter passes and re-rotoscope each piece. It felt slower, but the result stayed more stable and I didn’t have to chase every frame all at once.

[David98]

Sometimes I wonder if the real issue is lighting and background motion. If the subject treads in front of a busy background, the edge contrast changes frame to frame, and the tool can’t settle. On windy days the hair moves too much and the edge looks off in the next frame.

[Ryan_G]

Do you think the issue is motion blur? I keep thinking the edge detection is fighting blur and can’t lock on fine details.