How can i fix jagged edges when rotoscoping moving subjects in after effects?

[GeorgeOB]

I’ve been trying to get better at masking out moving objects for clean composites, but I keep getting these jagged, flickery edges whenever the subject moves quickly. I’m using the rotobrush in After Effects and it feels like I must be missing a step in the process to get smooth results.

[Kenneth29]

I’ve been there with the rotobrush. The jagged edges showed up most when the subject moved quickly across fine textures. My move was to run a rough pass, then switch to refine edge, nudge the feather up a touch, and freeze the frame so I could clean up a couple of tough frames by hand. It slowed me down, but the flicker faded.

[Scott.T]

I also tried turning off motion blur on the layer while refining the matte, then turned it back on after. In several shots that helped stop the edge shimmering and kept the edge a little more stable during fast moves.

[LoganWM]

Do you notice the flicker even when you scrub frame by frame, or only when you hit play?

[Joseph.M]

I once realized it wasn’t just the mask but background compression artifacts sneaking into the matte. I exported a lossless pass for the shot and rechecked the matte on a fresh file, not relying on a single render. Not a magic fix, just a slower workflow that sometimes helped on the next shot.