How to Use Roto Brush 2 in After Effects

After Effects recently added Roto Brush 2 to their After Effects beta version, meaning editors can start using this new AI-powered tool. If you want to quickly create amazing green screen effects without a green screen, this tool is your answer.

What is Roto Brush?

Roto Brush is a tool that editors can use in Adobe After Effects to create a green screen effect on a shot that had no green screen. This accomplishes the same goal as rotoscoping, which involves manually drawing bezier curves to create a mask. The Roto Brush is an alternative to this technique that lets you draw with the brush on some parts of the foreground and background, and After Effects will determine what is foreground and background in each frame, based on that information. In a best-case scenario, the tool will be able to adjust based on that information and analyze the whole clip. In the end, you will have a separate subject and background, so you can put that subject onto a new background.

Why is Roto Brush 2 Better?

Roto Brush 2 functions much the same as the original Roto Brush, but it improves upon the underlying technology. The result will be much smoother, thanks to the advanced Sensei AI technology behind this version of the tool.

How to Use Roto Brush 2

First of all, if you’re trying to use Roto Brush 2 before its official release, you will need to download the After Effects beta version from Adobe Creative Cloud. As long as you’re a subscriber, you should have this option. Simply go to the Beta Apps tab and download the After Effects beta version.

Once you open After Effects, you’ll see the Roto Brush tool in the same place as the original version. If you hover over it, you will see that it says to drag over foreground, and Alt+drag over background. Double-click the video layer in your timeline and select the Roto Brush tool. Paint with brush strokes over your foreground. You will see an outline as its selected, and each subsequent stroke will add more to the mask. Just make sure not to brush outside of the bounds of your subject, to get the most accurate mask. If it does start to select something in the background, you can hold Alt (Option on Mac) while dragging over that area, to select the background.

To change the brush size, hold Ctrl and left-click and move the mouse up or down. Reducing the brush size will help you refine the small details around your subject. In traditional rotoscoping, you need to spend a lot of time getting precise cuts frame-by-frame, but with the Roto Brush, you only need to do that once. But it’s very important that you spend the time to make this mask as accurate as possible, so that the program can get the best tracking data for past and future frames.

Once you’re happy with your cutout, we can start the Roto process. Click on the first frame in your timeline and advance the video frame-by-frame using the arrow keys while holding Ctrl. Roto Brush will analyze and apply to each frame as you do this. Doing it this way will let you notice any errors in the roto process. Then you can just use the brushes to correct the mistake. If it all looks good, you can click the Freeze button under the program monitor. This will save your roto and you’ll be able to use the subject you’ve cut out, on any other background!

Fine Tuning Your Roto

Before doing that, however, you should fine-tune your roto. In the Roto Brush settings, you can adjust the feather, contrast, shift edge, and reduce chatter. This can help clean up the subject and make the cutout smoother. Adjusting these settings can get rid of any harsh edges, and it’s much easier to see them if your subject is already separated from the background. Don’t adjust these settings from default when you’re doing the initial brushing. If your subject is in motion, you may want to turn on Use Motion Blur. And if your subject has some color bleed around their edges from the previous background, you can select Decontaminate Edge Colors, and After Effects will attempt to remove this color.

If you find a mistake in your roto later on, you can always go back to it by double-clicking the layer on your timeline, unfreezing it, and using the Roto Brush tool to paint around any mistakes. To save this, just freeze it again.

By long-pressing the Roto Brush icon, you will see a drop-down that includes the Refine Edge Tool. This tool is used to refine the selection around things like hair. Users of Photoshop may recognize this tool. Paint with the Refine Edge brush along the line that selects your subject’s hair. You can select X-Ray view to view these fine details better. Much like before, go frame by frame to apply the roto and adjust any mistakes along the way. Freeze the roto when you’re done, then adjust the feather and other controls as needed.

Finalizing the Edit

When you’re done with the roto, just go back to the main composition and you will see your subject cut out from your background. To put it on a new background simply drag a new video layer underneath the cut-out subject, then apply color correction and scaling to make the subject fit better in your scene.

What’s the Difference?

Using Roto Brush 2 will be a faster and less frustrating experience than the first version. Thanks to the AI-powered algorithm working behind the scenes, Roto Brush 2 is extremely accurate. You will spend less time fixing mistakes from the tool, and more time actually compositing your scene. All around, the tool functions more efficiently and is just smarter. It also can utilize the GPU rather than the CPU, meaning you could see Roto Brush 2 running incredibly smooth.

---

Now you can use the Roto Brush 2 to speed up your compositions and create seemingly-impossible edits in literal minutes. Let us know in the comments if you used the original Roto Brush and if you’re excited to use this new version!

To learn more about using After Effects, check out our complete crash-course guide to After Effects.