Using the new selective mask feature to remove backgrounds in Photoshop

In today’s post, Jeff will show you how to remove backgrounds in Photoshop using the new selective mask feature.

Note: remember to watch the video tutorial Jeff made for you, it will be much easier to follow the steps.

There is simply no way in After Effects for sure that you could select the hair very well on a video like this. On a still image it would be pretty tricky but here it is relatively easily.

Here’s the way we do that, we first select things that have hard edges like her clothing and her neck.

Then we make a selection inside the hair, not to the edge but inside it.

Then we refine that inside the selected mask option.

So that’s the strategy here. Let’s see how to do that!

You first go to the quick selection tool and select things that have distinct edges like her clothes okay, her neck also.

quick selection tool

Now I want to select up here but not go up to the edges.  I want to use the selected mask tool to do that.

I just need a lasso for that, I go to the lasso tool that’s this guy up here like a laureate like a cowboy. The lasso tool and I want it to not make a new selection, I want to add it to the current selection so I click on this.

add to selection

I just draw very simply, just draw close to where the edges are but not to the edge. Now I’ve created this mask here


this selection that does not go to the place where there’s any kind of gaps in her hair. I can expand it by just going over like that, I think that’s pretty good.

Now with the selection active, with the layer active I need to worry about making a duplicate layer now because I do that inside the selected mask feature, click on this.

Select and mask

Now we’re here, obviously it looks terrible right.

We’re going to change the view here in this little drop down list to the overlay which is kind of the standard way you work with the selected mask tool.


I keep on wanting it to call the refine edge tool, that’s what the old name was.

Here’s the video tutorial:

Join The After Effects Academy And Get Acces To This Full Lesson + Monthly Live Classes And After Effects Templates

Click away and then you can increase or decrease the opacity, this is a new feature which is nice.

You can change the color as well in case the background color is something other than red, you can change it to some other color. If the background were red you can change it to some other color here, red works fine.

Now what I’m going to do is I’m going to refine the edge here using this tool over here

Refine Edge Brush Tool

they call it the refine edge brush tool.

I’ll refine it so they look harder at things that have edges to them. You’re basically telling Photoshop I want you to go beyond this area and look a little bit harder.  And I want to be able to see that as I do that.

I say show edge up here, right now there is no edge. I have to then tell it how much do  I want to see, so I go to edge detection here

Edge Detection

and I say let’s increase the radius of the edge. Now that’s the edge and it’s going to look at things in the edge.  I don’t want the edge to be really big down here, that’s not necessary.

But I can fix that by doing smart radius later so I’m just going to make it like this.  I’m going to tell it to look beyond where I drew like so.
Then I’m going to draw farther using the refined edge tool over here, it’s literally called the refined edge brush tool.

I’m going to make the brush bigger by holding the alt and option key and dragging it to the right to make it bigger.

How to make the brush bigger

I’m going to make it softer too by dragging down, dragging up.

After Effects Academy Member Extra

Access your ‘Creating Photoshop layer masks to use in AE’ Course in the After Effects Academy.

Jeff Sengstack will teach you how to create layer masks in Photoshop.

Not an AE Academy member? Click here to learn more about the After Effects Academy

Now I’m going to paint around the edge here to say

look here at Photoshop for things that have edges to them.

Look over here a little bit more carefully, I’m just clicking here there’s no magic thing I’m doing I’m just clicking.  I want you to work harder, that’s too much I’ll use control command Z.  Let’s make you smaller just right there clicking where I want it to do a little bit more work. That should do it.

I don’t want to expand the selection so I turn on smart radius and I can reduce the radius here and it will try to compress the areas that are obviously hard edged.

You can’t go below 10 with the smart radius you got to stay at 10 for the smart radius to work so I turn it back on again at 10 there.

Now it’s not going to work hard to try and find edges here but it’s going to try and find edges there. Maybe a little bit further out there.Just doing all that should be sufficient to actually select the hair and to see how we did we don’t show the edge anymore.

We show the effect to see if that worked on that, like it could be a little better there. I wonder why it came back so poorly, paint a little bit harder there.  It could have worked better it’s showing too much red through there I wonder why that is.


Let me just brush with the brush tool instead that paints with white to fix this area there so that should improve the quality there.

Since it worked hard to find edges here but it actually worked too hard to find edges here.  I’m going to do it one more time, one more crack at this thing again.

I’ll go back to the refine edge tool here, show edge again, the radius a bit.

Now let’s try again.

All right there, I want to go out, the brush is too big. I want to go out, right outside the edge there where it’s red but where there’s hair, right there.  It should work a little bit better now we got the inside figured out.

I’ll turn off the show edge and see if that works, that’s better okay.

It’s not perfect but it’s better.
Let’s say I want to then now output this, so I’ll output it to a new layer with the mask.  Click okay.

Output to a new layer with a mask

There’s that new layer with the mask but yeah when you have that much frizz it’s  pretty darn hard to make a perfect selection so I left her arm there sorry. I’ll fix that real quick.  Click the brush and paint it white with a paintbrush, I hate to have it happen like that. That’s terrible.
I’ll make the brush a little bit larger there we go and paint that in. I’m sorry about losing your arm there.


I’m going to make the background darker so at least you can see how that works. I’ll make the background change and I need to turn it to something to set the background layer so I’ll double click on it.  That makes it something other than the background layer. The background layer has very special characteristics.

Then I can apply a blur effect to that, we go to filter, blur, and we’ll just go to increase the blur like so.

Pretty obviously blurred like that and you’ll see her hair is not blurred which is good.

It’s pretty hard to let’s say I’ll put several solid things into her background it will be pretty obvious there’s some problems here.

With that much frizz that’s a pretty good thing, let’s make it even more obvious by just applying an adjustment layer above there, curves and we’ll bring that down so you can see.

When you think about it that’s a pretty darn good selection and that’s the basic process there.

Thank you for watching this video. My name is Jeff Sengstack an Adobe certified expert and the lead instructor here at If you want to watch this entire video lesson as well as other live classes and After Effects crash courses, then I invite you to check out the BlueFx After Effects Academy.  Just click the link below this video to find out what we’ve prepared for you in the After Effects Academy.

Join The After Effects Academy And Get Acces To This Full Lesson + Monthly Live Classes And After Effects Templates

Remus Hosu

Remus Hosu

Leave a reply

Scroll to Top

By continuing to use this website, you consent to the use of cookies in accordance with our Cookie Policy.