Mastering Complex Selections in Photoshop

This entry is part 2 of 2 in the series Complex Selections in Photoshop

Complex Selections in Photoshop

Having the ability to effectively extract an image from its background is an essential skill for any designer, photographer, or creative professional. Many times we have unique challenges when attempting to separate a subject from its setting. Obstacles like poor contrast or the intricate shapes formed by hair and fur can make the process nearly impossible without the right techniques. In the first installment of Making Complex Selections in Photoshop, I covered fairly easy objects to select, such as a woman with a few wisps of hair sticking out, but this time we are going to cover making even more complex selections in Photoshop.

The example image below can be found here. To try and select each individual strand by hand would be extremely painstaking and nearly impossible. You could spend all day attempting to select this cactus with the lasso tools, and even revealing each strand with a mask would yield poor results. It would take hours to refine the mask for each strand by hand. Instead, select the Quick Selection Tool and make a loose selection of the object by clicking and dragging within the area. Hold Shift to add to the selection and hold the Option/Alt key to subtract from the selection. If you find that it’s taking too long to make a selection, increase the size of the Quick Selection Tool, as you would the Brush Tool, by hitting the “]” Key, until you have a decent sized cursor. Hit “["  to lower the size for more tedious areas. (If you’d like to learn more about helpful, time-saving Photoshop keystrokes, check out 30+ Time-​​Saving Photoshop Keyboard Shortcuts.)

You can see from the image above that the selection is very loose. With the selection active, go to the Options Bar and click Refine Edge. The Refine Edge dialog box will come up, where you can refine and adjust your selection very precisely. Click the Smart Radius option to enable it, and move the slider all the way to the right. On more solid objects, be careful when using this, because it will take away solid, continuous areas of color around the edge that blend with the background color. However, this setting is perfect for fine, wispy hair or fibers. Click Decontaminate Colors and move its slider all the way to the right. This removes edges where the background color may have added a thin, unwanted fringe around the object.

You can see that the Refine Edge menu has already done a great job of extracting the fine, fibrous edges around the cactus. Hit the “F” key to toggle between views and you can see that the mask view of the selection is very intricate and precise.

Select Output to New Layer with Layer Mask. This will create a new layer above the working layer based on the mask that we just made and refined. If there are any problem areas, we can fix them, but in this case we have none. Now, you can place your selected object over anything that you’d like without color contamination or any other concerns.

The fuzziness is just as prominent as it was in the original image, but now we can slide any color or image behind our object and composite a layout quickly and easily. Below, I added a bold purple to show the quality and precision of this selection technique.

The next sample, which you can download here, has a decent amount of contrast between subject and background, but the fur is long and feathery, which is nearly impossible to select by hand. Select the Quick Selection Tool and make a loose selection, making sure to include as much of the bear as possible.

Using Smart Radius and Decontaminate Colors seems to do the trick instantaneously, but before exporting to a new layer, it is always good idea to hit the “F” key multiple times to toggle through the different backgrounds to make sure that you haven’t missed anything. The selection below looks good on white…

However, when we toggle through the different views (using the “F” key), when we get to the black background, where I realized that I had apparently missed a spot. Notice the area of white to the left of the head.

It turns out that the brush tool would not remove this area. It was actually quicker to exit the Refine Edge Menu by hitting cancel, and then holding Alt/Option while clicking with the Quick Selection Tool to remove that area from the selection. Afterward, I re-entered the Refine Edge menu, used the Smart Radius and Decontaminate Colors options, and exported the image to its own layer with a layer mask. The result is below. Note that there are no longer remnants of white anywhere around the bear.

The next example is a little more challenging due to the fact that there is background interference caused by the water. You can find the sample image here. Just like with the image before, make a rough selection with the Quick Selection Tool.

The difficult part about this selection is that the woman’s hair feathers out at the ends, and the dark, feathered ends blend into the foliage in the background, making a precise selection nearly impossible. Click Refine Edge in the options bar to bring up the Refine Edge dialog box. It is here that we will end up making most of our refinements. Check Smart Radius, and move the slider all the way to the right. Then, click Decontaminate Colors, and move that slider all the way to the right. In the drop down menu, choose New Layer With Layer Mask, but don’t click OK yet, because our selection isn’t quite as good as it needs to be. You can see that it needs to be more refined by toggling the different views (using the “F” key) until your image is placed over a white background.

The problem is that the top area — where the hair blends into the rest of the image — has a lot of extra color information. It isn’t as clean of a selection as it could be.

This problem can be solved with the brush tool found in the Refine Edge dialog box area. You can use one of several methods to manipulate your selection. The first option is to simply single-click over an area that should be included in the mask. Photoshop will resample the selection and attempt to add this to the original selection. The gray areas within the hair need to be removed; that is color information that doesn’t need to be included in the selection. Holding Alt/Option and doing a single click over these areas will cause Photoshop to recalculate the selection and remove these unwanted gray areas from the results. Two well-placed single-clicks while holding the Alt/Option key in the grey areas to the left and right of the strands of hair that are sticking straight up made a much better selection in these areas.

Now, the problem is that too much of the hair has been removed. We need to add image information back into the document, but the brush isn’t precise enough to pull this off. To add this information back in — without accidentally adding the water and foliage back in — move the Shift Edge Slider to the right to add image information gradually back into the selection. The value used here is 25. Then, move the contrast slider to the right roughly 10 points, and you’ll get a much better selection. Click OK to have Photoshop Duplicate the layer and apply the selection as a mask.

If you need to make any further adjustments, you can use black to paint out unwanted areas of the image,  and you can use white to paint them back in on the mask. When you are finished, you can apply the mask to the image, so that you can drop it over a different background, or you can leave the mask in order to make more adjustments later. Here is the result placed on an entirely different background:

Conclusion

Making Complex Selections in Photoshop used to be a long, time-consuming process that took hours upon hours. Some images were thought of as impossible to extract from their background, but with the Refine Edge menu, you have the ability to quickly refine your selection to include fine, feathery sections of hair, animal fur, and long stands of fibers that are nearly impossible to work with by hand. The result is better Photoshop work, happier clients and colleagues, and lots of time saved.

Do you have any selection techniques of your own for complex selections? Are you often asked to perform very difficult Photoshop within a very small time frame?

Complex Selections in Photoshop

<< Making Complex Selections in Photoshop

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • http://www.greenmantle.biz Mark Greenmantle Photography

    Thanks James, that’s one of the best how-to’s I have seen for masking selections with such tricky subjects. It’ll certainly speed my workflow up quite a bit =)

  • http://www.vitessewebdesign.com.au VWD

    A very useful and practical tutorial. The Refine Edge functionality is a powerful timesaver. Thanks you, James, for elucidating in an easy to follow manner.

  • http://www.yoga-singapore.com Kian Ann

    WOW! This will be REALLY useful! Awesome tip! Didn’t know about the feature. I’m trying this on CS6, but since when was this “refine edge” feature available?

    • http://gsquaredstudios.com James George

      Kian, it was available starting with CS5, I believe.

  • Raghavendra

    Nice example.I want to know is it possible to insert image behind any background scene of other image.
    Pl advise thanks

  • http://hbflyte.com Heather

    This article may be the one that finally convinces me that patience and the smudge tool are not the only way to cut out hair. :)

  • Jim Ault

    Firstly – a very good article, but typifies one of my biggest tutorial pet peaves… there is no mention of the version of Photoshop. This article is only useful for CS6 users, thus less than 90% of the current user base. The first clue is the picture of the Refine Edge dialog. If you don’t have this dialog, you might as well stop reading.

    Another example is your – Color Matching Photos in Photoshop, where the reader should be told that this appeared as far back as CS, and is a destuctive function, therefore not part of the modern Smart Adjustment suite.

    Secondly – I would like to learn more about your tutorials and expertise. I like that you are on Google+ and posting regularly. Are you making plans to expand more in the social arena ? Are there already good designers on Google+ that would be competitors? / allies?

    Thanks, and I look forward to your reply.
    ( +Jim Ault Tampa, FL )

    • http://gsquaredstudios.com James George

      Jim,
      Thank you for your comments, compliments, and concerns. You are right, from now on I will try to mention which versions certain features are available to, especially if the entire technique or tutorial (like this one) are dependent on having these features.

      For destructive functions such as the color matching tutorial, I will inform the reader to duplicate their background layers, which is a common practice when performing destructive correction. I prefer to try to use Smart Adjustments, but they aren’t available for every single feature in Photoshop.

      I am fairly active on most social media venues, such as Twitter, Google+ and LinkedIn. I am always looking to expand my reach within the design community, and I announce all of my articles and tutorials via twitter and Google+, mainly, as well as on LinkedIn and other places. There are plenty of competitors out there, as well as allies. I just try to share useful information that will help fellow designers and make our lives easier, our work day more productive, and our professional lives less stressful. Thanks for reading and expressing your interest Jim. your presence and participation are much appreciated.

  • http://rodleg.wordpress.com/tutoriels/ Rodleg-design

    Nice photoshop tutorial

  • http://www.flashybee.com T K Ghosh

    Just want to know what photoshop version I have to use to get all the tools used here.

  • http://photoperformance.org/ David Salahi

    In the first example, the cactus, did you use the Refine Radius brush? You didn’t mention it in that part of the tutorial but I found that I was unable to achieve results as good as yours until I did that. I had been using selections similar to yours and the same settings as shown in your Refine Edge dialog. But, for me, the key to making it work was brushing all around the perimeter with the Refine Radius tool.

    Other than that, this is a terrific tutorial. It was great to have access to high-res versions of the same photos you used for demonstration purposes. That allowed me to make comparisons which showed whether my technique was right.

  • http://gsquaredstudios.com James George

    Thanks David!

    To answer your question, no I didn’t have to use the brush. The reason that this happens is simply because you and I may not have had the exact same general selection. Using the quick selection tool, every selection will be different, so it really just depends on the situation.I may have had a couple of extra pixels selected, which made all the difference.

  • http://www.flashybee.com T K Ghosh

    Is it in Photoshop CS 3?

  • Miranda

    I am a beginning graphic/web designer (just two years now with mostly web, just getting into more precise print work) and I have some questions. I am seeing many of the same type of tutorials for cutting out hair, and yes this certainly does help a LOT. But what I see in each tutorial is that the end result is frankly still not good enough. There is still a ton of the background color around the selections- on the cactus, on the left- look at all that blurry white- that is not acceptable to present as a professional project. In the last example, look at the right side of her hair- it appears as if there is fringe not even attached to the rest of her hair, black fringe just hanging out there and again the sides look blurry. How do we fix this?? Seems every tutorial I see just leaves it like that calling it perfect, and professionally, it is NOT perfect when it absolutely has to be for a commercial project….think of putting that on a magazine cover… it would not be acceptable. Any solutions?

  • Rami Haimov

    Hello there and thanks very much for the interesting instructions.
    I am trying to clean a peacock out of its background and so far with out any success.
    I was trying to use the channel selection technique and the refine selection technique that you’ve mentioned but as I said it did no good to my image.
    The problem is that not only the peacock has lots of stray hair, that hair holds as a background – similar and non similar colors.
    I was hoping to use the color selection tool for help but it did not do much good either.
    Thanks for your help – in advanced – Rami.
    p.s. – you can get a peacock image from the web and see what I am talking about.
    Cheers.