Here’s the official Photoshop definition for the Overlay blending mode:
Multiplies or screens the colors, depending on the base color. Patterns or colors overlay the existing pixels while preserving the highlights and shadows of the base color. The base color is not replaced but is mixed with the blend color to reflect the lightness or darkness of the original color.
Er… come again? I hope that was confusing to more people than just me. Although if you read it slowly, it starts to makes sense…
Recall that multiply darkens images (white makes the underlying layer show through completely, darker colors on the blend layer result in a darker image) while screen mode lightens images (black has no effect, while lighter colors on the blend layer result in a lighter image). You may remember that one application was duplicating the layer and setting the top one to multiply or screen mode. Multiply resulted in darker areas getting darker, while screen resulted in lighter areas getting lighter. Overlay does both of these things at the same time, so if we take our standard rose image, duplicate the layer, and set it to Overlay, the contrast of the picture is turned up quite a bit.
Duplicate layer on top, set to Overlay mode:
(I think that makes a pretty cool “edgy” or “glowy” effect. Like something you might see on a CD cover.)
The other aspects of Overlay mode are that it “preserves the highlights and shadows of the base color” and actually blends the colors of the two layers together. This experiment helped me to make more sense out of it:
Cherry blossom layer:
With the rose layer set to Overlay over the cherry blossom, you get this:
The cherry blossom layer keeps the strong highlight vs shadow areas, but gets tinted by the yellow and red of the overlaying rose layer.
While I’m still not fully satisfied with my understanding of the Overlay mode, it was pretty easy to come up with a practical application: Scanlines with a kick!
In my rose file, I start by creating a new layer that has black and “transparent” stripes:
(You can do this by making a 1×2 pixel document with a transparent background, drawing a black pixel on half of it, selecting the whole thing, choosing Edit > Define Pattern, closing the document without saving, going to your image document, making a new layer, and setting the paint bucket to fill with your new pattern, then filling the layer. Or, you can make a new layer in your document, draw a black line across it, duplicate the layer and move it up two pixels, merge the two layers, and keep duplicating/merging until the scanlines fill up the document.)
With the scanlines layer set to Overlay, you get a scanlines effect, but in my opinion it makes the picture kind of dark. (Although maybe you would find that effect useful.)
So let’s do a little more to it. First, duplicate the original layer and set it to Overlay mode. You’ll get the high-contrast image from our first example:
Now option-click on the scanlines layer (for a dizzifying selection effect), click on the Overlay (duplicate flower) layer in the Layers palette to select it, and then create a layer mask by clicking the Layer Mask icon at the bottom of the layers palette. This creates a layer mask made up of tiny pixel rows of the overlaid flower layer.
Hide the scanlines layer, and this is what you get — a scanlines effect with a vivid image!