HD Index Painting in Photoshop

[ptoing]

How does it look if you do it without dithering? And maybe if needed a few more colours per ramp.

[Atnas]

Index painting like this has saved me a lot of time in the past few months, thank you very much. Lots of instances of people just wanting the low color dithery look without actually caring about the pixels - and for less than half the time I'd of spent on the image anyway.

Oh and I actually prefer the 16 frame version. But it depends on what sort of game it would be used on in the first place. The 'resolution' of the animation should match the 'resolution' of detail imo, so I can see the source animation working in a very detailed game.

[Mathias]

It's a great technique. I tried implementations of this same trick in the past but never got it to work this well.

Looked at your blog article's PSD, finally. (using Photoshop CC 2014)
Smoothing out a pixel art workflow in Photoshop always interests me so I messed around a little:

Spoiler

Got 3 set of layers, for 3 different dynamically indexed elements in the canvas - two mountain ranges and the sky bg.

DITHER
Used Fill Layers set to Fill 0%, used Pattern Overlay in Layer Style.
Just a preference, really.
I find handling the dither pattern overlay easier this way.
Also, you can't accidentally draw on it and have any effect since its Fill is set to 0%.
I made sure to use the Position Lock (top of Layer Panel) for all Dither layers so I can click through them, since I leave my Move Tool's Auto-Select on all the time.
If you want to selectively dither only certain parts, use a Layer Mask on the Fill Layer. Use multiple selectively masked dither layers each using a different dither pattern.

ADJUSTMENT LAYERS MASKING
Rather than mask the indexing adjustment layers to one layer, I've masked them to groups that can contain other layers and layer groups.
This allows composite layers to all live in one layer group, getting the indexing effects. No need to commit to merging layers together - like the foreground figure in your youtube video. Multiple sets of indexed elements all in one PSD.
Notice all the cloud bg spiral Smart Objects (made into SO's just so I can rotate them freely) inside the "spiral moons" layer group. That layer group's blend mode is set to Lighten, affecting all layers within, without affecting the "clouds" layer.
Infinite possibilities.

LAYER COLORS
Using a Layer Panel color-coding standard can help when working with a PSD with lots of layers.
I have my own little meanings I've attached to the available colors.
Main goal is making things easier/faster to find.

And I wanted to find a way to eliminate all alpha transparency in a pixely way.



Using the Dissolve blend mode is all I really came up with. Always ugly, though.
But it does allow you to draw uninhibited with AA tools on a non transparency locked layer - all feathered transparency gets completely removed.
Maybe it's only useful as a safeguard - when using it, you know there won't be any accidental alpha blending going on (if you don't want it).

The motion paths of this flame are too large to pull off an 8 - i'd have to tone back the animation quite a bit and make it feel a lot higher-frequency in detail/motion. 

You're saying you'd have to use a much smaller patterns so the motion actually loops quicker, right?
Tween the pattern on the right moving over, and blending into, an underlying texture.



Once you've moved the smaller pattern a distance equal to the tile's dimensions you have a complete loop.
Just thinking out loud . . . visually . . . in a forum.



Anyway, super cool.
Thanks for posting your technique and opening my eyes to this.

[Indigo]

Very cool stuff Mathias.

DITHER
Used Fill Layers set to Fill 0%, used Pattern Overlay in Layer Style.
Just a preference, really.
I find handling the dither pattern overlay easier this way.

Probably an easier shortcut which I've adopted that accomplishes the same thing is to use a pattern "fill layer" (Layer > New Fill Layer > Pattern).  This ensures you can't paint on it, and you can easily select between multiple dither presets with just a couple clicks.

ADJUSTMENT LAYERS MASKING
Rather than mask the indexing adjustment layers to one layer, I've masked them to groups that can contain other layers and layer groups.

This is brilliant for organization, and I have no idea why I hadn't thought to do that.  Very good tip.

Using the Dissolve blend mode is all I really came up with. Always ugly, though.

So there's a trick to getting photoshop to handle the 1bit alpha for you, it's just a little less straight-forward to set up.  Basically, lets say you have a 16 color palette, what you'd do is define the ramp as though it's 17 colors, and make the darkest color on the ramp pure black.  On your paint layer, instead of painting on top of transparency, fill the entire layer with black.  Finally, open up the layer style for the paint layer and change the blend-if option for "this layer" so the dark-end is set to '1'.  This will ensure that pure black gets treated as transparent.

You're saying you'd have to use a much smaller patterns so the motion actually loops quicker, right?

Yup that's what i'm saying, but it's not just the bubbles texture i'm talking about.  I have even larger lower-frequency textures that produce the billowing wave effect of the overall flame.  Of course, it's not impossible to switch to higher frequency-textures for shorter loops, it just takes away a lot of depth from the animation and I didn't want to sacrifice that.

[Mathias]

Oh now we're getting somewhere! This is truly SUHWEET.
Glad to be able to contribute a little.

Probably an easier shortcut which I've adopted that accomplishes the same thing is to use a pattern "fill layer" (Layer > New Fill Layer > Pattern).

I've ignored pattern fill layers. Will try next time. Thank you for the reminder.
I finally read the comments on your site blog post. Good comments in there.
(A lot of confusion too, though. This technique is unfortunately rather advanced. I feel that even intermediate users will still have trouble it and/or not fully understand what they're doing even if they can get it to work. Not much can be done about that.)
I think you ought to update your tutorial information with this new stuff. I consider this a ground-breaking technique and your blog post is THEE how-to source.
And despite that, I'm getting the feeling even pixel artists aren't realizing how great this is. I am LOVING using this method.

This is brilliant for organization, and I have no idea why I hadn't thought to do that.  Very good tip.

The implementation of the ability to mask layers to layer groups is fairly new. CS6, I think. But it's absolutely revolutionary, in my opinion.

blend-if option

Aaaaaaaaaand I had to kick myself when I read your reply today - your 1-bit transparency solution is excellent.
I've never gotten useful results from the Blend-If layer settings (kinda fallen off the pixel art wagon so I'm always after smooth AA these days . . .) so, while I know what they do I've never really assimilated them into my little mental toolset. CRAP. So yeah . . . good call there!
Sure it means you can't use a pure black on the Blend-If layer/group, but you can probably mask an adjustment layer to it that darkens the darkest index in the ramp to MAKE it black again.
May have to tweak the "Advanced Blending" checkboxes in Layer Style to make it do what you want, but I think it might work. Need to just test it . . .
(Have you tried using ALT+RIGHT-CLICK DRAG to change your PS brush's size and feathering? Note the "Vary Round Brush Hardness based on HUD vertical movement" option in Preferences > General.)




I have another trick I'll post here, but you probably already know it - Using the Gradient Map adjustment layer to recolor underlying layers with an already existing palette.
Just like you've already been demonstrating, but with one additional gimmick - strictly stepped ramps. Like this:



Where you see the hard transition between colors there's two gradient stop thingies, both occupying the same exact location in the gradient ramp area.
This eliminates ALL intermediate colors. As you can see, I've created a ramp that uses the browns from the Commodore64 palette.
Otherwise, you get random intermediate colors re-coloring your underlying HD layers' pixels. If you still want to use the Poster adjustment layer, you can. But it's not needed.
Made a few more grads too, saving them as presets. GRAB MY PHOTOSHOP GRADIENT PRESETS FILE
When setting up the stops, it's necessary to click on ones that are stacked up on each other to toggle their stacking order. You'll see why.

Applying all these techniques:



32 frames          13 colors          1x version

What else?
Let's perfect this system.

[9_6]

I have an older version of photoshop without the colorpicker that only picks from the current layer so I had to come up with a way to control dithering.



1) Set everything up as described and make a black to white gradient.
With the box selection tool set to a fixed size, select areas in which you see each color level and dithering respectively.

2) With your drawing layer selected, press ctrl+c and ctrl+v.
You will get a new layer with your palette.
Make sure to move this layer above the pattern and posterize thingamajig!

3) Feel free to lay your palette out in any way you are comfortable working with.
You can copy it just above the drawing layer to get a preview of the colors you pick from.

Edit:

I thought my photoshop was bugged at first but then I found out that you need to untick "Blend Clipped Layers as Group" for it to work.

Edit 2:
Never mind.
Seems like that blend if option is bugged on cs2 and won't count as "transparency" for clipping groups.
Pencil eraser is the only option then.

[tim]

This is absolutely breathtaking. Thank you Dan.
It opens a lot of possibilities.

Right now I'm playing with Retro Dither, a color palette / dithering plugin for After Effects.
http://aescripts.com/retrodither/

It could really help for those of us who are familiar with After Effects. It would be very easy
to make pixel art ocean, windy trees, or any too complex / subpixel effect with this.

[Mathias]

Well now, instead of using filters in AE, you can render out your video or complex object animations and import them (even cut up into layers if you want, with alpha masking so they composite correctly in PS) and apply The Dan Technique selectively - different HD indexing settings per layer or masked area.

Hopefully this means The Last Night is about to get even more aWeSoMeR!

[9_6]

Oh so I get to have animated dither patterns now.
Well that's neat.

[9_6]

What else?
Let's perfect this system.

I thought the blend if option was bugged since it'll not treat the cut off values as transparency for masks but that's not how it works.
If you want to use the blend if transparency method, you need to include black in the color ramp.
Got a bit caught up in the "correct way to set up a ramp" large picture eyecatcher and missed when indigo said it so here's a ramp that supports the blend if transparency method:



This is just for clarities sake though, personally I prefer to take the dissolve spraycan hit and clean that up later.