So I was doing my usual periodic poke at the problem of automatic, respectful AAing of pixel art,
And recently began to understand how GMIC's 'anisotropic smoothing' function works. (gmic: http://gmic.sourceforge.net/)
. . .
Very exciting stuff, AI! I'm trying to get a coder friend to compile it for me so I can try it out.
Why..? There are precompiled Windows binaries. (
http://gmic.sourceforge.net/gimp.shtml). Although you might get better speed by compiling your own, when running a 64bit OS.
Anyway, the GIMP plugin is the way to try it, definitely; running from commandline does not compare in terms of experimenting.
My latest iteration -- I'm fairly happy with this one. Process changed:
* Edit->Copy original image
* GMIC 'smart upscale' (also in the enhance section) with smooth=1, anisotropy = 1, sharpen = 40.
* downscale to original size with linear interpolation (okay, now we have the smoothed version. Time to restore the small detail)
* Edit->Paste, and turn it into a new layer
* Add a layermask to that layer
* Edit->Paste (into the layermask)
* Alt-click to see the layer mask (obviously this is only if you are doing it manually)
* GMIC 'Difference of Gaussians' (in 'Contour') variances 0.10 and 0.20
* Apply Curves with a contrast curve (points at coordinates 84,39) This will mask out the low frequency detail (lines etc), and enhance the high frequency (dithering).
* Export a PNG

^^ intentionally designed to be automatable.

Then I adjusted the layer mask a bit so that every pixel of water showed the smoothed result.
Then I smoothed the water in the underlying layer some more (amplitude = 2, iterations= 2, other settings as specified in my previous post).
Pretty happy with how that's working -- only a little touchup of the highlights is needed IMO.

Next on my list is to autodetect the usable colors for a pixel with reasonable reliability.
(that is, given a pixel and surrounding 4 pixels, choose an appropriate color ramp based on image color ramps.)
I think I can do this by scanning the image and constructing a weighted graph where the weights are
frequency counts: This pixel color shows up N times next to this pixel color. Then 'cutting' the graph where
frequency is too low, and also cutting where improbably large jumps in hue are made, then iterating over all nodes connected to the source pixel color,
to determine a list of possible destination pixel colors.
Then quantizing auto-AAed images back to the source palette may become generally easy.