Experiment: using AI to turn pixel art into paintings

[Gil]

I'm using Waifu2x with noise reduction level 3, double resolution, then I'm running it through ESRGAN, and afterwards I do another Waifu2x scale pass, without noise reduction. I'm getting really nice results. If you want to do a similar experiment, you'll have to play around with it. ESRGAN is wonderful, but it's not very good at the initial step it seems.

Waifu2x: https://github.com/nagadomi/waifu2x
ESRGAN: https://github.com/xinntao/ESRGAN


Original:


Paintings:

[pistachio]

Hey Gil.
Looks pretty good, tho there is weird convolution going on where there is no AA.
So because of these artifacts it would probably be an interesting upscaling algorithm for more heavily rendered/realistic stuff. (AFAIK there was upscaling posted here many years back for flatter stuff, no AI invasion involved though, so it is basically just pragmatic.)
6 years on machine learning is still arcane voodoo to me so I don't have anything to say yet about the tech behind it.
But I'll compile and maybe give this a shot with PA in different styles for testing, with credit of course.

[Gil]

I don't mind the convolution artifacts that much (I could explain exactly why they appear btw, and you're correct that it's in part because of sharp edges), because it makes for a good story when I show these.

I've been thinking about going back and touching up areas that don't scale up nicely though. I assume that minor touchups could easily remove most of the artifacts.

That also brings me to maybe the most important aspect of this: I think this is another great tool for artists to use, but it's more fun if it's not a magic bullet. My next experiment will be to try and change the input to get better output results and to work with better intermediaries. I think the real fun will be to scale up, make adjustments, scale up, make adjustments, in several steps.

Very thought-provoking in any case to work with these new technologies.

[pistachio]

Can't get at those 2x2 clusters but I'll give it the slack it needs. It is pretty cool what it dredges up when you feed it even this limited info.

And this is way way out but I'm thinking it would beat the algorithms you get in some emulators, until you factor in the time cost (and probably being limited to static art, it would be cool to see if animations go wack, I must try that) Until that day I'm still seeing only a weird limited use case scenario here.

That said most useful stuff (and probably most revolutionary stuff!) starts out like that, a fun tool, "useless" side project. It gets you places you never go by making a straight tool with a single goal in mind. Hammer/nail. OFC it depends, in situ. Yes, keep it up dude.

[Gil]

I think it's very useful for upscaling PS1 games. Like you said, probably still a bit expensive to do it on the fly, but you can prebake all the textures and use an emulator that allows texture packs, there's many of those out there for different platforms (PS1 and N64 for sure, not sure if there's SNES ones).

https://www.theverge.com/2019/4/18/18311287/ai-upscaling-algorithms-video-games-mods-modding-esrgan-gigapixel

[pistachio]

Yeah, AI is all the rage in mod packs dealing with new textures, things like that. It's only another tool, props to them. I'm just wondering what the result would be if you pushed that further, some kind of screen-space refinement or fun filter. dunno.

I never looked that deep into NNs (FOOL!) and it shows below, but
The picture I'm getting is it's recursive, kind of finds a pattern and locks onto that and applies that to the whole image, parts of the image if it's hierarchical (which the latest ones more or less are).
It sees round or square details or paths, that's what it uses to butcher the whole image, or that section. And it has limits in detecting the "context" of the hierarchy, which I'm guessing lead to the convolution artifacts.
If you have trillion dollar datasets, it's informed by that, gets godly results, and your company affirms world domination.

So blundering in, using Gil's steps, I tried some dither-heavy pieces that were pretty out there, basically to see if it comes up with trippy shit. Alas just stuff I need hanging on my wall now.

This will probably cause eyeburn, or did for at least one guy. Let me know if I should spoiler this.



Originals: [1] [2] [3]

Can't make heads or tails of a straight piece tho. here and original. But waifu handled it pretty well. Probably to do with its use case.

All hail AI...I mean NNs.

[Gil]

have you tried doing several steps of Waifu2x? I've gotten some very interesting results with it. Btw, one big thing to realize: both NN will generate images that, when downscaled using bicubic, result in the exact same image. So that's a large part of the artifacting, making sure it still downscales to the same image.

[Ozego]

screen-space refinement

I think that's how the raytracing already works. renders at super low sample rate and gets a grainy render, then AI filter magic time.

The issue with using this for pixel art is that you really should be using a deep filtering which has been trained using pixel art training data. But who wants to manually up-res down-res thousands of artworks for training data?
With a raytracer it is easy, you just render once at low samplerate and then at high sample rate and use these as training data.

[pistachio]

screen-space refinement

I think that's how the raytracing already works. renders at super low sample rate and gets a grainy render, then AI filter magic time.

The issue with using this for pixel art is that you really should be using a deep filtering which has been trained using pixel art training data. But who wants to manually up-res down-res thousands of artworks for training data?
With a raytracer it is easy, you just render once at low samplerate and then at high sample rate and use these as training data.

Sounds in the first line like you're suggesting some kind of raytracing-AI hybrid here would be optimal for the filtering/upscaling thing, correct me if I'm wrong. Not sure what sample rates are gonna do to a bunch of squares...I need a better example of that...