I have no idea what is causing it, but I know it isn't really normal.
I think you're actually onto it - the problem is a bad normal map I think. Most likely it's as simple as an inverted axis and it can be fixed in your material properties, but it might be interted palette data on the map itself.
Also as far as the res goes you can still end up with mixed texels if your materials are not uniform, or if you've scaled further for one than the others. I feel like your texture, if it's not a different resolution, is still both noisy and blurry (not sure if there's a word for this - basically looks like you shot on the wrong filmspeed but it's not a photograph so i don't know what to call it).
I still think that your lantern, as th eonly source of light, needs a few more lights set up with different falloffs and more defined shadows. You skylight should probably be in the sage-slate range too, right now it's like, why are the rooves lit with an orange glow?