It's probably because the noisiness on his do-rag is more apparent than it was when you had all those extra colours softening the transition. With fewer colours, just as with fewer pixels, the position of every pixel counts, and you're not using your shadows to show the form of his head. The dithering(?) looks like noise, because at this size there isn't really room for it. Instead, use that limited space to make the head form clear, and maybe suggest some folds for a little extra polish.
I don't feel like you need the middle colour on his beard, the colours are similar enough that you can probably do just fine with only two of them. Also, it looks a bit weird that his beard is lightest in the middle and darker above and below. I'm guessing you were trying to have outlines on the hair? At this size, there isn't really room for them, so you're better off using that space to show volume of the beard. So, bring the lighter colour into the upper bits.
You changed the eye shape on him, making him look like he's squinting. I think that's an improvement over the eyes from before, but if that's what looks weird to you, try lightening his lower eye-lid, have each of those three pixels be the lightest colour (with appropriate AA below it). That will give the impression that the eye is open a little wider when the image is viewed at 1x-2x.
Speaking of 1x-2x, unless you're planning to only ever show a piece at a higher zoom, make sure you have a 1x preview open (or zoom out to 1x very frequently if your program doesn't allow making such a preview). When you're zoomed in, it's hard to see the relationships between clusters and the details that they create, and it's easy to get stuck placing pixels according to specific objects you're trying to represent without concern for what they actually end up looking like. With pixel art, especially at smaller sizes, each pixel often serves to represent more than just one thing, but it's hard to see how well that's working while you're zoomed in.