The far border has spurious highlights, which also contribute to making it look jagged. The reference doesn't look like that.
The middle is perfectly flat, like a floor tile would be. The reference obviously isn't; you might need to exaggerate the curvature (and to invent a random bump pattern), but it is necessary to make it look like a pillow.
I don't think you have a sufficient pixel budget for the wrinkles around the middle of the right side; I don't think they are necessary to make the surface look like cloth (the natural shapes around the corners are enough).