Waves on the shore pull out after they've crashed in. You're animation doesn't really convey this. see .TakaM's tile for a nice example of this:

now your current animation still isn't smooth. you might have to increase the framerate or increase the frames.
some things to consider
Waves on the shore:
Once the water is pushed in it is sucked back out.
Once a wave hits the shore it travels more slowly due to the friction of the ground underneath.
This is what actually causes the wave to break, the top part of the wave is travelling faster than the bottom part and the wave curls over and breaks.
Waves off the shore:
Motion on the top of the wave can be modelled by a
sine wave.
These waves do not have crests unless there is something like a sandbank underneath.
Don't know how useful this stuff will be for you. I suggest sketching out the motion with a couple of colours before adding detail. This will help with the smoothness issue.
Are you imposing any constraints on this animation?