Building up a mental (or physical!) library of reference is very useful for this. When you play a lot of games, see a lot of art, etc, you build up a set of associations between different styles and traits of styles and the effects that they have, as well as what types of games they might be associated with and why. This knowledge will narrow down the field of styles that'll be effective for your project, and you can choose the most doable stuff from there. In addition, many genres have visual conventions that your game should probably also follow to make it easier for players to get into your game.
Some of the factors to consider when choosing an art style:
- What are the most important things in the visuals? Will the player be looking at/for things in the environment, or in the characters? For example, in fighting games, the characters are most important because you'll be looking at them to determine how to react to their moves, so the sprites are usually very big. In RPGs, the character sprites are usually small so that much more of the world can be seen, as a lot of the gameplay involves navigating the world. In addition to sizing, this will also affect where you put the detail.
- What is the mood you want to evoke? Chibi characters and pastel colours will probably feel out of place if you want a dark, moody feel to your game. Large character sprites don't really work if you want the player to feel insignificant in a grand world. A highly cartoony art style that reduces trees to green lollipops is probably not a good choice if you want to depict a world that feels real and mysterious.
- What are some of the key visuals in the game? For example, if you want a setting with a lot of mud (the GoT-style "Muddy Middle Ages" look), you'll want to look for a style that can depict mud effectively, as well as a muted colour palette. If you want a game that evokes the blocky look of Mayan art and architecture, then you'll probably want a blocky style. Let your key visuals influence the way everything else looks through either similarity or contrast.
- What do well-known games with similar stories and/or gameplay look like? You'll want to distinguish your game from them, as well as draw on their strengths and learn from their mistakes. Many players have visual expectations of certain genres that are informed by these popular games.
- What can you reasonably achieve? It makes no sense to go for a highly-detailed look where everything is animated if you're the sole artist and want to get the game done in under 15 years.
