Rounded/octogonal rooms might be rare, but it's very common to see rooms that aren't square or rectangles, but more like a bunch of rectangles super-imposed, such as L-shapes or blocks with corners missing. It's not uncommon to have weird angles, too. Rooms are rarely perfectly symmetrical. Symmetry might seem to make sense when you think of just a room, but in the context of a building with multiple rooms, it's usually impractical.
Check out the weird bedroom shapes here, the result of having to fit in closets and a bathroom.
Even this simple, blocky layout doesn't have a single room that's a proper rectangle.
This example has some 45 degree angles, perhaps because the house is in an odd location.
While I doubt anyone expects you to become a proper architect just to make a game, thinking about how your different areas interconnect and fit inside a building can help you make a more believable location, one that feels like it could exist. It doesn't have to make perfect sense; it's perfectly fine to have some impossible architecture like rooms that are larger or smaller than the hallways between them would suggest, or overlapping spaces, if those would make the gameplay better. However, each individual space should feel like it's part of a larger building, and the occasional angled corner or L-shaped room is a great way to achieve that.
