You’ve probably walked past the corner of Fifth Avenue and 52nd Street in Manhattan and seen nothing but luxury retail and the polished glass of modern New York. It’s hard to imagine that this specific patch of dirt was once the site of the most hated house in the city. The residence of Charles H. Lohman wasn't just a home; it was a four-story middle finger to the Victorian elite.
Back in the 1860s, if you were wealthy, you stayed away from the Lohmans. Charles and his wife, Ann Trow Lohman—better known by her scandalous alias, Madame Restell—didn't make their fortune in railroads or shipping. They made it in "female medicine." In plain English? They were the most famous and successful abortion providers and birth control advocates in 19th-century America.
When they built their massive brownstone mansion at 657 Fifth Avenue, it wasn't just a real estate move. It was an act of war.
The Most Hated House on Fifth Avenue
Let’s be real: New York society in 1864 was obsessed with "Old Money" and "Character." The Lohmans had neither, according to the neighbors. Charles R. Lohman, a radical printer by trade, was the brains behind the branding, while Ann was the face of the operation. They were rich. Filthy rich.
They spent roughly $600,000—a staggering sum in the mid-1800s—to build a French Second Empire mansion that put most Astor and Vanderbilt homes to shame.
- The location: The northeast corner of 5th Ave and 52nd St.
- The materials: Massive blocks of Connecticut brownstone and brick.
- The Vibe: Pure, unadulterated flex.
The mansion featured a grand stone staircase, multiple balconies, and a roofline topped with a regal balustrade and giant finials. It was beautiful. It was also a monument to what the upper crust called "vice."
📖 Related: Finding the Right Words: Quotes About Sons That Actually Mean Something
Honestly, the neighbors tried to stop them from even moving in. They sued. They petitioned. They did everything they could to keep the "wickedest woman in New York" and her husband from living among them. It didn't work. The residence of Charles H. Lohman stood tall, and for over a decade, the couple lived there in a state of high-society isolation.
Living in a Gilded Cage
It’s kinda fascinating to think about their daily life. While the mansion was "sumptuously furnished," as the New York Times begrudgingly admitted, the Lohmans were social pariahs.
They had many servants. They drove fast horses. They wore the finest clothes. But nobody came to their dinner parties. Their only neighbor for years was the Roman Catholic Orphan Asylum, which is a bit of dark irony considering Madame Restell’s profession.
Charles was more than just a bystander in this. He was a freethinker and a radical. He helped publish medical companions under the name "A. M. Mauriceau" that advertised birth control and condoms. In a way, their residence was the corporate headquarters for a revolutionary (and illegal) healthcare empire.
The Dark History of 657 Fifth Avenue
The residence of Charles H. Lohman is perhaps most famous for how it all ended. In 1878, the notorious morals crusader Anthony Comstock went on a personal mission to take the Lohmans down. He used a sting operation to catch Ann selling "contraceptive materials."
👉 See also: Williams Sonoma Deer Park IL: What Most People Get Wrong About This Kitchen Icon
On the morning of April 1, 1878—April Fool's Day—Ann Trow Lohman was found in her bathtub in the mansion. She had slit her own throat.
The house that was built as a symbol of defiance became a crime scene. Charles was left with the mansion, but the soul of the place was gone. After his death, the house didn't last long. New York has a habit of tearing down its complicated history to make room for the next big thing.
What Happened to the Lohman Residence?
If you go looking for the house today, you won't find it.
- Demolition: The mansion was eventually torn down in the early 20th century.
- Replacement: The site was later occupied by the Morton F. Plant House, which is now the Cartier building.
- Legacy: Today, the spot is one of the most expensive retail locations in the world.
A Different Lohman: The Jefferson City Connection
It’s worth noting that "Charles Lohman" is a name that pops up elsewhere in architectural history. If you're searching for the residence of Charles H. Lohman and finding stone buildings in Missouri, you're looking at a different guy—Charles F. Lohman.
That Lohman was a merchant in Jefferson City. His "residence" was actually the upper floors of what is now known as the Lohman Building at Jefferson Landing. It’s a sturdy, 18-inch-thick stone warehouse that served as a grocery store and hotel. It's now a State Historic Site.
✨ Don't miss: Finding the most affordable way to live when everything feels too expensive
Kinda weird how two different Charles Lohmans in the 1800s both ended up with buildings that became historic landmarks for totally opposite reasons—one for scandalous Fifth Avenue wealth, the other for frontier commerce.
Why We Still Care About the Lohman Mansion
Why does the residence of Charles H. Lohman matter in 2026?
Because it represents the tension between private life and public morality. The Lohmans weren't just "criminals" or "doctors"; they were people who used architecture to demand a seat at a table that didn't want them. Their house was a physical manifestation of the "New Money" struggle and the fight for reproductive autonomy long before that was a standard political term.
If you’re interested in exploring this history further, here’s how to do it:
- Visit the site: Stand on the corner of 52nd and 5th. Look at the Cartier building. Imagine a dark, imposing brownstone with a "Female Physician" sign nearby.
- Read the archives: The NYC LGBTQ Historic Sites Project and various 19th-century newspapers have digitised accounts of the "Restell Mansion."
- Check the museums: The Museum of the City of New York often holds lithographs of the Lohman residence in its Gilded Age collections.
The house is gone, but the story of the Lohmans—and the audacity of building a mansion on the most prestigious street in the world using the profits of a forbidden trade—remains one of New York’s most compelling ghost stories.
Summary of Actionable Insights
- Research the context: Understand that 1860s Fifth Avenue was the "Wild West" of high society. The Lohmans were early adopters of the area.
- Verify the person: Don't confuse the New York City Lohman (the printer/abortionist ally) with the Missouri Lohman (the merchant/landing owner).
- Map the history: Use resources like the Daytonian in Manhattan blog or Wikimedia Commons to see the actual lithographs of the building before it was demolished.
The residence of Charles H. Lohman serves as a reminder that the buildings we see today are often built on top of secrets that the city once tried very hard to bury.