Head to your local Halloween haunted house or watch a horror movie, and you’ll probably see a creepy Victorian structure that simply exudes terror. But as art historian Sarah Burns points out, in the 1870s, Victorian houses were just…houses. “Half a century later, however,” she writes, “that very same style had become a signifier of terror, death, and decay.” When did we start to associate these houses with creepiness?

Reconstructing the context in which Victorian houses went from homes to horrors, Burns traces the origins of the haunted house trope. By the early twentieth century, all things “Victorian” had fallen out of style. Artists like Edward Hopper and Charles Ephraim Burchfield began painting abandoned Victorian houses, imbuing them with creepy character.

A broader cultural movement away from Victorianism “condemned all things of the later Victorian epoch as ugly, excessive, and un-American.” The mansard roofs and gingerbread accents that had made Victorian homeowners so proud were now mocked as indicative of new money and ostentation. Architecture writers eager to embrace a new modernism played up the supposed tacky excess of the past and promoted slick alternatives like Frank Lloyd Wright’s “pure” designs.

After World War I, America turned its back on Victorian design even more vehemently. Returning soldiers saw death in the once uplifting factories and bright dreams of their Victorian fathers, and began to portray Victorian houses as ghostly remnants of a corrupt past.

“Short of demolition,” Burns writes, “only extreme measures could exorcise the ghosts, purge the present of the diseased past, and sweep its filthy droppings from the land.” Houses were stripped of their characteristic gingerbread in real life. But in the imagination, they began to take on a new architecture of fear.

Suddenly, Victorian houses were alive with uncomfortable connotations. Artists drew them as seemingly empty shells that signified decay and danger; they imbued them with their own psychological angst. In the 1920s, writes Burns, Victorian structures became a kind of shorthand for fear as artists began incorporating them in literature and theater. Murder mysteries were set in empty Victorians—and as more and more were torn down in real life to make room for modernity, they took root in the imagination instead.

During the 1930s, photographs like Walker Evans’s Victorian Series, a documentary project that captured America’s largely abandoned Victorian-era houses, solidified the popular view of the homes as spooky and dangerous.

Eventually, art began to portray the houses as haunted, a quick shorthand for creepiness that made sense to audiences of films like Psycho and television shows like The Addams Family. “The old dark house,” Burns writes, “had become a capacious symbolic vessel serving a range of cultural functions and desires.”