For years, much of the recycling Americans set out at our curbs has ended up in less wealthy countries like China and India, but now these countries are putting more limits on the dirty industry. Recycling has always been something most people would prefer to keep at arm’s length, as Carl Zimring explained in a 2004 paper.

Zimring writes that scrap trading in the U.S. began in the colonial era, but became big business after the Civil War. Growing manufacturing firms demanded scrap paper, steel, rags, and other cast-off materials. Meanwhile, households were discarding more stuff then could be reused. In 1840, New York City directories listed 40 junk sellers and no companies specializing in trading rags. By 1866, there were 250 junk companies and 132 rag sellers. The industry continued to grow in the decades that followed, fueled by military production during World War I and by a growing consumer culture in the 1920s.

Recycling was hard, dangerous work. Scrappers had to take apart decommissioned ships, rail cars, and weaponry, using shears and acetylene torches. Rag dealers handled cloth contaminated with germs and stored it in their warehouses and shops, creating fire hazards. Anyone with other employment prospects tended to avoid the work. Since it was mostly done by hand and required minimal capital investment, however, it was an easy business for immigrants with few other options to enter. By the early twentieth century scrap dealers were largely Jewish immigrants.

With many small operators competing for customers, some dealers misrepresented their products to undercut the competition. The real threat of unscrupulous dealers merged in the public imagination with anti-immigrant and anti-Semitic stereotypes. “The popular image of the junk dealer at the turn of the twentieth century was a hook-nosed, scheming Jewish immigrant bent on cheating honest American businessmen,” Zimring writes.

The public at large came to distrust junk sellers. Some municipalities passed laws to limit rag sellers’ operations because of the threats of fire and disease they posed. Metal dealers were also seen as a threat to children. Social reformer Jane Addams described boys in Chicago breaking into vacant houses, selling stolen faucets and pipes, and using the money they received to buy beer.

In response to all this negative publicity, scrap dealers formed associations. They lobbied against government restrictions, punished rogue members, and launched public relations campaigns. As concerns about the destruction of natural resources arose in the twentieth century, they sought to associate themselves with conservation. The companies portrayed their work as a rational, efficient effort to keep valuable resources from being wasted. Part of what made this rebranding possible was that the companies were increasingly located in industrial areas near railroads, rather than in dealers’ backyards in plain view of the neighbors.

Decades later, reclaiming U.S. junk is often done even farther out of sight, on the other side of the world. But the growing resistance from foreign governments may force us to pay more attention to this dirty work.